Inspecting a DataFrame

DataFrame inspection is the short investigation between “the file loaded” and “the transformation is safe.” It establishes grain, types, missingness, categories, and suspicious records before code turns assumptions into output.

Problem

A customer CSV loaded without error, but you do not yet know whether one row means one customer, whether IDs are unique, whether dates parsed, or which fields contain nulls. Calling head() is useful, but five convenient rows cannot answer those questions alone.

Dataset used

Use the Customer Dataset and download customers.csv. Its issues are controlled: three blank emails, three inconsistent country labels, and one exact duplicate. That gives each inspection a known finding.

Code: a repeatable first profile

from pathlib import Path

import pandas as pd

customers = pd.read_csv(
    Path("data") / "customers.csv",
    parse_dates=["signup_date"],
)

print("shape:", customers.shape)
print("columns:", customers.columns.tolist())
print(customers.dtypes)
print(customers.head(3))
print(customers.tail(3))

Shape provides the table boundary. Column names expose unexpected whitespace or schema drift. Data types show whether numbers and dates are usable as intended. Head and tail samples catch obvious delimiter, header, and trailer problems. Neither sample proves quality throughout the file.

Output

Check	Expected observation
Shape	100 rows, 6 columns
Customer ID	Loaded as integer in this file
Signup date	datetime64[ns] after parse_dates
Email	Three missing values
Duplicate rows	One exact duplicate

Use info() for a compact schema and non-null summary. It writes to a buffer or standard output and returns None, so do not assign its result back to the DataFrame.

customers.info()

profile = pd.DataFrame({
    "dtype": customers.dtypes.astype(str),
    "null_count": customers.isna().sum(),
    "null_percent": customers.isna().mean().mul(100).round(1),
    "distinct_count": customers.nunique(dropna=False),
})

print(profile)

This small profile is often more useful in a pipeline log than a full descriptive report. It keeps the comparisons explicit and works for strings, dates, and numeric columns.

Code: test the expected grain

If the business grain is one row per customer, customer_id should be unique and non-null. Test that statement directly.

duplicate_rows = customers[customers.duplicated(keep=False)]
duplicate_ids = customers[customers.duplicated(subset=["customer_id"], keep=False)]

print("null customer IDs:", customers["customer_id"].isna().sum())
print("unique customer IDs:", customers["customer_id"].nunique())
print("exact duplicate rows:", customers.duplicated().sum())
print(duplicate_rows)

The file has 100 rows but only 99 unique customer IDs because one record is repeated exactly. In real data, duplicate IDs may contain conflicting attributes rather than identical rows. drop_duplicates() would then keep one record according to row order, which may be arbitrary. Inspection comes before resolution.

Explanation: inspect distributions, not just types

A string column can be technically valid while semantically inconsistent. Normalize only after seeing the original values and counts.

country_counts = (
    customers["country"]
    .value_counts(dropna=False)
    .rename_axis("country")
    .reset_index(name="customer_count")
)

print(country_counts.to_string(index=False))
print(customers["segment"].value_counts(dropna=False))

United States and united states appear as separate categories. The same is true for Canada and CANADA, and for Spain and spain. That matters before grouping or joining to reference data. A lowercase display is not necessarily the final canonical value; production systems often map source variants to governed codes.

Numeric describe() output is useful but incomplete for mixed tables. Include specific percentiles when they answer an operational question, and inspect categorical frequencies separately. For dates, check minimum, maximum, future values, and parsing failures.

Common mistakes

• Looking only at head() and assuming the sample represents the file.
• Using describe() as a universal data-quality report.
• Treating an object/string dtype as evidence that values are clean.
• Dropping duplicates before defining the intended business key.
• Counting distinct values without deciding whether null is a category.
• Printing an entire large DataFrame and losing the important signals in noise.
• Changing values during inspection, which makes it harder to compare raw evidence.

Try it yourself

Build a one-row summary containing total rows, unique customer IDs, missing emails, earliest signup, latest signup, and exact duplicates. Assert that the required columns exist. Then list country values whose normalized form maps to more than one original spelling.

As a production-minded extension, turn the expectations into explicit checks that raise useful errors. Avoid asserting that emails can never be null—the dataset intentionally permits them. Good validation distinguishes allowed missingness from contract violations instead of treating every imperfection the same way.

Production notes

Inspection should produce evidence that can be compared between runs. Persist compact metrics such as row count, distinct business keys, null percentages, date boundaries, and category counts. A single profile describes today’s file; a sequence reveals drift. Alert thresholds should reflect business tolerance: one new country may be expected, while a customer-ID null rate above zero may be a contract failure.

Be careful when profiling sensitive data. Logs should contain counts and approved aggregates, not complete customer records or raw values that expose personal information. Sample only the columns and rows needed for diagnosis, redact where appropriate, and keep profiling outputs under the same access controls as the pipeline.

For large DataFrames, exact distinct counts and full scans have a cost. Select checks according to risk, run lightweight contract validation at ingestion, and perform deeper quality analysis where resources allow. The purpose is not to generate every statistic; it is to test the assumptions on which the next transformation depends.