Handling Missing Data and Basic Data Cleaning

Real-world data is almost never perfectly clean — missing values, duplicate rows, and inconsistent formatting are the norm rather than the exception. Knowing how to detect and handle these issues is one of the most practically important pandas skills, since messy data silently produces wrong analysis results if ignored.

What Missing Data Looks Like in Pandas

import pandas as pd
import numpy as np

data = {
    "name": ["Vishwas", "Priya", "Arjun", "Sneha"],
    "score": [88, np.nan, 75, 95],
    "city": ["Bengaluru", "Mumbai", None, "Bengaluru"]
}
df = pd.DataFrame(data)
print(df)
#       name  score       city
# 0  Vishwas   88.0  Bengaluru
# 1    Priya    NaN     Mumbai
# 2    Arjun   75.0       None
# 3    Sneha   95.0  Bengaluru

Missing values appear as NaN (Not a Number) for numeric columns or None for object columns — both are pandas' way of representing "no value present."

Detecting Missing Values

print(df.isnull())
#     name  score   city
# 0  False  False  False
# 1  False   True  False
# 2  False  False   True
# 3  False  False  False

print(df.isnull().sum())   # count of missing values PER COLUMN
# name     0
# score    1
# city     1
# dtype: int64

Dropping Rows or Columns With Missing Data

df_dropped = df.dropna()              # drops ANY row containing at least one NaN
print(df_dropped)

df_dropped_cols = df.dropna(axis=1)   # drops ANY column containing at least one NaN

dropna() returns a NEW DataFrame by default — it does not modify the original unless you pass inplace=True, the same pattern used throughout pandas for almost every cleaning method.

Filling Missing Values

df_filled = df.fillna(0)                          # replace every NaN with 0
print(df_filled)

df["score"] = df["score"].fillna(df["score"].mean())   # fill with the column's average
print(df)

Finding and Removing Duplicate Rows

data = {"name": ["Vishwas", "Priya", "Vishwas"], "age": [22, 21, 22]}
df = pd.DataFrame(data)

print(df.duplicated())   # boolean Series flagging duplicate rows
# 0    False
# 1    False
# 2     True

df_unique = df.drop_duplicates()
print(df_unique)

Renaming Columns

df = df.rename(columns={"name": "full_name", "age": "student_age"})
print(df.columns)   # Index(['full_name', 'student_age'], dtype='object')

Fixing Inconsistent Text Data

cities = pd.Series([" bengaluru", "Mumbai ", "CHENNAI"])

cleaned = cities.str.strip().str.title()
print(cleaned)
# 0    Bengaluru
# 1       Mumbai
# 2      Chennai
# dtype: object

The .str accessor unlocks vectorized string methods on an entire column at once — .strip() removes stray whitespace, and .title() standardizes capitalization, both applied to every row in a single call.

Converting a Column's Data Type

df["score"] = df["score"].astype(int)   # convert a float column to integer
print(df.dtypes)

Interview tip: "How would you handle missing data in a real dataset?" is a near-guaranteed data-analysis interview question — the strongest answers don't just say "drop it" or "fill it" reflexively, but explain that the right strategy depends on how much data is missing, whether it is missing at random, and what the downstream analysis can tolerate.