Skip to main content
Editor

Pandas Data Cleaning: Missing Values, Duplicates, and Outliers

Intermediate30 min7 exercises105 XP
0/7 exercises

You just got a dataset from a survey. Some people skipped questions, one person typed "N/A" instead of leaving a field blank, there are three duplicate entries, and someone claims to be 900 years old. Welcome to real-world data.

Data scientists spend roughly 80% of their time cleaning data and only 20% actually analyzing it. The good news: pandas has built-in tools for every common cleaning task. In this tutorial, you'll learn how to detect and handle missing values, remove duplicates, fix data types, and deal with outliers.

How Do You Find Missing Values?

In pandas, missing data is represented as NaN (Not a Number). You can also create it explicitly with None or float('nan'). The key methods are isna() (returns True where values are missing) and notna() (returns True where values exist).

Counting missing values
import pandas as pd
import numpy as np
df = pd.DataFrame({
'name': ['Alice', 'Bob', None, 'Diana'],
'age': [28, None, 22, 45],
'city': ['NYC', 'LA', 'NYC', None]
})
# Count missing values per column
print(df.isna().sum())
print()
print('Total missing:', df.isna().sum().sum())
Output
name    1
age     1
city    1
dtype: int64

Total missing: 3

Should You Drop or Fill Missing Values?

dropna() removes rows (or columns) that contain missing values. Use it when missing data is rare and dropping rows won't bias your results.

dropna() options
import pandas as pd
df = pd.DataFrame({
'name': ['Alice', 'Bob', None, 'Diana'],
'age': [28, None, 22, 45],
'score': [90, 85, None, 78]
})
# Drop rows where ANY value is missing
cleaned = df.dropna()
print('After dropna():')
print(cleaned['name'].tolist())
print()
# Drop rows where a SPECIFIC column is missing
cleaned2 = df.dropna(subset=['name'])
print('After dropna(subset=[name]):')
print(cleaned2['name'].tolist())
Output
After dropna():
['Alice', 'Diana']

After dropna(subset=[name]):
['Alice', 'Bob', 'Diana']

How Do You Fill in Missing Values?

fillna() replaces missing values with a value you choose. Common strategies: fill with zero, fill with the column mean, or fill with the most recent value (forward fill).

fillna() strategies
import pandas as pd
df = pd.DataFrame({
'product': ['A', 'B', 'C', 'D'],
'price': [10.0, None, 25.0, None],
'category': ['food', None, 'food', 'drink']
})
# Fill numeric with mean
df['price'] = df['price'].fillna(df['price'].mean())
# Fill text with a placeholder
df['category'] = df['category'].fillna('unknown')
print(df['price'].tolist())
print(df['category'].tolist())
Output
[10.0, 17.5, 25.0, 17.5]
['food', 'unknown', 'food', 'drink']

How Do You Handle Duplicate Rows?

Duplicate rows sneak in from data entry errors, merging datasets, or buggy ETL pipelines. duplicated() flags them and drop_duplicates() removes them.

Finding and removing duplicates
import pandas as pd
df = pd.DataFrame({
'name': ['Alice', 'Bob', 'Alice', 'Charlie', 'Bob'],
'score': [90, 85, 90, 78, 85]
})
# Find duplicates
print('Duplicated rows:')
print(df.duplicated().tolist())
print()
# Remove duplicates
cleaned = df.drop_duplicates()
print('After removing duplicates:')
print(cleaned['name'].tolist())
Output
Duplicated rows:
[False, False, True, False, True]

After removing duplicates:
['Alice', 'Bob', 'Charlie']

You can also check for duplicates based on specific columns. For example, if a customer name should be unique, you can pass subset=['name'] to check only that column.

Dedup by specific columns
import pandas as pd
df = pd.DataFrame({
'email': ['a@test.com', 'b@test.com', 'a@test.com', 'c@test.com'],
'name': ['Alice', 'Bob', 'Alice Updated', 'Charlie']
})
# Keep last occurrence of duplicate emails
cleaned = df.drop_duplicates(subset=['email'], keep='last')
print(cleaned[['email', 'name']].values.tolist())
Output
[['b@test.com', 'Bob'], ['a@test.com', 'Alice Updated'], ['c@test.com', 'Charlie']]

How Do You Fix Inconsistent Data?

Real datasets are messy. Someone typed "Yes" and someone else typed "yes" and a third person typed "Y". The replace() method standardizes these inconsistencies. And astype() fixes wrong data types.

replace() and astype()
import pandas as pd
df = pd.DataFrame({
'response': ['Yes', 'yes', 'Y', 'No', 'no', 'N'],
'age': ['28', '34', '22', '45', '31', '29']
})
# Standardize responses
df['response'] = df['response'].replace({
'Yes': 'yes', 'Y': 'yes',
'No': 'no', 'N': 'no'
})
# Convert age from string to integer
df['age'] = df['age'].astype(int)
print(df['response'].tolist())
print(df['age'].dtype)
Output
['yes', 'yes', 'yes', 'no', 'no', 'no']
int32

How Do You Rename Columns?

Column names from raw data are often ugly - spaces, mixed case, abbreviations. Clean column names make your code readable.

Renaming columns
import pandas as pd
df = pd.DataFrame({
'First Name': ['Alice', 'Bob'],
'SALARY (USD)': [70000, 85000]
})
df = df.rename(columns={
'First Name': 'first_name',
'SALARY (USD)': 'salary_usd'
})
print(df.columns.tolist())
Output
['first_name', 'salary_usd']

How Do You Spot Outliers?

An outlier is a data point that's far outside the normal range. A 900-year-old person or a $-500 salary. One common approach is to flag values beyond 2 or 3 standard deviations from the mean, or use the IQR (interquartile range) method.

IQR method for outlier detection
import pandas as pd
df = pd.DataFrame({
'name': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'],
'age': [28, 34, 22, 900, 30]
})
# IQR method to detect outliers
Q1 = df['age'].quantile(0.25)
Q3 = df['age'].quantile(0.75)
IQR = Q3 - Q1
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR
outliers = df[(df['age'] < lower) | (df['age'] > upper)]
print('Outliers:')
print(outliers['name'].tolist())
Output
Outliers:
['Diana']

How Do You Chain Cleaning Steps Together?

In practice, you rarely apply just one cleaning operation. A typical pipeline drops duplicates, fills missing values, fixes types, and renames columns - all in sequence. Pandas supports method chaining so you can do this in a readable flow.

Chained cleaning pipeline
import pandas as pd
df = pd.DataFrame({
'Name': ['Alice', 'Bob', 'Alice', 'Charlie'],
'Age': ['28', '34', '28', None],
'Score': [90.0, None, 90.0, 78.0]
})
cleaned = (
df
.drop_duplicates()
.rename(columns={'Name': 'name', 'Age': 'age', 'Score': 'score'})
.assign(score=lambda x: x['score'].fillna(x['score'].mean()))
)
print(cleaned['name'].tolist())
print(cleaned['score'].tolist())
Output
['Alice', 'Bob', 'Charlie']
[90.0, 84.0, 78.0]

Quick Recap: Data Cleaning Checklist

Every time you get a new dataset, run through this checklist:

  • Missing values: df.isna().sum() - How much is missing? Drop or fill?
  • Duplicates: df.duplicated().sum() - Any exact duplicates?
  • Data types: df.dtypes - Are numbers stored as strings?
  • Inconsistencies: df['col'].unique() - Are there variations like Yes/yes/Y?
  • Outliers: df.describe() - Do the min/max values make sense?
  • Column names: Are they clean, lowercase, snake_case?

  • Practice Exercises

    Count Missing Values
    Write Code

    Count the total number of missing values across the entire DataFrame and print the result as an integer.

    import pandas as pd
    df = pd.DataFrame({
    'name': ['Alice', None, 'Charlie', 'Diana'],
    'age': [28, 34, None, 45],
    'city': [None, 'LA', 'NYC', None]
    })
    # Count total missing values
    Fill Missing Scores with the Mean
    Write Code

    Fill missing values in the 'score' column with the column's mean. Print the score column as a list. The mean should be computed before filling.

    import pandas as pd
    df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie', 'Diana'],
    'score': [80.0, None, 90.0, None]
    })
    # Fill missing scores with the mean
    Drop Rows Missing a Name
    Write Code

    Drop only the rows where the 'name' column is missing (don't drop rows that have missing values in other columns). Print the remaining names as a list.

    import pandas as pd
    df = pd.DataFrame({
    'name': ['Alice', None, 'Charlie', None, 'Eve'],
    'age': [28, 34, None, 45, 30]
    })
    # Drop rows where name is missing
    Remove Duplicate Emails
    Write Code

    Remove rows with duplicate 'email' values, keeping the first occurrence. Print the remaining emails as a list.

    import pandas as pd
    df = pd.DataFrame({
    'email': ['a@test.com', 'b@test.com', 'a@test.com', 'c@test.com', 'b@test.com'],
    'name': ['Alice', 'Bob', 'Alice2', 'Charlie', 'Bob2']
    })
    # Remove duplicate emails, keep first
    Standardize Yes/No Responses
    Write Code

    Standardize the 'answer' column so that all variations of yes ('Yes', 'Y', 'y') become 'yes' and all variations of no ('No', 'N', 'n') become 'no'. Print the cleaned answers as a list.

    import pandas as pd
    df = pd.DataFrame({
    'answer': ['Yes', 'N', 'y', 'No', 'Y', 'n']
    })
    # Standardize the answer column
    Predict the Output: astype
    Predict Output

    What will this code print?

    import pandas as pd
    df = pd.DataFrame({'val': ['1', '2', '3']})
    df['val'] = df['val'].astype(int)
    print(df['val'].sum())
    import pandas as pd
    df = pd.DataFrame({'val': ['1', '2', '3']})
    df['val'] = df['val'].astype(int)
    print(df['val'].sum())
    Remove Outlier Ages
    Write Code

    Remove rows where the 'age' column has values less than 0 or greater than 120 (impossible ages). Print the remaining ages as a list.

    import pandas as pd
    df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'],
    'age': [28, -5, 34, 900, 45]
    })
    # Remove impossible ages (< 0 or > 120)

    Harder problems

    More problems on this topic: Working with data

    Related Tutorials

    Was this tutorial helpful?