Skip to main content
Editor

Pandas String and DateTime Operations for Real-World Data

Intermediate25 min6 exercises85 XP
0/6 exercises

Real-world data is messy. Names arrive in ALL CAPS, dates come as random text strings like "Jan 5, 2024" or "2024/01/05", and phone numbers have dashes in some rows but not others. Before you can analyze anything, you need to clean this stuff up.

Pandas has two powerful accessories for this job: the .str accessor for string operations and the .dt accessor for datetime operations. Think of them as specialized toolboxes that attach to any Series of strings or dates.

In this tutorial, you'll learn how to clean text data, extract patterns from strings, convert messy dates into proper datetime objects, and pull out components like year, month, and day of the week.

How Do You Clean Text with the .str Accessor?

When a column contains strings, you can use .str followed by any Python string method. The method runs on every value in the column at once -- no loops needed.

Basic string cleaning with .str
import pandas as pd
names = pd.Series([' Alice Smith ', 'BOB JONES', 'carol lee', 'Dave Brown'])
# Strip whitespace, convert to title case
cleaned = names.str.strip().str.title()
print(cleaned.tolist())
# Check string length
print(cleaned.str.len().tolist())
# Convert to lowercase
print(cleaned.str.lower().tolist())
Output
['Alice Smith', 'Bob Jones', 'Carol Lee', 'Dave Brown']
[11, 9, 9, 10]
['alice smith', 'bob jones', 'carol lee', 'dave brown']

You can chain .str methods together just like regular Python string methods. Each step produces a new Series, so the original data stays untouched.

How Do You Search and Replace in String Columns?

The .str.contains() method checks if each string matches a pattern. It returns a boolean Series that you can use as a filter. For replacing text, .str.replace() works like Python's str.replace() but on the whole column.

Searching and replacing in string columns
import pandas as pd
emails = pd.Series(['alice@gmail.com', 'bob@yahoo.com', 'carol@gmail.com', 'dave@outlook.com'])
# Find Gmail users
is_gmail = emails.str.contains('gmail')
print('Gmail users:', emails[is_gmail].tolist())
# Replace domain
updated = emails.str.replace('yahoo.com', 'yahoo.co.uk', regex=False)
print('Updated:', updated.tolist())
Output
Gmail users: ['alice@gmail.com', 'carol@gmail.com']
Updated: ['alice@gmail.com', 'bob@yahoo.co.uk', 'carol@gmail.com', 'dave@outlook.com']

How Do You Split Strings and Extract Parts?

.str.split() splits each string into a list. You can expand it into separate columns with expand=True. For more precise extraction, .str.extract() uses regex capture groups to pull out specific patterns.

Splitting strings and extracting patterns
import pandas as pd
full_names = pd.Series(['Alice Smith', 'Bob Jones', 'Carol Lee'])
# Split into first and last name columns
split_names = full_names.str.split(' ', expand=True)
split_names.columns = ['first', 'last']
print(split_names)
print()
# Extract just the first name using a regex group
first_only = full_names.str.extract(r'(\w+)')
print('First names:', first_only[0].tolist())
Output
   first   last
0  Alice  Smith
1    Bob  Jones
2  Carol    Lee

First names: ['Alice', 'Bob', 'Carol']
Extracting numbers with regex
import pandas as pd
# Extract numbers from messy text
data = pd.Series(['Order #1234', 'Order #5678', 'Order #9012'])
numbers = data.str.extract(r'#(\d+)')
print(numbers[0].tolist())
Output
['1234', '5678', '9012']

How Do You Convert Strings to Dates?

The pd.to_datetime() function is your go-to tool for turning messy date strings into proper datetime objects. It is surprisingly smart and can automatically parse many common formats.

Automatic date parsing with pd.to_datetime()
import pandas as pd
# Pandas can parse many date formats automatically
dates = pd.Series(['2024-01-15', 'Jan 20, 2024', '2024/02/28'])
parsed = pd.to_datetime(dates)
print(parsed)
print()
print('Types:', parsed.dtype)
Output
Traceback (most recent call last):
  File "<exec>", line 5, in <module>
  File "pandas/_libs/tslibs/strptime.pyx", line 501, in pandas._libs.tslibs.strptime.array_strptime
  File "pandas/_libs/tslibs/strptime.pyx", line 451, in pandas._libs.tslibs.strptime.array_strptime
  File "pandas/_libs/tslibs/strptime.pyx", line 583, in pandas._libs.tslibs.strptime._parse_with_format
ValueError: time data "Jan 20, 2024" doesn't match format "%Y-%m-%d", at position 1. You might want to try:
    - passing `format` if your strings have a consistent format;
    - passing `format='ISO8601'` if your strings are all ISO8601 but not necessarily in exactly the same format;
    - passing `format='mixed'`, and the format will be inferred for each element individually. You might want to use `dayfirst` alongside this.

When the automatic parser can't figure out your format, you can specify it explicitly using format=. This is also faster because Pandas doesn't have to guess.

Specifying a date format explicitly
import pandas as pd
# Explicit format for day-first dates
dates = pd.Series(['15-01-2024', '20-01-2024', '28-02-2024'])
parsed = pd.to_datetime(dates, format='%d-%m-%Y')
print(parsed.tolist())
Output
[Timestamp('2024-01-15 00:00:00'), Timestamp('2024-01-20 00:00:00'), Timestamp('2024-02-28 00:00:00')]

How Do You Extract Date Components?

Once you have a datetime column, the .dt accessor lets you pull out individual components like year, month, day, day of the week, and more. This is incredibly useful for time-based analysis.

Extracting date components with .dt
import pandas as pd
df = pd.DataFrame({
'order_date': pd.to_datetime(['2024-01-15', '2024-03-22', '2024-07-04', '2024-12-25']),
'amount': [100, 200, 150, 300]
})
# Extract components with .dt
df['year'] = df['order_date'].dt.year
df['month'] = df['order_date'].dt.month
df['day_name'] = df['order_date'].dt.day_name()
df['quarter'] = df['order_date'].dt.quarter
print(df.to_string(index=False))
Output
order_date  amount  year  month  day_name  quarter
2024-01-15     100  2024      1    Monday        1
2024-03-22     200  2024      3    Friday        1
2024-07-04     150  2024      7  Thursday        3
2024-12-25     300  2024     12 Wednesday        4

How Do You Do Math with Dates?

You can subtract dates to get a timedelta (duration), add timedeltas to shift dates, and create date ranges with pd.date_range(). This is the foundation of time series analysis.

Date math and date ranges
import pandas as pd
# Subtracting dates gives a timedelta
start = pd.to_datetime('2024-01-01')
end = pd.to_datetime('2024-03-15')
print('Duration:', end - start)
print('Days:', (end - start).days)
print()
# Adding a timedelta to shift dates
df = pd.DataFrame({
'order_date': pd.to_datetime(['2024-01-10', '2024-02-20', '2024-03-05'])
})
df['delivery_date'] = df['order_date'] + pd.Timedelta(days=5)
print(df.to_string(index=False))
print()
# Creating a date range
week = pd.date_range('2024-01-01', periods=7, freq='D')
print('One week:', week.tolist())
Output
Duration: 74 days 00:00:00
Days: 74

order_date delivery_date
2024-01-10    2024-01-15
2024-02-20    2024-02-25
2024-03-05    2024-03-10

One week: [Timestamp('2024-01-01 00:00:00'), Timestamp('2024-01-02 00:00:00'), Timestamp('2024-01-03 00:00:00'), Timestamp('2024-01-04 00:00:00'), Timestamp('2024-01-05 00:00:00'), Timestamp('2024-01-06 00:00:00'), Timestamp('2024-01-07 00:00:00')]

Practice Exercises

Clean Messy Names
Write Code

Clean the names Series by stripping whitespace and converting to title case. Print the cleaned names as a list.

import pandas as pd
names = pd.Series([' john doe ', 'JANE SMITH', ' bob brown'])
# Strip whitespace, convert to title case, print as list
Filter with str.contains()
Write Code

Filter the DataFrame to keep only rows where the email column contains "gmail". Print the name column of the filtered rows as a list.

import pandas as pd
df = pd.DataFrame({
'name': ['Alice', 'Bob', 'Carol', 'Dave'],
'email': ['alice@gmail.com', 'bob@yahoo.com', 'carol@gmail.com', 'dave@outlook.com']
})
# Filter for Gmail users and print their names
Predict the String Operation Output
Predict Output

What will this code print?

import pandas as pd
s = pd.Series(['hello world', 'foo bar', 'python pandas'])
print(s.str.split(' ').str.get(0).str.upper().tolist())
# Type the exact output
Parse Dates and Extract Month
Write Code

Convert the date_str column to datetime, then add a month column with the month number. Print the month column as a list.

import pandas as pd
df = pd.DataFrame({
'event': ['Launch', 'Update', 'Review'],
'date_str': ['2024-03-15', '2024-07-22', '2024-11-01']
})
# Convert date_str to datetime, extract month, print as list
Calculate Days Between Dates
Write Code

Add a column days_to_deliver that shows how many days passed between order_date and delivery_date. Print the column as a list of integers.

import pandas as pd
df = pd.DataFrame({
'order_date': pd.to_datetime(['2024-01-01', '2024-02-10', '2024-03-20']),
'delivery_date': pd.to_datetime(['2024-01-04', '2024-02-15', '2024-03-22'])
})
# Calculate days between order and delivery
Fix the String Extraction Bug
Fix the Bug

This code tries to extract area codes from phone numbers, but the regex pattern is wrong. Fix the str.extract() call so it correctly pulls out the 3-digit area code from the parentheses.

Print the extracted area codes as a list.

import pandas as pd
phones = pd.Series(['(212) 555-1234', '(415) 555-5678', '(310) 555-9012'])
# Bug: wrong regex pattern
area_codes = phones.str.extract(r'(\d+)')
print(area_codes[0].tolist())

Harder problems

More problems on this topic: Working with data

Related Tutorials

Was this tutorial helpful?