DS Missing Values

Missing values are blank spots in a dataset. A customer skipped the age field. A sensor stopped reporting for an hour. A clerk forgot to type a price. Most real datasets contain gaps, and models cannot learn well from gaps without careful handling.

Imagine a class attendance sheet with some boxes left empty. You cannot tell whether those students were absent or the teacher forgot to mark them. Data scientists face that same puzzle.

Why Values Go Missing

Reason	Plain Meaning	Example
Random	Gaps appear by chance	A network glitch drops a few records
Related to other columns	Gaps depend on something you can see	Younger users skip the income field
Related to the value itself	The hidden value causes the gap	High earners refuse to share salary

The reason guides the fix. Random gaps are safe to fill with simple methods. Gaps tied to the value itself need more care because they can bend results.

Step 1: Detect Missing Values

import pandas as pd

df = pd.read_csv("customers.csv")
print(df.isnull().sum())
print(df.isnull().mean() * 100)

The first line counts blanks in each column. The second line shows the percentage of blanks. A column with 70 percent blanks often deserves removal.

Step 2: Choose a Strategy

Is the column mostly empty?
   |-- Yes --> Drop the column
   |-- No  --> Is the data a number?
                 |-- Yes --> Are there extreme values?
                 |             |-- Yes --> Fill with median
                 |             |-- No  --> Fill with mean
                 |-- No  --> Fill with most common value or "Unknown"

Strategy A: Drop Rows or Columns

df_rows = df.dropna()
df_cols = df.drop(columns=["fax_number"])

Dropping works when few rows are affected. Dropping too many rows throws away useful information.

Strategy B: Fill with a Summary Value

df["age"] = df["age"].fillna(df["age"].median())
df["city"] = df["city"].fillna("Unknown")

Mean versus median

The mean is the usual average. The median is the middle value after sorting. A few very large salaries pull the mean upward, and the median stays steady. Choose the median when the column has extreme values.

Strategy C: Fill Using Neighbors in Time

Time-based data such as temperature readings often changes slowly. Copy the previous reading forward to fill a gap.

df["temp"] = df["temp"].ffill()
df["temp"] = df["temp"].interpolate()

The interpolate method draws a straight line between the known points on each side of the gap.

Strategy D: Add a Flag Column

Sometimes the fact that a value is missing carries meaning. A blank in the phone number column may signal a customer who prefers email. Keep that signal by adding a flag before filling.

df["phone_missing"] = df["phone"].isnull().astype(int)
df["phone"] = df["phone"].fillna("None")

Example: Fixing a Small Table

Name	Age	City
Ravi	29	Pune
Meena	(blank)	Delhi
Kabir	41	(blank)
Sara	35	Pune

The ages known are 29, 41, and 35, so the median is 35. The city column has Pune as its most common value. After filling, Meena receives age 35, and Kabir receives city Pune. A flag column records which cells were filled.

Best Practices

• Study the pattern of gaps before choosing a fix.
• Fill after splitting data into training and test sets, using only training values to compute means and medians.
• Record every cleaning choice in notebook notes.
• Compare model results across two strategies when unsure.

Advanced Option: Model-Based Filling

Simple fills ignore the other columns. Model-based fills use them. The KNN imputer looks at the most similar rows and borrows their values. A missing age for a customer with a student email and a small basket may receive the typical age of similar customers.

from sklearn.impute import KNNImputer

imputer = KNNImputer(n_neighbors=5)
filled = imputer.fit_transform(df[["age", "income", "orders"]])

Test both a simple fill and a model-based fill. Keep the simpler one when scores stay similar, since simple methods are easier to explain and maintain.

Key Points

• Detect gaps with isnull, and measure them as percentages.
• Drop, fill, interpolate, or flag, depending on the column.
• The reason behind a gap guides the best fix.