DS Outlier Detection

An outlier is a value that sits far away from the rest of the data. A shop with daily sales near 200 units suddenly records 9000. A student scores 15 out of 100 while classmates score between 60 and 90. Outliers can signal typing errors, broken sensors, or rare but real events, and they can bend averages and confuse models.

Imagine measuring the height of ten friends and finding one entry of 17 meters. Everyone senses the mistake. Detection methods give that same instinct a clear rule.

Two Kinds of Outliers

Kind	Cause	Action
Error	Typing slip, faulty sensor, wrong unit	Fix or remove
Genuine extreme	Real rare event, such as a festival sale spike	Keep and study

Deleting every outlier hides useful stories. Investigate first, and decide after.

Method 1: Look at a Box Plot

  0   10   20   30   40   50   60   70   80   90  100
  |----|----|----|----|----|----|----|----|----|----|
          [=====|=====]                          *
          Q1  median  Q3                      outlier
        |-------- whiskers --------|

The box spans the middle half of the data. Dots beyond the whiskers mark suspects. A quick plot often reveals problems faster than any formula.

import seaborn as sns
sns.boxplot(x=df["amount"])

Method 2: The IQR Rule

IQR stands for interquartile range, the distance between the 25th percentile (Q1) and the 75th percentile (Q3). The rule flags values more than 1.5 times the IQR below Q1 or above Q3.

q1 = df["amount"].quantile(0.25)
q3 = df["amount"].quantile(0.75)
iqr = q3 - q1

low = q1 - 1.5 * iqr
high = q3 + 1.5 * iqr

outliers = df[(df["amount"] < low) | (df["amount"] > high)]
print(len(outliers))

This method works well for skewed data because it relies on percentiles instead of averages.

Method 3: The Z-Score Rule

A z-score tells how many standard deviations a value sits from the mean. Values beyond 3 or below negative 3 usually count as outliers.

z = (df["amount"] - df["amount"].mean()) / df["amount"].std()
outliers = df[z.abs() > 3]

Z-scores suit bell-shaped data. Extreme values inflate the mean and the standard deviation, which can hide the very outliers you want to catch.

Method 4: Isolation Forest

Isolation Forest is a model that finds odd points across many columns at once. It builds random trees and measures how quickly each point gets separated from the others. Outliers separate in very few steps.

from sklearn.ensemble import IsolationForest

model = IsolationForest(contamination=0.02, random_state=0)
df["flag"] = model.fit_predict(df[["amount", "quantity"]])
print(df[df["flag"] == -1])

The contamination setting estimates the share of outliers. A label of negative 1 marks a suspect row.

Choose a Method

Situation	Best Method
Quick visual check of one column	Box plot
Skewed numbers	IQR rule
Bell-shaped numbers	Z-score
Many columns together	Isolation Forest

Ways to Handle Outliers

Remove

Drop rows that clearly come from errors.

Cap

Replace extreme values with the nearest limit. This step is called winsorizing.

df["amount"] = df["amount"].clip(lower=low, upper=high)

Transform

Apply a log transform to shrink large values.

Keep and model

Use models that resist outliers, such as random forests, and keep the rare events in the data.

Example: Daily Cafe Sales

A cafe records daily sales of 180, 195, 210, 205, 190, 8800, and 200. The value 8800 stands out. A check with the owner reveals a typing slip, and the real figure is 188. The corrected entry restores the weekly average to a sensible level. A box plot, the IQR rule, and a simple call to the owner each reach the same conclusion.

Key Points

• Outliers are values far from the pattern, and they may be errors or real events.
• Box plots, the IQR rule, z-scores, and Isolation Forest find them.
• Investigate before deleting, and record every decision.