DS Correlation

Correlation measures how strongly two numeric columns move together. Hours of study and exam marks tend to rise together. Outdoor temperature and heater use tend to move in opposite directions. A single number, from negative 1 to 1, captures the strength and direction of that link.

Read the Correlation Value

 -1.0          -0.5           0           +0.5          +1.0
  |-------------|-------------|-------------|-------------|
  perfect       moderate      no link       moderate      perfect
  opposite      opposite                    same way      same way

Value	Meaning	Example
Near +1	Both rise together	Study hours and marks
Near 0	No straight-line link	Shoe size and exam marks
Near -1	One rises as the other falls	Daily temperature and sweater sales

Rough guides call 0.7 and above strong, 0.4 to 0.7 moderate, and below 0.3 weak. Field norms vary, so treat these bands as hints.

Scatter Plot Pictures

 STRONG POSITIVE      NO LINK            STRONG NEGATIVE
 y                    y                  y
 |        . .         | .   .  .         | . .
 |      . .           |  .  .   .        |   . .
 |    . .             | .   . .          |     . .
 |  . .               |   .   .          |       . .
 +--------- x         +--------- x       +--------- x

Compute Correlation in Python

import pandas as pd

df = pd.DataFrame({
    "study_hours": [1, 2, 3, 4, 5, 6, 7, 8],
    "marks":       [35, 42, 50, 55, 63, 70, 74, 85],
    "tv_hours":    [6, 5, 5, 4, 3, 3, 2, 1]
})

print(df.corr())

The output shows a table. Study hours and marks score close to 1. TV hours and marks score close to negative 1. The diagonal always shows 1, since every column matches itself.

Heatmap of All Pairs

import seaborn as sns
import matplotlib.pyplot as plt

sns.heatmap(df.corr(), annot=True, cmap="coolwarm", vmin=-1, vmax=1)
plt.show()

Warm colors mark positive links, and cool colors mark negative links. Heatmaps let analysts scan dozens of columns in seconds.

Three Kinds of Correlation

Method	Measures	Best For
Pearson	Straight-line link	Numbers with a roughly bell shape
Spearman	Rank order link	Curved but steadily rising patterns, or outliers
Kendall	Agreement in ranks	Small datasets with many ties

df.corr(method="spearman")

Correlation Does Not Prove Cause

Two columns can move together without one causing the other. Ice cream sales and sunburn cases rise together in summer. Ice cream does not cause sunburn. Hot weather drives both. The hidden driver is called a confounder.

        Hot weather
        /         \
       v           v
 Ice cream sales   Sunburn cases

 Ice cream and sunburn correlate, yet neither causes the other.

Controlled experiments, such as A/B tests, give far stronger evidence of cause than correlation tables.

Covariance in One Paragraph

Covariance is a related measure of joint movement, though its size depends on the units of the columns. A covariance of 500 between income and spending tells little by itself. Correlation divides covariance by the spreads of both columns, which squeezes the answer into the range from negative 1 to 1 and makes comparisons fair.

Use Correlation in Projects

• Spot columns that strongly relate to the prediction target.
• Find pairs of nearly identical features, and drop one to reduce repetition.
• Check for odd links that hint at data errors or leakage.
• Use correlation as a starting clue, then test the idea with models.

Common Pitfalls

• Outliers can inflate or hide a correlation, so view the scatter plot first.
• Curved patterns can show a near-zero Pearson value despite a strong link.
• Small samples produce unstable values.
• Categories need other tools, such as the chi-square test.

Example: Choosing Columns for a Price Model

A real estate analyst studies 20 columns about houses. A correlation table shows that area, rooms, and bathrooms relate strongly to price. Area and rooms relate strongly to each other, so the analyst keeps area and drops rooms to reduce repetition. Distance from the city center shows a negative link with price. The analyst builds the first model with the strongest and least repeated columns, and the model trains faster with clearer results.

Key Points

• Correlation ranges from negative 1 to 1 and shows direction and strength.
• Pearson suits straight-line links, and Spearman suits rank-based links.
• A strong link never proves cause by itself.