DS Hypothesis Testing
A hypothesis is a claim that data can support or challenge. Hypothesis testing gives a fair method to decide whether a result reflects a real effect or only random luck. Teams use it to judge new website designs, new teaching methods, and new factory settings.
Think of a court trial. The defendant counts as innocent until the evidence proves otherwise. Hypothesis testing follows the same rule: assume nothing changed until the data shows strong proof.
The Two Hypotheses
| Name | Symbol | Meaning | Example |
|---|---|---|---|
| Null hypothesis | H0 | No real effect exists | The new button color does not change sign-ups |
| Alternative hypothesis | H1 | A real effect exists | The new button color changes sign-ups |
The Testing Flow
State H0 and H1
|
Choose a significance level (often 0.05)
|
Collect data and run a test
|
Get the p-value
|
p < 0.05 ?
/ \
Yes No
| |
Reject H0 Keep H0
(effect (not enough
is likely) evidence)
Understand the P-Value
The p-value is the chance of seeing results at least as extreme as yours if the null hypothesis were true. A small p-value says the result would be rare under pure luck. A p-value of 0.03 means that luck alone would produce such a result about 3 times in 100.
Significance level
The significance level, written alpha, is the line you draw before the test. Many fields use 0.05. A p-value below alpha counts as statistically significant.
Common misunderstanding
A p-value does not give the chance that the null hypothesis is true. The value says nothing about the size or importance of the effect. A huge dataset can make tiny, useless differences look significant.
Example: Plant Fertilizer Test
A gardener tests a new fertilizer. Ten plants receive the new fertilizer, and ten receive the old one. After four weeks the gardener measures heights in centimeters.
from scipy import stats
new = [32, 35, 31, 36, 34, 33, 37, 35, 34, 36]
old = [30, 31, 29, 32, 30, 31, 33, 30, 29, 31]
t_stat, p_value = stats.ttest_ind(new, old)
print("t:", round(t_stat, 2))
print("p:", round(p_value, 4))
if p_value < 0.05:
print("Reject H0: the fertilizers differ")
else:
print("Keep H0: no clear difference")The p-value comes out far below 0.05, so the gardener rejects the null hypothesis. The new fertilizer likely produces taller plants.
Popular Tests and When to Use Them
| Test | Use It To | Example Question |
|---|---|---|
| One-sample t-test | Compare one group mean with a known value | Does our average delivery time equal 30 minutes? |
| Two-sample t-test | Compare means of two groups | Do two classes score differently? |
| Paired t-test | Compare before and after on the same items | Did page load time drop after an update? |
| Chi-square test | Check links between categories | Does device type relate to purchase choice? |
| ANOVA | Compare three or more group means | Do three ad designs produce different clicks? |
Two Kinds of Errors
| Error | What Happens | Everyday Picture |
|---|---|---|
| Type I (false alarm) | You reject H0 when it is true | A smoke alarm rings with no fire |
| Type II (missed signal) | You keep H0 when it is false | A smoke alarm stays silent during a fire |
Lowering alpha reduces false alarms and raises the risk of missed signals. Larger samples reduce both risks.
Confidence Intervals
A confidence interval gives a range of likely values for the true effect. A 95 percent interval for the height difference might read 2.1 to 4.3 centimeters. The range shows both the direction and the size of the effect, so reports often include it beside the p-value.
Good Practice
- Write hypotheses and alpha before looking at results.
- Plan sample size in advance.
- Avoid running many tests and reporting only the lucky one.
- Report effect size along with the p-value.
- Check test assumptions, such as roughly bell-shaped data for t-tests.
Key Points
- The null hypothesis assumes no effect, and the data must challenge it.
- A small p-value signals that luck alone is an unlikely explanation.
- Pick the test that matches the data type and the number of groups.