A/B Testing
A/B testing is a method of comparing two versions of something — a button, a headline, a feature, or an entire page — to see which version performs better. Instead of debating which version is better in a meeting, the team shows both versions to real users and lets the data decide.
The Core Concept
Split your users into two groups at random. Show Group A the original version (control). Show Group B the new version (variant). Measure which group achieves the desired outcome more often. The winning version is the one you keep.
All Users
↓ split randomly
Group A (50%) Group B (50%)
[Original Button] [New Green Button]
"Submit" "Get Started Free"
↓ ↓
Conversion: 4.2% Conversion: 6.8%
Winner: Group B — "Get Started Free" green button
Why A/B Testing Beats Opinions
In most product discussions, the loudest voice wins. The CEO likes blue. The designer prefers white. The PM thinks red converts better. Nobody knows. A/B testing replaces this debate with actual evidence from real users making real decisions.
What You Can A/B Test
- Button text ("Buy Now" vs. "Add to Cart")
- Button color or placement on a page
- Headline or description copy
- Onboarding flow (4 steps vs. 6 steps)
- Pricing page layout
- Email subject lines
- Feature placement in the navigation
- Homepage design
The A/B Testing Process
Step 1: Identify an opportunity
Pick a metric you want to improve (e.g., signup rate)
↓
Step 2: Form a hypothesis
"If we change X to Y, we expect Z to improve because..."
↓
Step 3: Build the variant
Create the new version (B) while keeping the control (A)
↓
Step 4: Calculate sample size needed
Use a sample size calculator to know how many users you need
↓
Step 5: Run the test
Expose both versions to users simultaneously
↓
Step 6: Analyze results
Wait for statistical significance before reading results
↓
Step 7: Decide and ship
Roll out the winner to 100% of users
Statistical Significance Explained Simply
Statistical significance tells you how confident you can be that the result is real and not just random chance. PMs typically aim for 95% statistical significance before calling a test result conclusive.
Think of flipping a coin. If you flip 3 times and get 3 heads, you might think the coin is rigged. But if you flip 1,000 times and get 600 heads, you are very confident the coin is biased. The same logic applies to A/B tests — more data gives more confidence.
Confidence Level: Below 90%: Results are unreliable. Run longer. 90%–95%: Fairly confident. Reasonable to act. Above 95%: Statistically significant. Confident to decide.
Common A/B Testing Mistakes
| Mistake | What Happens | Fix |
|---|---|---|
| Stopping the test too early | You see early results and think you know the winner | Wait for statistical significance |
| Testing too many things at once | You don't know which change caused the result | Change only one variable per test |
| Small sample size | Results are unreliable | Calculate needed sample size before starting |
| Running during unusual events | Holiday traffic skews results | Run tests during typical traffic periods |
| Not tracking the right metric | Button clicks go up but purchases don't change | Tie the test to a business outcome, not a surface metric |
A/B Testing vs. Multivariate Testing
A/B testing compares two versions. Multivariate testing compares multiple variables simultaneously.
A/B Test: Version A: Blue button, text "Submit" Version B: Green button, text "Submit" → Tests button color Multivariate Test: Combo 1: Blue button + "Submit" Combo 2: Blue button + "Get Started" Combo 3: Green button + "Submit" Combo 4: Green button + "Get Started" → Tests color AND text simultaneously
Multivariate testing requires significantly more traffic to reach statistical significance. For most products, A/B testing is the right starting point.
Reading an A/B Test Report
When reviewing test results, check these four things:
- Conversion rate for each variant — Which version converted more users?
- Relative lift — How much better is the variant compared to control? (e.g., 6.8% vs. 4.2% = 62% relative lift)
- Statistical confidence — Is the result above 95%?
- Sample size — Did enough users see each version for the result to be trustworthy?
Key Takeaway
A/B testing removes opinion from product decisions. It gives teams a scientific way to improve the product continuously based on real user behavior. PMs who run regular A/B tests compound small improvements into significant product gains over time — often without adding a single new feature.