DS A/B Testing

An A/B test compares two versions of something to learn which performs better. Version A is the current design, and version B carries one change. Visitors are split at random between the two, and a chosen metric decides the winner. Websites, apps, emails, and ads rely on this method to make decisions with evidence instead of guesses.

Picture a bakery testing two shop-window signs. Half the days show sign A, and half show sign B. The sign that brings more customers wins.

The Test Layout

                 All visitors
                      |
              Random split 50/50
              /               \
        Group A               Group B
     (old button)          (new button)
          |                      |
     Measure sign-ups      Measure sign-ups
          \                      /
           Compare with a statistical test
                      |
              Keep A, ship B, or run longer

Key Terms

Term	Meaning
Control	The current version (A)
Variant	The changed version (B)
Metric	The number you want to improve, such as sign-up rate
Conversion rate	Visitors who finish a goal divided by all visitors
Lift	Relative improvement of B over A
Sample size	Visitors needed in each group

Step 1: Write a Clear Hypothesis

State the change, the metric, and the expected result. A good example reads: changing the sign-up button from grey to green will raise the sign-up rate. Test one change at a time, so the result points to a single cause.

Step 2: Pick the Metric and the Guardrails

The main metric measures success, such as sign-up rate. Guardrail metrics protect the experience. If a new layout lifts sign-ups but doubles page load time, a guardrail on speed raises a warning.

Step 3: Plan the Sample Size

Small tests produce noisy results. Three inputs drive the needed sample size: the current rate, the smallest lift worth detecting, and the acceptable error levels.

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

effect = proportion_effectsize(0.10, 0.12)
n = NormalIndPower().solve_power(effect_size=effect, alpha=0.05, power=0.8)
print(int(n))

This code estimates visitors needed per group to detect a rise from 10 percent to 12 percent. Decide the sample size before the test begins, and stop only when the plan finishes.

Step 4: Randomize Fairly

Random assignment makes the two groups alike in every way except the change. Assign users by a stable user ID, so each person always sees the same version. Run both versions at the same time to avoid effects from weekdays, holidays, or news events.

Step 5: Analyze the Result

from statsmodels.stats.proportion import proportions_ztest

signups = [480, 540]
visitors = [5000, 5000]

stat, p = proportions_ztest(signups, visitors)
print("A rate:", signups[0] / visitors[0])
print("B rate:", signups[1] / visitors[1])
print("p-value:", round(p, 4))

Version A converts at 9.6 percent, and version B converts at 10.8 percent. The p-value falls below 0.05, so the gain likely reflects a real effect. A confidence interval around the lift shows the plausible range, such as 0.2 to 2.2 percentage points.

Step 6: Decide and Record

Result	Action
B wins clearly, guardrails fine	Roll out B
No clear difference	Keep A, or test a bolder change
B loses	Keep A and learn why
Mixed signals	Run longer or segment the results

Write the hypothesis, dates, sample sizes, results, and decision in a shared log. Future teams will avoid repeating tests.

Common Mistakes

• Peeking: checking results daily and stopping when the p-value dips below 0.05 inflates false wins.
• Too many variants: each extra version raises the chance of a lucky result.
• Tiny samples: small groups hide real effects.
• Changing several things at once: the cause of any lift stays unclear.
• Ignoring the novelty effect: new designs attract curiosity clicks that fade.

Ethics and Privacy

Test changes that help users, and avoid tricks that mislead them. Collect only needed data, follow privacy laws, and honor consent choices. Tell users how their data supports product improvements in the privacy policy.

Key Points

• A/B tests split users at random and compare a metric between two versions.
• Plan sample size first, test one change, and avoid peeking.
• Statistical tests and confidence intervals support the final decision.