Hypothesis tests and A/B testing

A new button raises the click rate from 2.0% to 2.3% in an experiment with 20,000 users per group. Is the button better, or is the difference noise? A hypothesis test turns the question into a calculation: assume there is no effect, compute how surprising the data would be under that assumption, and act if they are surprising enough. This note builds the test, the A/B test as its most common application, the power that sets how large an experiment must be, and the ways tests are misread and misused: the p-value taken for a probability that the hypothesis is true, peeking at running experiments, and testing many things at once.

Notation. $H_0$ is the null hypothesis (no effect), $H_1$ the alternative. $T$ is a test statistic, $\alpha$ the significance level, usually $0.05$. $\Phi$ is the standard normal CDF.

The logic of a test#

A test has four parts:

1. The null hypothesis $H_0$: the coin is fair, the button changes nothing.
2. A test statistic $T$ whose distribution under $H_0$ is known: the number of heads, a standardized difference.
3. The p-value: the probability, under $H_0$, of a statistic at least as extreme as the one observed.
4. The decision: reject $H_0$ if $p \le \alpha$.

A coin gives 60 heads in 100 flips. Under the fair-coin hypothesis the number of heads is $\mathrm{Bin}(100, 0.5)$, and a result at least as far from 50 in either direction has probability $0.057$: not below $0.05$, so the test does not reject. It does not conclude that the coin is fair either; it concludes that these 100 flips are not enough to decide.

Two kinds of error are possible. A type I error rejects a true $H_0$; its probability is at most $\alpha$ by construction. A type II error keeps a false $H_0$; one minus its probability is the power, the probability of detecting an effect that is there. The level is chosen; the power depends on the size of the effect and on the amount of data.

The A/B test#

Users are split at random into two groups, A sees the old button and B the new one, and the click rates $\hat p_A = 0.020$ and $\hat p_B = 0.023$ are compared. By the central limit theorem their difference is approximately normal, with standard error $\sqrt{p_A(1-p_A)/n + p_B(1-p_B)/n} = 0.00145$, and the test statistic

$$z = \frac{\hat p_B - \hat p_A}{\widehat{\operatorname{se}}} = 2.07$$

is approximately $N(0, 1)$ if the button changes nothing. The p-value is $P(\lvert Z \rvert \ge 2.07) = 0.039$:

Figure

The null distribution, the p-value and the power

Randomization is what makes the comparison causal: any difference between the groups other than the button is due to chance, which the test accounts for. Without it, users who see the new button might differ in other ways, and no test could separate the effects.

Power and sample size. If the true rates are 2.0% and 2.3%, the experiment with 20,000 users per group detects the difference with probability only $0.54$; nearly half of such experiments would miss a real effect. The number of users per group needed for power $1 - \beta$ at level $\alpha$ is

$$n = \frac{(z_{1-\alpha/2} + z_{1-\beta})^2 \big(p_A(1-p_A) + p_B(1-p_B)\big)}{(p_B - p_A)^2} ,$$

which for 80% power gives 36,691 users per group. Small effects on small rates need large experiments: the required $n$ grows like one over the square of the effect, so halving the detectable effect quadruples the experiment.

What a p-value is not#

• Not $P(H_0 \mid \text{data})$. Suppose that 10% of the ideas a team tests really work, tests have 80% power, and $\alpha = 0.05$. Of 1,000 ideas, 100 work and 80 of them are detected; 900 do not and 45 of them are significant by chance. About a third of the significant results, $45/125 = 0.36$, are false, although every one of them has $p < 0.05$. This is the base-rate calculation of the opening note, with tests instead of patients.
• Not the size of the effect. With enough data a negligible difference is significant; with too little a large one is not. Report the effect with a confidence interval, not only the p-value.
• Not evidence of no effect when large. $p = 0.3$ means the data are compatible with no effect, and usually also with a sizeable one.
• Not a measure of reproducibility. A result with $p = 0.04$ is replicated, at the same sample size and if the effect equals the observed one, only about half the time, because the observed effect is just above the threshold.

Peeking#

Checking a running experiment every day and stopping as soon as $p < 0.05$ feels harmless and is not. Each check is another chance for noise to cross the threshold:

Figure

Peeking at an A/A test

In A/A tests, where both groups are identical, 20 looks at the data produce a “significant” result in 25% of experiments instead of 5%. The fixes are a fixed sample size decided in advance, sequential tests whose thresholds account for repeated looks (alpha spending, always-valid p-values).

Multiple comparisons#

An experiment that reports 20 metrics, each tested at $\alpha = 0.05$, finds at least one false effect with probability $1 - 0.95^{20} = 0.64$ if none of the metrics is really affected and they are independent, as the axioms note computed. The Bonferroni correction tests each of the $m$ hypotheses at $\alpha/m$ and controls the probability of any false rejection through the union bound. When many hypotheses are tested and some false discoveries are acceptable, the Benjamini–Hochberg procedure controls the expected share of false discoveries among the rejections, the false discovery rate (FDR), instead of the probability of any false rejection, the family-wise error rate that Bonferroni controls, and keeps much more power. Its guarantee assumes valid p-values that are independent or positively dependent in a specific sense; for metrics with arbitrary dependence, the more conservative Benjamini–Yekutieli variant (Benjamini and Yekutieli, 2001) or a resampling method that keeps the dependence is needed. The same problem appears without anyone noticing when an analysis tries several segments, metrics or time windows and reports the one that worked.

Permutation tests#

The z-test relies on the central limit theorem. A permutation test relies only on the randomization itself: if the treatment does nothing, the group labels are arbitrary, and every reassignment of labels is equally likely. Compute the statistic for every way of splitting the data into groups of the original sizes, and the p-value is the share of splits with a statistic at least as extreme as the observed one. When the splits are too many to enumerate, draw $B$ random ones and use $p = (b + 1)/(B + 1)$, where $b$ counts the random splits at least as extreme as the observed one: counting the observed split itself keeps the test valid and never gives $p = 0$. For two groups of six measurements with a difference in means of $2.13$, 20 of the 924 splits give a difference at least as large in absolute value: $p = 0.022$, against $0.015$ from the t-test, which assumes normal data. Under the null hypothesis that the treatment has no effect at all, permutation tests are exact for any distribution and any statistic, which makes them the natural test for medians, AUC differences or any metric without a formula.

What changed#

	Before	With tests
“Is the difference real?”	a judgement	a p-value: how surprising the data are under no effect
Errors		type I at most $\alpha$; type II set by power
Experiment size	whatever traffic allows	$n$ for the desired power; $\propto 1/\text{effect}^2$
Reading results	$p < 0.05$ means true	a third of significant results can be false when few ideas work
Process	look whenever, report what worked	fixed horizon or sequential tests; corrections for multiple comparisons

Limitations#

• Statistical significance is not practical significance. A real effect can be too small to matter, and a large effect can be undetectable with the data at hand.
• Tests assume independent units and a stable setting. Users interacting with each other, novelty effects, or traffic that changes during the experiment violate the assumptions behind the p-value.
• The threshold 0.05 is a convention. It controls one error rate per test; decisions involve costs that a fixed threshold ignores.

What you should remember#

• p-value = $P(\text{a statistic at least this extreme} \mid H_0)$; reject at level $\alpha$ when $p \le \alpha$.
• Type I error at most $\alpha$; power = probability of detecting a real effect; plan $n$ for power, $n \propto 1/\text{effect}^2$.
• A/B test of proportions: $z = (\hat p_B - \hat p_A)/\widehat{\operatorname{se}}$; 2.0% vs 2.3% with 20,000 per group gives $z = 2.07$, $p = 0.039$, power 0.54.
• A p-value is not $P(H_0 \mid \text{data})$: with 10% true ideas, 80% power and $\alpha = 0.05$, 36% of significant results are false.
• Peeking and many metrics inflate false positives: 25% with 20 looks, 64% with 20 metrics; use fixed horizons, sequential tests, Bonferroni or Benjamini–Hochberg.
• Permutation tests: shuffle the labels; exact under randomization for any statistic.

Code#

The coin, the A/B test with its power and sample size, the false-discovery share, peeking, and a permutation test:

import numpy as np, math, itertools
from scipy import stats

# 60 heads in 100 flips: exact two-sided p-value
print(round(float(stats.binomtest(60, 100, 0.5).pvalue), 4))      # 0.0569

# the A/B test: 2.0% vs 2.3% with 20,000 users per group; power now, and users per group for 80% power
p1, p2, n = 0.02, 0.023, 20_000
se = math.sqrt(p1 * (1 - p1) / n + p2 * (1 - p2) / n)
z = (p2 - p1) / se
print(round(z, 2), round(float(2 * stats.norm.sf(z)), 3))         # 2.07 0.039
za, zb = stats.norm.ppf(0.975), stats.norm.ppf(0.8)
print(round(float(stats.norm.sf(za - z)), 2), math.ceil((za + zb) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) / (p2 - p1) ** 2))   # 0.54 36691

# false discoveries when 10% of tested ideas work, power 0.8, alpha 0.05
print(round(0.9 * 0.05 / (0.9 * 0.05 + 0.1 * 0.8), 2))            # 0.36

# peeking: 20,000 A/A tests checked after every 1,000 users per group, 20 checks
rng = np.random.default_rng(0)
a = rng.binomial(1000, 0.02, (20_000, 20)).cumsum(axis=1)
b = rng.binomial(1000, 0.02, (20_000, 20)).cumsum(axis=1)
m = 1000 * np.arange(1, 21)
pool = (a + b) / (2 * m)
zz = (b - a) / m / np.sqrt(2 * pool * (1 - pool) / m)
print(round(float((np.abs(zz) > 1.96).any(axis=1).mean()), 3), round(float((np.abs(zz[:, -1]) > 1.96).mean()), 3))   # 0.248 0.048

# a permutation test for two groups of six: all 924 splits
g1 = np.array([12.1, 14.3, 11.8, 15.2, 13.0, 12.7])
g2 = np.array([14.9, 16.2, 13.8, 15.5, 17.1, 14.4])
both, obs = np.concatenate([g1, g2]), g2.mean() - g1.mean()
diffs = []
for chosen in itertools.combinations(range(12), 6):
    mask = np.zeros(12, bool)
    mask[list(chosen)] = True
    diffs.append(both[mask].mean() - both[~mask].mean())
print(round(float(obs), 2), len(diffs), round(float((np.abs(diffs) >= abs(obs) - 1e-9).mean()), 4), round(float(stats.ttest_ind(g2, g1).pvalue), 4))
# 2.13 924 0.0216 0.0152

Next#

Statistics ends here; the last section of the course turns to random processes. So far every sample was independent. A sequence of words, a random walk on web pages and the states of a sampler all have memory: the next state depends on the current one. The next note introduces Markov chains, the simplest model of such memory, their stationary distributions, and the n-gram language model and PageRank as two of them.

References#

• J. Neyman, E. S. Pearson. On the Problem of the Most Efficient Tests of Statistical Hypotheses. Philosophical Transactions of the Royal Society A, 231, 1933.
• R. L. Wasserstein, N. A. Lazar. The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician, 70(2), 2016.
• R. Kohavi, D. Tang, Y. Xu. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020.
• Y. Benjamini, Y. Hochberg. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society B, 57(1), 1995.
• Y. Benjamini, D. Yekutieli. The Control of the False Discovery Rate in Multiple Testing under Dependency. The Annals of Statistics, 29(4), 2001.
• R. Johari, P. Koomen, L. Pekelis, D. Walsh. Peeking at A/B Tests: Why It Matters, and What to Do About It. KDD, 2017.
• L. Wasserman. All of Statistics. Springer, 2004. Chapter 10 (hypothesis testing and p-values).