Confidence intervals and the bootstrap

An estimate without an error bar invites over-interpretation: a model at 85.9% accuracy beats one at 83.3%, unless the test set is small. The Bayesian inference note answered with a credible interval, a probability statement about the parameter that needs a prior. The frequentist answer is a confidence interval: a procedure that, applied to repeated samples, captures the true value a stated fraction of the time. This note explains what that fraction means, builds intervals from the central limit theorem, and introduces the bootstrap, which builds them from the data alone when no formula is available.

Notation. $\theta$ is the unknown parameter, $\hat\theta$ its estimate, $\operatorname{se}$ its standard error. $z_{0.975} = 1.96$ is the 97.5% quantile of $N(0, 1)$ and $t_{k, 0.975}$ that of Student’s $t$ distribution with $k$ degrees of freedom.

What 95% means#

A 95% confidence interval is a pair of statistics $[L, U]$, functions of the data, such that

$$P_\theta\big(L \le \theta \le U\big) \ge 0.95 \quad \text{for every } \theta .$$

An interval with this guarantee, for every $\theta$ and every sample size, is exact, or conservative if the coverage can exceed 95%. Most intervals in practice, including the normal-approximation, Wilson and bootstrap intervals below, are only nominal 95% intervals: their coverage approaches 95% as $n$ grows, under the conditions of the approximation, but for a given $n$ and $\theta$ it can be lower or higher. A simulated coverage at one parameter value checks the interval there, not for every $\theta$.

The probability is over the data: the interval is random and the parameter is fixed. Over many repetitions of the experiment, the share of intervals that contain the true value approaches their coverage, here exactly 95% for the t-interval of normal data:

Figure

A hundred confidence intervals for the same mean

Once the data are in, a particular interval either contains $\theta$ or does not; the statement ”$\theta$ lies in $[4.1, 6.3]$ with probability 0.95” is the Bayesian reading, which a confidence interval does not license. In practice, with large samples and weak priors, the two kinds of interval nearly coincide, as the Bayesian note showed; they differ when data are scarce or prior knowledge is strong.

Intervals from the normal approximation#

When an estimator is approximately normal, $\hat\theta \approx N(\theta, \operatorname{se}^2)$, the interval

$$\hat\theta \pm 1.96\, \widehat{\operatorname{se}}$$

has coverage close to 95%. For a mean the standard error is $\sigma/\sqrt n$, and $\sigma$ must be estimated by the sample standard deviation $s$. With few observations that estimate is itself noisy, and the interval must be widened: if the data are normal, $(\bar x - \mu)/(s/\sqrt n)$ has Student’s $t$ distribution with $n - 1$ degrees of freedom, and the interval

$$\bar x \pm t_{n-1,\, 0.975}\, \frac{s}{\sqrt n}$$

has exactly 95% coverage. For $n = 10$ the multiplier is $2.262$ instead of $1.96$; with $1.96$ the coverage of such intervals drops to 92%. The difference vanishes as $n$ grows: $t_{99, 0.975} = 1.984$.

Proportions. The textbook interval for a proportion, $\hat p \pm 1.96\sqrt{\hat p(1 - \hat p)/n}$ (the Wald interval), is unreliable when $n$ is small or $p$ is close to 0 or 1. With $n = 20$ and $p = 0.1$ its actual coverage is 88%, not 95%: whenever no success is observed, which happens with probability $0.9^{20} = 0.12$, the interval collapses to the single point 0. The Wilson interval, obtained by solving $\lvert \hat p - p \rvert \le 1.96 \sqrt{p(1-p)/n}$ for $p$ instead of plugging in $\hat p$, has coverage 96% in the same setting and is the better default for click rates and accuracies on small test sets. It is still an approximate interval: with $n = 20$ its coverage falls to 85% at $p = 0.008$.

The bootstrap#

Many statistics have no convenient formula for their standard error: a median, a correlation, the gain of one model over another, or classifier metrics such as the area under the ROC curve (AUC) and the F1 score of the classification metrics note. The bootstrap estimates it by simulation. The unknown distribution of the data is replaced by the empirical distribution, which puts mass $1/n$ on each observation, and sampling from the population is replaced by resampling the data with replacement:

1. Draw $n$ values from the sample with replacement.
2. Compute the statistic on this resample.
3. Repeat $B$ times (typically 1,000 to 10,000).
4. The standard deviation of the $B$ values estimates the standard error; their 2.5% and 97.5% percentiles give a 95% percentile interval.

Figure

The bootstrap for a median, step by step

For the median of 50 values from an exponential distribution, one sample has median $0.549$ (the true median is $\ln 2 = 0.693$). The bootstrap standard error from that one sample is $0.136$; the true standard error, from 100,000 fresh samples, is $0.140$. The percentile interval is $[0.415, 0.999]$, and it covers the true median. The bootstrap does this without knowing that the data were exponential.

When the bootstrap fails#

The bootstrap works when the statistic depends smoothly on the distribution of the data and the sample is reasonably large. It fails, or needs modification, in a few well-known cases:

• Extremes. The maximum of a resample can never exceed the sample maximum, and it equals it in about 63% of resamples ($1 - (1 - 1/n)^n$). The bootstrap distribution of the maximum is nothing like the true one.
• Small samples. With 10 observations, the empirical distribution is a poor stand-in for the population, and percentile intervals undercover; bias-corrected and accelerated (BCa) intervals help, but not much.
• Dependent data. Resampling single observations destroys dependence. Time series need the block bootstrap, and data in groups (several examples per user or per document) need resampling of whole groups.
• Heavy tails. For the sample mean of data with infinite variance, such as tails with exponent below 2, the bootstrap is inconsistent.

Comparing two models on one test set#

Two classifiers evaluated on the same 1,000 examples, with accuracies $0.833$ and $0.859$: is the difference real? The errors of the two models are strongly correlated, since hard examples are hard for both. A paired bootstrap resamples examples, keeps both models’ results on each resampled example together, and computes the difference on each resample. Its 95% interval for the difference is $[0.006, 0.046]$, which excludes zero. Treating the two accuracies as independent, resampling them separately, gives $[-0.005, 0.057]$, which does not: ignoring the pairing throws away the information that the models fail on the same examples. The same principle, compare on the same units and resample the units, applies to A/B tests with repeated measurements and to evaluations across prompts or documents.

What changed#

	Point estimate	Confidence interval
Reports	$\hat\theta$	$[L, U]$ with stated coverage
Meaning of 95%		over repeated samples, the procedure captures $\theta$ at least 95% of the time
Mean, small $n$		$\bar x \pm t_{n-1} s/\sqrt n$
Proportion	$\hat p$	Wilson rather than Wald
No formula		bootstrap: resample, recompute, take percentiles
Two models, one test set	two numbers	paired interval for the difference

Limitations#

• Coverage is a promise about the procedure, under its assumptions. Independent observations, a correct model, the right standard error; clustered data and data snooping break it silently.
• The bootstrap cannot see beyond the sample. If the sample is unrepresentative, resampling it reproduces the same error; the bootstrap can estimate an estimator’s bias, but a percentile interval does not correct for it.
• One interval answers one question. Reporting many intervals and highlighting the ones that exclude zero is a multiple-comparisons problem, the subject of the next note.

What you should remember#

• An exact 95% confidence interval comes from a procedure that covers the true value in at least 95% of repeated samples, for every value of the parameter; normal-approximation, Wilson and bootstrap intervals reach 95% only approximately. Any one interval either covers the value or not.
• $\hat\theta \pm 1.96\, \widehat{\operatorname{se}}$ for approximately normal estimators; for a mean with small $n$, $\bar x \pm t_{n-1, 0.975}\, s/\sqrt n$.
• The Wald interval for a proportion undercovers for small $n$ or extreme $p$; use the Wilson interval.
• Bootstrap: resample with replacement, recompute the statistic, use the standard deviation or the percentiles.
• It fails for extremes, tiny samples, dependent data (use blocks or groups) and infinite variance.
• Compare two models on the same test set with a paired interval.

Code#

Coverage of the t, Wald and Wilson intervals, the bootstrap of a median and a maximum, and a paired comparison:

import numpy as np
from scipy import stats

rng = np.random.default_rng(0)

# coverage of 95% intervals for a mean, n = 10 from N(5, 2^2): the t quantile 2.262 against the normal 1.96
X = rng.normal(5, 2, (100_000, 10))
m, s = X.mean(axis=1), X.std(axis=1, ddof=1)
for q in (stats.t(9).ppf(0.975), 1.96):
    h = q * s / np.sqrt(10)
    print(round(float(q), 3), round(float(((m - h <= 5) & (5 <= m + h)).mean()), 3))
# 2.262 0.951 | 1.96 0.919

# a proportion with n = 20, p = 0.1: coverage of the Wald and the Wilson intervals
n, p, z = 20, 0.1, 1.96
ph = rng.binomial(n, p, 200_000) / n
w = z * np.sqrt(ph * (1 - ph) / n)
c = (ph + z * z / (2 * n)) / (1 + z * z / n)
h = z * np.sqrt(ph * (1 - ph) / n + z * z / (4 * n * n)) / (1 + z * z / n)
print(round(float(((ph - w <= p) & (p <= ph + w)).mean()), 3), round(float(((c - h <= p) & (p <= c + h)).mean()), 3))   # 0.877 0.957

# the bootstrap for the median of 50 exponential values: standard error and percentile interval
x = rng.exponential(1, 50)
boot = np.median(rng.choice(x, (2000, 50), replace=True), axis=1)
true = np.median(rng.exponential(1, (100_000, 50)), axis=1)
print(round(float(np.median(x)), 3), round(float(boot.std()), 3), np.percentile(boot, [2.5, 97.5]).round(3), round(float(true.std()), 3))
# 0.549 0.136 [0.415 0.999] 0.14

# the bootstrap fails for the maximum: the sample maximum reappears in about 63% of resamples
u = rng.random(50)
print(round(float((rng.choice(u, (2000, 50)).max(axis=1) == u.max()).mean()), 3), round(1 - (1 - 1 / 50) ** 50, 3))   # 0.625 0.636

# comparing two models on the same 1,000 test examples: paired against unpaired bootstrap
N = 1000
difficulty = rng.normal(size=N)
a = difficulty + 0.4 * rng.normal(size=N) > -1.04         # model A is correct on these examples
b = difficulty + 0.4 * rng.normal(size=N) > -1.18         # model B, correct mostly on the same ones
idx, idx2 = rng.integers(0, N, (5000, N)), rng.integers(0, N, (5000, N))
paired, unpaired = b[idx].mean(axis=1) - a[idx].mean(axis=1), b[idx2].mean(axis=1) - a[idx].mean(axis=1)
print(a.mean(), b.mean(), np.percentile(paired, [2.5, 97.5]).round(3), np.percentile(unpaired, [2.5, 97.5]).round(3))
# 0.833 0.859 [0.006 0.046] [-0.005  0.057]

Next#

A 95% confidence interval for a difference that excludes zero rejects the hypothesis of no difference at the 5% level, under the assumptions of the interval. It does not prove that the difference is real, nor that it is large enough to matter, and it means less when it was one of many intervals looked at. The next note makes that decision explicit: null hypotheses, test statistics, p-values and their many misreadings, the power to detect an effect, and the A/B test from design to the pitfalls of peeking and multiple comparisons.

References#

• J. Neyman. Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability. Philosophical Transactions of the Royal Society A, 236, 1937.
• B. Efron. Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics, 7(1), 1979.
• B. Efron, R. J. Tibshirani. An Introduction to the Bootstrap. Chapman & Hall, 1993.
• L. D. Brown, T. T. Cai, A. DasGupta. Interval Estimation for a Binomial Proportion. Statistical Science, 16(2), 2001.
• L. Wasserman. All of Statistics. Springer, 2004. Chapter 6 (confidence intervals), Chapter 8 (the bootstrap).