Estimators: bias and variance

Probability goes from a known distribution to the data it produces: given a fair coin, how many heads? Statistics goes the other way: given the heads, what is the coin? The unknown is a parameter $\theta$ of a family of distributions, and a rule that turns data into a guess is an estimator $\hat\theta$. Every number fitted in machine learning, from a mean to the weights of a network, is an estimator. This note introduces the vocabulary that judges them: bias, variance, mean squared error, consistency and the standard error, and ends with the bias–variance trade-off that governs the choice of a model’s complexity.

Notation. $X_1, \dots, X_n$ are i.i.d. from a distribution $P_\theta$ with unknown parameter $\theta$. An estimator $\hat\theta = \hat\theta(X_1, \dots, X_n)$ is a function of the data. $E_\theta$ and $\operatorname{Var}_\theta$ are computed under $P_\theta$.

Estimators#

The sample mean $\bar X$ estimates the mean, the share of successes estimates a probability, the sample variance estimates the variance, a fitted regression line estimates the regression function of the conditional expectation note. Because the data are random, an estimator is a random variable: a different sample gives a different value. Its distribution over repeated samples, the sampling distribution, is what makes it good or bad. The average of $n$ observations, for instance, has sampling distribution approximately $N(\mu, \sigma^2/n)$ by the central limit theorem.

The same parameter usually has many estimators. For a sample from $U(0, \theta)$, the end point $\theta$ can be estimated by twice the sample mean, since $E[X] = \theta/2$, or by the sample maximum, the largest value seen, or by the maximum scaled up slightly. They behave very differently.

Bias, variance and mean squared error#

The bias is the systematic error, the variance the random one:

$$\operatorname{bias}(\hat\theta) = E_\theta[\hat\theta] - \theta, \qquad \operatorname{Var}(\hat\theta) = E_\theta\big[(\hat\theta - E_\theta[\hat\theta])^2\big] .$$

The mean squared error combines them. Adding and subtracting $E_\theta[\hat\theta]$ inside the square, the cross term has mean zero:

$$\operatorname{MSE}(\hat\theta) = E_\theta\big[(\hat\theta - \theta)^2\big] = \operatorname{bias}(\hat\theta)^2 + \operatorname{Var}(\hat\theta) .$$

For the uniform end point with $n = 10$ and $\theta = 1$:

Figure

Three estimators of the end point of a uniform distribution

Estimator	Bias	Variance	MSE
$2\bar X$	0	$\theta^2/(3n) = 0.0333$	0.0333
$\max_i X_i$	$-\theta/(n+1) = -0.091$	$0.0069$	0.0152
$\frac{n+1}{n} \max_i X_i$	0	$\theta^2/(n(n+2)) = 0.0083$	0.0083

The numbers for the maximum come from its distribution function: the maximum is at most $t$ exactly when all $n$ observations are, so

$$\begin{aligned} P\big(\max_i X_i \le t\big) &= (t/\theta)^n, \quad 0 \le t \le \theta, \qquad E\big[\max_i X_i\big] = \frac{n}{n+1}\,\theta, \\ \operatorname{Var}\big(\max_i X_i\big) &= E\big[(\max_i X_i)^2\big] - \Big(\frac{n\theta}{n+1}\Big)^2 \\ &= \frac{n\theta^2}{n+2} - \frac{n^2\theta^2}{(n+1)^2} = \frac{n\theta^2}{(n+1)^2(n+2)} , \end{aligned}$$

with the moments computed from the density $n t^{n-1}/\theta^n$; for $n = 10$ the variance is $10/(121 \cdot 12) = 0.0069$.

Unbiasedness is not the goal. The maximum is always too small, yet its mean squared error is less than half that of the unbiased $2\bar X$, because the maximum uses the one observation that carries the most information about the end point. Correcting its bias keeps that advantage and gives the best of the three; its variance even falls like $1/n^2$, faster than the $1/n$ of any average.

The sample variance and n − 1#

The natural estimator of $\sigma^2$ averages the squared deviations from the sample mean. It is biased:

$$E\Big[\frac1n \sum_{i=1}^n (X_i - \bar X)^2\Big] = \frac{n-1}{n}\, \sigma^2 ,$$

because the deviations are measured from $\bar X$, which is fitted to the same data and so sits closer to them than the true mean $\mu$ does. Dividing by $n - 1$ instead (Bessel’s correction) removes the bias; this $s^2$ is what most statistical software reports. But for normal data the mean squared error is smallest with yet another divisor, $n + 1$: for $n = 10$ the divisors $9$, $10$ and $11$ give mean squared errors $0.222\sigma^4$, $0.190\sigma^4$ and $0.182\sigma^4$. Again the unbiased choice is not the most accurate one, though for moderate $n$ the differences are small.

Consistency#

An estimator is consistent if it converges in probability to the true value as $n$ grows: $\hat\theta_n \to \theta$. A sufficient condition is that both the bias and the variance go to zero, since then the mean squared error does. The sample mean is consistent by the law of large numbers. The maximum of a uniform sample is biased for every $n$ and still consistent. The first observation alone, $\hat\theta = X_1$, is unbiased for the mean and not consistent: it ignores all the other data. Consistency is the minimum requirement; bias, variance and their rates are what distinguish consistent estimators.

Standard errors#

The standard deviation of an estimator is its standard error, the typical size of the random part of its error; for an unbiased estimator, the typical deviation from the truth. For the sample mean it is $\sigma/\sqrt n$, estimated by $s/\sqrt n$. Quoting an estimate without its standard error says nothing about its precision; with it, the central limit theorem gives an approximate interval, the subject of the confidence intervals note. For estimators without a simple formula, the standard error can be estimated by resampling the data, the bootstrap of the same note.

The bias–variance trade-off#

A model $\hat f$ trained with squared error is an estimator of the regression function $m(x) = E[Y \mid X = x]$ (other losses estimate other summaries of the conditional distribution, as the conditional expectation note shows), and its sampling distribution comes from the randomness of the training set. At a point $x$, with a new $Y$ independent of the training data, the expected squared error splits in three:

$$E\big[(Y - \hat f(x))^2\big] = \underbrace{\operatorname{Var}(Y \mid X = x)}_{\text{noise}} + \underbrace{\big(E[\hat f(x)] - m(x)\big)^2}_{\text{bias}^2} + \underbrace{\operatorname{Var}\big(\hat f(x)\big)}_{\text{variance}} .$$

The noise is the irreducible error of the conditional expectation note. A model class too simple for $m$ has high bias: every training set gives about the same wrong answer. A class flexible enough to follow $m$ also follows the noise of each training set and has high variance:

Figure

Bias and variance of polynomial regression

With 12 training points, degree 1 has total expected error $0.218 + 0.014 + 0.09 = 0.32$, degree 3 has $0.006 + 0.026 + 0.09 = 0.12$, degree 9 has $0.000 + 0.116 + 0.09 = 0.21$. The best complexity balances the two, and it depends on the amount of data: more data lowers the variance of every class and moves the balance toward more flexible models. This U shape is a common regime, not a law. Bias and variance need not move monotonically with every measure of flexibility, and heavily overparameterized models, such as large neural networks, can see the test error fall again beyond the point where they fit the training data exactly, the double descent of Belkin et al. (2019). Regularization (ridge, weight decay, early stopping, dropout) is a deliberate trade of a little bias for a large cut in variance, the same trade that made the uniform maximum beat $2\bar X$. In many dimensions it is unavoidable: the James–Stein estimator shows that, for three or more normal means, shrinking all sample means by the right amount toward a fixed point chosen in advance lowers the total mean squared error for every value of the true means.

What changed#

	Probability	Statistics
Known	the distribution, $\theta$	the data
Unknown	the data	$\theta$
Object of study	a random variable	an estimator, a random variable through the data
Quality		bias, variance, $\operatorname{MSE} = \operatorname{bias}^2 + \operatorname{variance}$, consistency, standard error
For a trained model		noise + bias² + variance; complexity and regularization set the balance

Limitations#

• Bias and MSE depend on the unknown $\theta$. An estimator can be better for some values of the parameter and worse for others; comparing them needs a criterion such as the worst case or an average over a prior.
• Mean squared error is one loss among many. Absolute error, coverage of intervals or the cost of a wrong decision can rank estimators differently.
• The decomposition for models is pointwise and over training sets. In practice there is one training set; bias and variance are estimated by resampling or by validation curves, not observed directly.

What you should remember#

• An estimator is a function of the data and so a random variable; its sampling distribution decides its quality.
• $\operatorname{MSE} = \operatorname{bias}^2 + \operatorname{variance}$; a biased estimator can have a smaller MSE than an unbiased one.
• The sample variance with $1/n$ underestimates $\sigma^2$ on average by the factor $(n-1)/n$; $1/(n-1)$ removes the bias.
• Consistent: $\hat\theta_n \to \theta$ in probability; vanishing bias and variance suffice.
• The standard error is the standard deviation of the estimator; for a mean, $\sigma/\sqrt n$.
• Expected prediction error = noise + bias² + variance; in the common regime of the polynomial example flexibility lowers bias and raises variance, though not always (double descent); regularization trades bias for variance.

Code#

The uniform estimators, the divisors of the sample variance and the polynomial fits, by simulation:

import numpy as np

rng = np.random.default_rng(0)

# three estimators of theta = 1 from n = 10 draws of U(0, theta): bias, variance, MSE over 200,000 samples
n = 10
X = rng.random((200_000, n))
for name, est in [('2 x mean', 2 * X.mean(axis=1)), ('max', X.max(axis=1)), ('max x 11/10', X.max(axis=1) * (n + 1) / n)]:
    print(name, round(float(est.mean() - 1), 3), round(float(est.var()), 4), round(float(((est - 1) ** 2).mean()), 4))
# 2 x mean 0.0 0.0333 0.0333 | max -0.091 0.0069 0.0151 | max x 11/10 0.0 0.0083 0.0083

# the sample variance of 10 standard normals with divisors n - 1, n, n + 1: bias and MSE
Z = rng.normal(size=(200_000, n))
ss = ((Z - Z.mean(axis=1, keepdims=True)) ** 2).sum(axis=1)
for d in (n - 1, n, n + 1):
    print(d, round(float((ss / d).mean() - 1), 3), round(float(((ss / d - 1) ** 2).mean()), 3))
# 9 -0.001 0.221 | 10 -0.101 0.189 | 11 -0.182 0.181

# squared bias and variance of polynomial fits: 12 points of sin(2 pi x) + N(0, 0.3^2), 2,000 datasets
f = lambda x: np.sin(2 * np.pi * x)
x, grid = np.linspace(0, 1, 12), np.linspace(0, 1, 101)
for deg in (1, 3, 9):
    P = np.array([np.polynomial.polynomial.polyval(2 * grid - 1, np.polynomial.polynomial.polyfit(2 * x - 1, f(x) + 0.3 * rng.normal(size=12), deg)) for _ in range(2000)])
    print(deg, round(float(((P.mean(axis=0) - f(grid)) ** 2).mean()), 3), round(float(P.var(axis=0).mean()), 3))
# 1 0.218 0.014 | 3 0.006 0.026 | 9 0.0 0.116

Next#

Bias and variance judge an estimator once it is given; they do not say where to find a good one. The next note introduces the most widely used recipe, maximum likelihood: choose the parameter under which the observed data are most probable. It derives squared error and cross-entropy as negative log-likelihoods, and shows why maximum likelihood estimators are, for large samples, about as good as any estimator can be.

References#

• L. Wasserman. All of Statistics. Springer, 2004. Chapter 6 (point estimation, bias, variance, MSE, consistency, standard errors).
• G. Casella, R. L. Berger. Statistical Inference, 2nd ed. Duxbury, 2002. Chapter 7 (methods of finding and evaluating estimators).
• T. Hastie, R. Tibshirani, J. Friedman. The Elements of Statistical Learning, 2nd ed. Springer, 2009. Sections 2.9 and 7.3 (model complexity, the bias–variance decomposition).
• S. Geman, E. Bienenstock, R. Doursat. Neural Networks and the Bias/Variance Dilemma. Neural Computation, 4(1), 1992.
• M. Belkin, D. Hsu, S. Ma, S. Mandal. Reconciling modern machine-learning practice and the classical bias–variance trade-off. PNAS, 116(32), 2019.
• C. Stein. Inadmissibility of the Usual Estimator for the Mean of a Multivariate Normal Distribution. Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, 1956.