Expectation and variance

A distribution is a whole function, and most questions need much less: where is it centred, and how far from the centre do the values typically fall? The expectation and the variance answer these two questions. They are also the quantities machine learning optimizes and reports: a risk is an expected loss, an accuracy is an expected indicator, and the error bar of a benchmark is a standard deviation. This note defines both, shows the one property that makes expectations easy to compute, linearity, and derives the $1/\sqrt n$ rule for averages that the opening note stated without proof.

Notation. $E[X]$ is the expectation (mean) of $X$, often $\mu$; $\operatorname{Var}(X)$ its variance, $\sigma^2$; $\sigma = \sqrt{\operatorname{Var}(X)}$ its standard deviation. $X_1, \dots, X_n$ are i.i.d. if they are independent and have the same distribution; $\bar X_n = (X_1 + \dots + X_n)/n$ is their average.

Expectation#

The expectation weights every value by its probability:

$$E[X] = \sum_k k\, P(X = k) \quad \text{(discrete)}, \qquad E[X] = \int_{-\infty}^{\infty} x\, f_X(x)\, dx \quad \text{(continuous)} .$$

A fair die has $E[X] = (1 + 2 + \dots + 6)/6 = 3.5$, a value it never shows. Two readings make the definition natural. Physically, $E[X]$ is the balance point: a rod loaded with mass $P(X = k)$ at position $k$ balances there. Statistically, it is the long-run average: the average of many independent copies of $X$ approaches $E[X]$, which the law of large numbers will prove. The distributions of the previous notes have the means

Distribution	$\mathrm{Bernoulli}(p)$	$\mathrm{Bin}(n, p)$	$\mathrm{Geom}(p)$	$\mathrm{Poisson}(\lambda)$	$U(a, b)$	$\mathrm{Exp}(\lambda)$	$N(\mu, \sigma^2)$
Mean	$p$	$np$	$1/p$	$\lambda$	$(a+b)/2$	$1/\lambda$	$\mu$

The mean is one of several centres. For a symmetric distribution the mean and the median coincide; for a skewed one they separate, and the mean is pulled toward the long tail:

Figure

Mode, median and mean of a skewed distribution

This is why reports of incomes or house prices quote the median: a few very large values move the mean far more than they move the median.

Linearity#

For any random variables and constants,

$$E[aX + bY + c] = a E[X] + b E[Y] + c .$$

Written as a weighted sum over outcomes rather than over values, the expectation is linear in the random variable, so the proof is one line, and it needs no independence. Its power comes from indicators: $E[\mathbb 1_A] = 1 \cdot P(A) + 0 \cdot P(A^c) = P(A)$. A count of events is a sum of indicators, so its expectation is a sum of probabilities, however the events depend on each other.

• Binomial mean. $X \sim \mathrm{Bin}(n, p)$ is a sum of $n$ indicators, each with probability $p$: $E[X] = np$, without touching the PMF.
• Secret Santa. The axioms note needed inclusion–exclusion to find that nobody draws their own name with probability close to $1/e$. The expected number of people who draw their own name is a one-liner: person $i$ does with probability $1/n$, so the expectation is $n \cdot \frac1n = 1$, for every group size.
• Shared birthdays. Among 23 people there are $\binom{23}{2} = 253$ pairs, each sharing a birthday with probability $1/365$; the expected number of shared pairs is $253/365 = 0.69$.
• The coupon collector. How many rolls of a die until every face has appeared? Split the wait into six stages, one per new face; each stage is geometric, and linearity adds their means:

Figure

How many rolls it takes to see every face of a die

With $n$ equally likely coupons the same argument gives $n (1 + \frac12 + \dots + \frac1n) \approx n (\ln n + 0.58)$: about 519 packs to complete a collection of 100 stickers, most of them spent on the last few.

The expectation of a function#

To find $E[g(X)]$, there is no need to find the distribution of $g(X)$ first: weight $g$ of each value by the probability of the value,

$$E[g(X)] = \sum_k g(k)\, P(X = k), \qquad E[g(X)] = \int g(x)\, f_X(x)\, dx .$$

For the log-normal $Y = e^X$ with $X \sim N(0, 1)$ of the previous note, completing the square gives

$$E[e^X] = \int e^x \frac{e^{-x^2/2}}{\sqrt{2\pi}}\, dx = e^{1/2} \int \frac{e^{-(x-1)^2/2}}{\sqrt{2\pi}}\, dx = e^{1/2} = 1.6487 ,$$

the mean in the figure above. It is not $e^{E[X]} = e^0 = 1$: in general $E[g(X)] \ne g(E[X])$. For a convex $g$ (a function that bends upward) the inequality has a direction, Jensen’s inequality:

$$E[g(X)] \ge g(E[X]) .$$

With $g(x) = x^2$ it says $E[X^2] \ge (E[X])^2$. With the concave $\log$ it reverses, $E[\log X] \le \log E[X]$, and this one line gives the evidence lower bound of the variational autoencoder and the fact that the KL divergence of the entropy note is never negative. In practice it is a warning: the average of a loss over predictions is not the loss of the average prediction.

Variance#

The variance measures the typical squared distance from the mean:

$$\operatorname{Var}(X) = E\big[(X - \mu)^2\big] = E[X^2] - \mu^2 .$$

The second form follows from linearity: $E[X^2 - 2\mu X + \mu^2] = E[X^2] - 2\mu^2 + \mu^2$. For a die, $E[X^2] = 91/6$, so $\operatorname{Var}(X) = 91/6 - 3.5^2 = 35/12 \approx 2.92$ and $\sigma \approx 1.71$. The standard deviation is in the units of $X$, which makes it the number to report; the variance is the one that adds up.

Distribution	$\mathrm{Bernoulli}(p)$	$\mathrm{Bin}(n, p)$	$\mathrm{Geom}(p)$	$\mathrm{Poisson}(\lambda)$	$U(a, b)$	$\mathrm{Exp}(\lambda)$	$N(\mu, \sigma^2)$
Variance	$p(1-p)$	$np(1-p)$	$(1-p)/p^2$	$\lambda$	$(b-a)^2/12$	$1/\lambda^2$	$\sigma^2$

Two rules do most of the work. Shifting does not change the spread and scaling multiplies it: $\operatorname{Var}(aX + b) = a^2 \operatorname{Var}(X)$. And for independent $X$ and $Y$, variances add: $\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y)$; the joint distributions note proves it and shows the extra covariance term for dependent variables.

The $1/\sqrt n$ rule. For i.i.d. $X_1, \dots, X_n$ with variance $\sigma^2$, the two rules give

$$\operatorname{Var}(\bar X_n) = \frac{1}{n^2} \cdot n \sigma^2 = \frac{\sigma^2}{n}, \qquad \operatorname{sd}(\bar X_n) = \frac{\sigma}{\sqrt n} .$$

The average is $n$ times less variable than one observation in variance, but only $\sqrt n$ times in standard deviation. A classifier with true accuracy $0.9$, measured on 1,000 independent test examples, gets a measured accuracy with standard deviation $\sqrt{0.9 \cdot 0.1 / 1000} = 0.0095$, about one percentage point; halving that requires 4,000 examples.

Moment and characteristic functions#

Optional depth: moments beyond the variance, and the transforms that turn sums into products and prove the central limit theorem. The rest of the note does not depend on it.

The mean and the variance are built from the first two moments $E[X^k]$, $k = 1, 2$. The third and fourth standardized moments, $E[(X - \mu)^3]/\sigma^3$ and $E[(X - \mu)^4]/\sigma^4$, are the skewness and the kurtosis: lopsidedness, and how much of the variance comes from rare large deviations.

All the moments are packed into one function, the moment generating function

$$M_X(t) = E\big[e^{tX}\big], \qquad M_X^{(k)}(0) = E[X^k] ,$$

since the $k$-th derivative of $e^{tX}$ at $t = 0$ is $X^k$. Its main use is sums: for independent $X$ and $Y$, $e^{t(X+Y)} = e^{tX} e^{tY}$ and the expectation of a product of independent variables factorizes, so $M_{X+Y}(t) = M_X(t)\, M_Y(t)$. A Poisson($\lambda$) variable has $M(t) = \exp(\lambda(e^t - 1))$; multiplying two of them gives the MGF of Poisson($\lambda_1 + \lambda_2$), and since an MGF that exists near 0 determines the distribution, the sum of independent Poisson variables is Poisson. The normal has $M(t) = \exp(\mu t + \sigma^2 t^2 / 2)$, and the same argument shows that sums of independent normals are normal.

The MGF does not exist for heavy-tailed distributions such as the log-normal, where $E[e^{tY}]$ is infinite for every $t > 0$. The characteristic function $\psi_X(t) = E[e^{itX}]$ always exists. Here $i$ is the imaginary unit, $i^2 = -1$, and $e^{itX} = \cos(tX) + i \sin(tX)$, so $\psi_X(t) = E[\cos(tX)] + i\, E[\sin(tX)]$ is a pair of ordinary expectations of bounded quantities, and $\lvert e^{itX} \rvert = 1$. It also determines the distribution and turns sums into products. Its other property is Lévy’s continuity theorem, the tool behind the proof of the central limit theorem: the distributions of $X_n$ converge to the distribution of $X$ exactly when $\psi_{X_n}(t) \to \psi_X(t)$ for every $t$; and if $\psi_{X_n}(t)$ converges for every $t$ to a limit that is continuous at $t = 0$, that limit is the characteristic function of some distribution, to which the $X_n$ converge. The condition at $0$ matters: $N(0, n)$ has $\psi(t) = e^{-nt^2/2}$, which tends to $0$ for every $t \ne 0$ and equals $1$ at $t = 0$. The limit jumps at $0$, it is not a characteristic function, and the distributions spread out without converging to any distribution.

What changed#

	A distribution	Its summaries
Centre	a whole PMF or density	$E[X]$, a balance point and long-run average
Spread		$\operatorname{Var}(X)$, $\sigma$
Sums	convolution of distributions	means always add; variances add under independence; MGFs multiply
Averages of $n$		mean unchanged, standard deviation $\sigma/\sqrt n$

Limitations#

• The expectation may not exist. The Cauchy distribution, the ratio of two independent standard normals, has a density but no mean: its averages do not settle however many samples are taken (see the code).
• Means are sensitive to tails. A single extreme value can move a mean arbitrarily far; the median is robust, and for skewed quantities such as latencies, incomes or losses on rare inputs the mean alone misleads.
• Two numbers do not describe a shape. Distributions with the same mean and variance can have very different tails, which is exactly where risk, failures and outliers live.

What you should remember#

• $E[X] = \sum k\, P(X = k)$ or $\int x f(x)\, dx$: a balance point and the long-run average.
• Linearity: $E[aX + bY + c] = aE[X] + bE[Y] + c$ for any dependence; $E[\mathbb 1_A] = P(A)$ turns expected counts into sums of probabilities.
• $E[g(X)] = \sum g(k) P(X = k)$; in general $E[g(X)] \ne g(E[X])$, and $E[g(X)] \ge g(E[X])$ for convex $g$ (Jensen).
• $\operatorname{Var}(X) = E[X^2] - (E[X])^2$; $\operatorname{Var}(aX + b) = a^2 \operatorname{Var}(X)$; variances of independent variables add.
• The average of $n$ i.i.d. variables has standard deviation $\sigma/\sqrt n$.
• MGFs and characteristic functions turn sums of independent variables into products.

Code#

Each number of the note, computed exactly or by simulation:

import numpy as np, math
from scipy import stats

rng = np.random.default_rng(0)

# a fair die: the variance from the definition and as E[X^2] - (E[X])^2
die = np.arange(1, 7)
mean = die.mean()
print(mean, round(((die - mean) ** 2).mean(), 4), round((die**2).mean() - mean**2, 4))   # 3.5 2.9167 2.9167

# linearity with indicators: own names drawn among 20 people; shared-birthday pairs among 23
perm = np.argsort(rng.random((100_000, 20)), axis=1)
print(round((perm == np.arange(20)).sum(axis=1).mean(), 3), round(math.comb(23, 2) / 365, 4))   # 0.996 0.6932

# coupon collector: rolls until all six faces have appeared
rolls = []
for _ in range(20_000):
    seen, n = set(), 0
    while len(seen) < 6:
        seen.add(int(rng.integers(6))); n += 1
    rolls.append(n)
print(round(sum(6 / k for k in range(1, 7)), 2), round(float(np.mean(rolls)), 2))   # 14.7 14.71

# the expectation of a function: E[e^X] for X ~ N(0, 1), against e^E[X] = 1 and the median of e^X
x = rng.normal(size=1_000_000)
print(round(float(np.exp(x).mean()), 3), round(math.exp(0.5), 4), round(float(np.median(np.exp(x))), 3))   # 1.645 1.6487 0.998

# the 1/sqrt(n) rule: accuracy measured on 1,000 test examples when the true accuracy is 0.9
print(round(math.sqrt(0.9 * 0.1 / 1000), 4), round(float(rng.binomial(1000, 0.9, 100_000).std()) / 1000, 4))   # 0.0095 0.0095

# generating functions: Poisson(2) + Poisson(3), independent, is Poisson(5)
s = rng.poisson(2, 1_000_000) + rng.poisson(3, 1_000_000)
print(round(float(s.mean()), 3), round(float(s.var()), 3), round(float((s == 5).mean()), 4), round(stats.poisson(5).pmf(5), 4))   # 4.998 5.004 0.1756 0.1755

# no mean at all: averages of Cauchy samples do not settle
c = rng.standard_cauchy(1_000_000)
print([round(float(c[:n].mean()), 2) for n in (10**3, 10**4, 10**5, 10**6)])   # [0.83, 2.88, 0.4, -0.23]

Next#

The expectation and the variance describe one random variable. Data come in several at once: a person’s height and weight, a word and the next one, a feature vector and a label. The next note describes several random variables together: joint and marginal distributions, independence, and the covariance and correlation that measure, partly, how they move together.

References#

• J. K. Blitzstein, J. Hwang. Introduction to Probability, 2nd ed. CRC Press, 2019. Chapter 4 (expectation, linearity, indicators, LOTUS, variance, the coupon collector), Chapter 6 (moments, moment generating functions).
• W. Feller. An Introduction to Probability Theory and Its Applications, Vol. 1, 3rd ed. Wiley, 1968. Chapter IX (random variables, expectation), Chapter XI (generating functions).
• A. N. Shiryaev. Probability-1, 3rd ed. Springer, 2016. Chapter 2, §6 (expectation as an integral), §12 (characteristic functions).