Continuous distributions

A waiting time, a height, the error of a measurement or the position of a random point can take any value in an interval. Each single value then has probability zero, as the axioms note showed for a point in $[0, 1]$, so a list of probabilities of values, the PMF of the previous note, describes nothing. The probability sits on intervals instead, and its concentration at each point is described by a density. This note introduces densities, the distribution function and quantiles, and three distributions: the uniform, from which random numbers are made, the exponential, the waiting time with no memory, and the normal, which the central limit theorem will put behind every average.

Notation. $f_X$ is the density of $X$, $F_X(x) = P(X \le x)$ its distribution function (CDF), $F_X^{-1}(q)$ its $q$-quantile. $X \sim N(\mu, \sigma^2)$ reads ”$X$ is normal with mean $\mu$ and variance $\sigma^2$”; $\varphi$ and $\Phi$ are the density and CDF of the standard normal $N(0, 1)$.

Densities#

A random variable $X$ is continuous (strictly, absolutely continuous) if there is a function $f_X \ge 0$ with $\int_{-\infty}^{\infty} f_X(x)\, dx = 1$ such that for every interval

$$P(a \le X \le b) = \int_a^b f_X(x)\, dx .$$

Probabilities are areas under the density. Whether the endpoints are included does not matter, since single points have probability 0. Sums over values become integrals; otherwise everything said about PMFs carries over.

A density is not a probability. For a short interval, $P(x \le X \le x + h) \approx f_X(x)\, h$: the density is probability per unit length, and it can exceed 1. A point uniform on $[0, 0.5]$ has density 2 on that interval. A density also depends on the units: a height measured in metres instead of centimetres has a density 100 times larger, while all probabilities stay the same.

The distribution function and quantiles#

The CDF $F_X(x) = P(X \le x) = \int_{-\infty}^x f_X(t)\, dt$ rises from 0 to 1. At every point where $f_X$ is continuous, the derivative of the CDF is the density, $F_X' = f_X$, so every probability is both an area under $f$ and a rise of $F$:

Figure

One probability as an area and as a rise

The CDF answers “how likely is $X$ to be at most $x$”; its inverse answers “below which value does $X$ fall with probability $q$”. The $q$-quantile is the smallest $x$ with $F_X(x) \ge q$,

$$Q(q) = \inf\{x : F_X(x) \ge q\} .$$

Where $F_X$ is continuous and strictly increasing, this is the unique $x$ with $F_X(x) = q$, written $F_X^{-1}(q)$. The general definition handles the two exceptions: a flat stretch of $F_X$, a range of values that $X$ never takes, would give many solutions of $F_X(x) = q$, and a jump, a value with positive probability, would give none. The median is the $0.5$-quantile, the quartiles the $0.25$- and $0.75$-quantiles. Service-level targets are quantiles: a “p99 latency of 300 ms” says that the $0.99$-quantile of the response time is 300 ms. Inverting $F$ is also how random numbers with any distribution are made from uniform ones, the subject of the next note.

Uniform#

$X \sim U(a, b)$ has the constant density $1/(b - a)$ on $[a, b]$ and 0 outside: the probability of an interval inside $[a, b]$ is its length divided by $b - a$. A computer’s random number generator produces (approximations to) independent $U(0, 1)$ numbers, and every other distribution in a simulation is made from them.

Exponential#

A machine fails, a request arrives, an atom decays. Suppose such events happen at a constant rate $\lambda$ per unit time and without memory: in every short interval of length $\Delta$ an event happens with probability about $\lambda \Delta$, whatever happened before (a homogeneous Poisson process). Then the waiting time $T$ until the first one has the exponential distribution, $T \sim \mathrm{Exp}(\lambda)$:

$$f_T(t) = \lambda e^{-\lambda t}, \qquad P(T > t) = e^{-\lambda t}, \qquad t \ge 0 .$$

It is the continuous version of the geometric distribution. Divide time into steps of length $\Delta$ and run a trial at every step with success probability $\lambda \Delta$. The number of steps until the first success is geometric, so $P(T > t) = (1 - \lambda \Delta)^{t/\Delta}$, which tends to $e^{-\lambda t}$ as $\Delta \to 0$:

Figure

The exponential distribution is the continuous geometric

It is also the other face of the Poisson distribution. If the number of events in an interval of length $t$ is $\mathrm{Poisson}(\lambda t)$, the first event comes after $t$ exactly when the interval contains none: $P(T > t) = e^{-\lambda t} (\lambda t)^0 / 0! = e^{-\lambda t}$. For requests arriving at 2 per second, a whole second without a request has probability $e^{-2} = 0.135$. A constant average rate alone is not enough. Buses that come exactly every 10 minutes also arrive at a rate of $1/10$ per minute, but a passenger who turns up at a random moment waits a uniform time on $[0, 10]$, with mean 5 minutes, not an exponential one with mean 10.

Like the geometric, the exponential is memoryless: $P(T > s + t \mid T > s) = e^{-\lambda (s + t)} / e^{-\lambda s} = P(T > t)$. A component whose lifetime is exponential is as good as new at every moment; it does not age. The converse also holds: a survival function with $P(T > s + t) = P(T > s) P(T > t)$ for all $s, t$ must be exponential, so the exponential is the only memoryless continuous distribution. Real components do wear out, and their lifetimes are then modelled by distributions whose failure rate grows with age.

The exponential is skewed. With $\lambda = 2$ per second the mean wait is $1/\lambda = 0.5$ s, the median $\ln 2 / \lambda = 0.35$ s, and the $0.99$-quantile $\ln 100 / \lambda = 2.3$ s, more than four times the mean. A system sized for the average wait misses the tail.

Normal#

The normal (Gaussian) distribution $N(\mu, \sigma^2)$ has the density

$$f(x) = \frac{1}{\sigma \sqrt{2\pi}} \exp\Big(-\frac{(x - \mu)^2}{2\sigma^2}\Big) ,$$

a bell centred at $\mu$ with width set by $\sigma$. Every normal is a shifted and rescaled standard one: if $X \sim N(\mu, \sigma^2)$, then $Z = (X - \mu)/\sigma \sim N(0, 1)$, which the next note proves. So one table, of $\Phi$, serves all normals; $\Phi$ has no formula in elementary functions and is computed numerically.

The 68–95–99.7 rule gives the probability of falling within 1, 2 and 3 standard deviations of the mean: $0.6827$, $0.9545$ and $0.9973$. A machine cuts rods of length $N(100, 0.2^2)$ mm, and rods outside $[99.5, 100.5]$ are rejected. Both limits are $2.5$ standard deviations from the mean, so the reject rate is $P(\lvert Z \rvert > 2.5) = 0.0124$.

The normal’s tails are thin: the density falls like $e^{-x^2/2}$. $P(Z > 3) = 0.00135$, and $P(Z > 5) = 2.9 \cdot 10^{-7}$, about one in 3.5 million. Measurement errors and averages of many small effects are close to normal, for reasons the central limit theorem explains. Returns on financial assets, file sizes or word frequencies are not, and modelling them as normal underestimates extreme events by orders of magnitude.

What changed#

	Discrete	Continuous
Described by	PMF $p_X(k) = P(X = k)$	density $f_X(x)$, probability per unit length
$P(X = x)$	can be positive	always 0
$P(a \le X \le b)$	sum of $p_X(k)$	integral of $f_X$
CDF	a staircase	continuous, with $F_X' = f_X$
Waiting time	geometric, memoryless	exponential, the only memoryless one

Limitations#

• Not every variable is discrete or continuous. Daily rainfall is exactly 0 with positive probability and continuous otherwise; such mixed distributions need a mass at 0 and a density elsewhere. (There are also continuous CDFs with no density at all, which do not appear in practice.)
• A named distribution is a model. The exponential assumes a constant rate, the normal assumes thin tails; checking the fit, as with the horse kicks of the previous note, comes before using either.
• Densities depend on the units and the parametrization. Probabilities do not; comparing density values across scales, or reading a density value as a probability, is a common error.

What you should remember#

• A continuous variable has $P(a \le X \le b) = \int_a^b f_X$; single values have probability 0; a density is probability per unit length and can exceed 1.
• $F_X(x) = P(X \le x)$, $F_X' = f_X$; the $q$-quantile solves $F_X(x) = q$ (median, quartiles, p99).
• Uniform $U(a, b)$: constant density $1/(b-a)$; the raw material of simulation.
• Exponential $\mathrm{Exp}(\lambda)$: $P(T > t) = e^{-\lambda t}$, mean $1/\lambda$, memoryless; the waiting time of a Poisson stream of events.
• Normal $N(\mu, \sigma^2)$: $Z = (X - \mu)/\sigma$ is standard normal; within 1, 2, 3 standard deviations with probability $0.683$, $0.954$, $0.997$; thin tails.

Code#

The numbers of the note with scipy.stats, and memorylessness by simulation:

import numpy as np
from scipy import stats

Z = stats.norm()                                                   # the standard normal
print([round(float(Z.cdf(k) - Z.cdf(-k)), 4) for k in (1, 2, 3)])   # [0.6827, 0.9545, 0.9973]
print(round(2 * Z.sf(2.5), 4), round(Z.sf(3), 5), f'{Z.sf(5):.1e}')   # 0.0124 0.00135 2.9e-07

print(stats.uniform(0, 0.5).pdf(0.2))                              # 2.0: a density, not a probability

T = stats.expon(scale=1 / 2)                                       # waiting time at a rate of 2 per second
print(round(T.sf(1), 4), T.mean(), round(T.median(), 3), round(T.ppf(0.99), 3))   # 0.1353 0.5 0.347 2.303

# memorylessness by simulation: P(T > 1.5 | T > 0.5) = P(T > 1)
t = np.random.default_rng(0).exponential(1 / 2, 1_000_000)
print(round((t > 1).mean(), 4), round((t[t > 0.5] > 1.5).mean(), 4))   # 0.1353 0.136

# a trial every d seconds with success probability d: P(wait > 1) tends to e^-1
print([round((1 - d) ** round(1 / d), 4) for d in (0.5, 0.1, 0.01)], round(np.exp(-1), 4))   # [0.25, 0.3487, 0.366] 0.3679

Next#

The normal distribution was reduced to the standard one by $Z = (X - \mu)/\sigma$, and simulation turns uniform numbers into any distribution. Both are cases of one question: if $X$ has a known distribution, what is the distribution of $g(X)$? The next note answers it with the change-of-variables formula, which is also the one-dimensional case of the normalizing flows of the generative models course.

References#

• J. K. Blitzstein, J. Hwang. Introduction to Probability, 2nd ed. CRC Press, 2019. Chapter 5 (continuous random variables, uniform, normal, exponential, Poisson processes).
• W. Feller. An Introduction to Probability Theory and Its Applications, Vol. 2, 2nd ed. Wiley, 1971. Chapter I (the exponential and the uniform densities).
• A. N. Shiryaev. Probability-1, 3rd ed. Springer, 2016. Chapter 2, §3 (distribution functions and densities on the real line).