Functions of random variables

A temperature in Celsius becomes one in Fahrenheit, a measurement is standardized, a log-price is exponentiated back to a price, a uniform random number is turned into an exponential waiting time. Each time a random variable $X$ with a known distribution is passed through a function $g$, and the question is the distribution of $Y = g(X)$. For counts the answer is a sum; for densities it is the change-of-variables formula, which says that probability is conserved while density is squeezed or stretched. The same formula, read backwards, is how computers sample from any distribution and how normalizing flows compute likelihoods.

Notation. $Y = g(X)$. For an invertible $g$, $x = g^{-1}(y)$ is the preimage of $y$, and $g'$ the derivative. $f_X, F_X$ and $f_Y, F_Y$ are the densities and CDFs of $X$ and $Y$.

The discrete case#

If $X$ takes countably many values, so does $Y$, and $Y = y$ happens exactly when $X$ lands on one of the values that $g$ sends to $y$:

$$P(Y = y) = \sum_{x \,:\, g(x) = y} P(X = x) .$$

For a fair die $X$ and $Y = (X - 3)^2$, the value $Y = 1$ comes from $X = 2$ and $X = 4$, so $P(Y = 1) = 2/6$. Nothing more is needed: probabilities of values are carried along and added up where values merge.

The distribution function method#

For a continuous $X$ the safe general method goes through the CDF, because $\{Y \le y\}$ is an event about $X$:

$$F_Y(y) = P(g(X) \le y) ,$$

and the density of $Y$ is the derivative of the result.

A square. Let $X \sim U(-1, 1)$ and $Y = X^2$. For $0 \le y \le 1$, $Y \le y$ means $-\sqrt y \le X \le \sqrt y$, an interval of length $2\sqrt y$ out of 2:

$$F_Y(y) = \sqrt y, \qquad f_Y(y) = \frac{1}{2 \sqrt y} .$$

Half the time $X^2$ is below $1/4$, although $1/4$ is only a quarter of its range; the density of $Y$ is infinite at 0, where the parabola is flat and many values of $X$ crowd into a few values of $Y$.

A linear map. For $Y = aX + b$ with $a > 0$, $F_Y(y) = P(X \le (y - b)/a) = F_X((y - b)/a)$, so

$$f_Y(y) = \frac{1}{\lvert a \rvert} f_X\Big(\frac{y - b}{a}\Big) ,$$

and the same holds for $a < 0$. Applied to $X \sim N(\mu, \sigma^2)$ with $a = 1/\sigma$, $b = -\mu/\sigma$, it gives the density of $N(0, 1)$: the standardization $Z = (X - \mu)/\sigma$ promised in the previous note. The other direction also holds: $aX + b$ of a normal $X$ is normal, so a temperature of $N(20, 3^2)$ °C is one of $N(68, 5.4^2)$ °F.

The change-of-variables formula#

For a strictly increasing differentiable $g$, $F_Y(y) = F_X(g^{-1}(y))$, and differentiating gives the density of $Y$ directly; for a decreasing $g$ the derivative changes sign, which the absolute value absorbs:

$$f_Y(y) = \frac{f_X(x)}{\lvert g'(x) \rvert}, \qquad x = g^{-1}(y) .$$

The formula says that probability is conserved band by band. A short interval of length $dx$ around $x$ carries probability $f_X(x)\, dx$, and $g$ maps it to an interval of length $dy = \lvert g'(x) \rvert\, dx$ around $y$ that must carry the same probability, $f_Y(y)\, dy = f_X(x)\, dx$. Where $g$ squeezes ($\lvert g' \rvert < 1$), the density rises; where it stretches, the density falls:

Figure

Change of variables: probability is kept, density changes

The example is the log-normal distribution: $Y = e^X$ with $X \sim N(0, 1)$ has $g'(x) = e^x = y$ and

$$f_Y(y) = \frac{\varphi(\ln y)}{y}, \qquad y > 0 .$$

Monotone maps carry quantiles along, so the median of $Y$ is $e^0 = 1$ and $P(Y > e) = P(X > 1) = 0.159$. Quantities that are products of many positive factors, such as incomes, file sizes or response times, are often close to log-normal: their logarithm is a sum, close to normal for the reasons of the central limit theorem.

If $g$ is not monotone, every preimage contributes: $f_Y(y) = \sum_{x : g(x) = y} f_X(x) / \lvert g'(x) \rvert$. For $Y = X^2$ above, the two preimages $\pm\sqrt y$ each contribute $\frac{1/2}{2\sqrt y}$, which gives the same $1/(2\sqrt y)$.

Inverse transform sampling#

Read in reverse, the CDF method turns uniform random numbers into any distribution. If $U \sim U(0, 1)$ and $F$ is a continuous, strictly increasing CDF, then $X = F^{-1}(U)$ has CDF $F$, because

$$P(F^{-1}(U) \le x) = P(U \le F(x)) = F(x) .$$

For any CDF, with flat stretches or jumps, the same holds with the quantile function $Q(u) = \inf\{x : F(x) \ge u\}$ of the previous note in place of $F^{-1}$: $Q(u) \le x$ exactly when $u \le F(x)$, because $F$ is non-decreasing and continuous from the right. The discrete sampler below is this $Q$.

For the exponential distribution, $F(x) = 1 - e^{-x}$ and $F^{-1}(u) = -\ln(1 - u)$:

Figure

Inverse transform sampling: from a uniform u to an exponential x

The same idea samples a discrete distribution. For a categorical distribution with probabilities $p_1, \dots, p_K$, cut $[0, 1]$ into consecutive pieces of lengths $p_1, \dots, p_K$ and return the piece in which $U$ lands; that is one lookup in the cumulative sums. A language model samples its next token exactly this way, after temperature or top-$p$ have reshaped the probabilities.

The converse is the probability integral transform: if $X$ has a continuous CDF $F$, then $F(X) \sim U(0, 1)$. It is how probabilistic forecasts are checked, since the values $F(x)$ of a well-calibrated forecaster at the observed outcomes must look uniform, and it is why the p-value of a test whose statistic has a continuous distribution under the null hypothesis is uniformly distributed when that hypothesis is true. For a discrete statistic the p-value is only super-uniform, $P(p \le \alpha) \le \alpha$: in the two-sided test of 100 coin flips of the hypothesis tests note, a fair coin gives $p \le 0.05$ with probability $0.035$. The test stays valid, but it is conservative.

Normalizing flows in one dimension#

A normalizing flow models a complicated density as a simple one pushed through a chain of invertible maps, $Y = g_K(\cdots g_1(Z))$ with $Z \sim N(0, 1)$. Applying the change-of-variables formula once per map gives the exact log-density of a data point:

$$\log f_Y(y) = \log \varphi(z) - \sum_{k=1}^{K} \log \lvert g_k'(z_{k-1}) \rvert, \qquad z_0 = z = g^{-1}(y), \; z_k = g_k(z_{k-1}) .$$

Training maximizes this log-likelihood over the parameters of the maps. In more than one dimension the derivative becomes the determinant of the Jacobian, and making that determinant cheap is the design problem of the normalizing flows note in the generative models course.

What changed#

	Before	With the change-of-variables formula
Distribution of $g(X)$	simulate and look at a histogram	exact: $f_X(x) / \lvert g'(x) \rvert$, or $F_Y$ through $F_X$
Standardizing a normal	assumed	proved: $(X - \mu)/\sigma \sim N(0, 1)$
Sampling a distribution	needs a special trick per distribution	$F^{-1}(U)$ for any CDF; cumulative sums for a categorical
Density of a flow	—	exact log-likelihood, one $\log \lvert g' \rvert$ per map

Limitations#

• $F^{-1}$ is not always available in closed form. The normal CDF has no elementary inverse; samplers use other transformations (Box–Muller, which needs two variables) or numerical inversion.
• The formula needs a differentiable, locally invertible $g$. A function with flat pieces, such as rounding or clipping, puts positive probability on single values, and $Y$ is then not continuous: rounding makes it discrete, clipping makes it a mixture of masses and a density.
• Functions of several variables need their joint distribution. The distribution of the sum or the ratio of two random variables depends on how they depend on each other, the subject of the joint distributions note.

What you should remember#

• Discrete: $P(g(X) = y)$ sums $P(X = x)$ over the preimages of $y$.
• Continuous, general: compute $F_Y(y) = P(g(X) \le y)$, then differentiate.
• Continuous, monotone $g$: $f_Y(y) = f_X(x) / \lvert g'(x) \rvert$ at $x = g^{-1}(y)$; probability is conserved, density is squeezed or stretched.
• Linear maps keep normals normal; $(X - \mu)/\sigma \sim N(0, 1)$; $e^X$ of a normal is log-normal.
• $F^{-1}(U)$, in general $Q(U)$, has CDF $F$ (inverse transform sampling); $F(X)$ is uniform when $F$ is continuous (probability integral transform).

Code#

The two bands of the figure, the square of a uniform, and inverse transform sampling, checked numerically:

import numpy as np
from scipy import stats

rng = np.random.default_rng(0)
Z = stats.norm()

# Y = e^X for X ~ N(0, 1): each band keeps its probability over an interval of a different length
for a, b in [(-1.2, -0.7), (0.8, 1.3)]:
    p = Z.cdf(b) - Z.cdf(a)
    print(round(p, 3), round(np.exp(b) - np.exp(a), 3), round(p / (np.exp(b) - np.exp(a)), 3))
# 0.127 0.195 0.649   (squeezed: high density)
# 0.115 1.444 0.08    (stretched: low density)
print(round(Z.sf(1), 4), stats.lognorm(1).median())              # 0.1587 1.0

# Y = X^2 for X ~ U(-1, 1): P(Y <= 1/4) = sqrt(1/4)
y = rng.uniform(-1, 1, 1_000_000) ** 2
print(round((y <= 0.25).mean(), 3))                              # 0.5

# inverse transform: -ln(1 - U) is Exp(1)
u = np.array([0.1, 0.3, 0.5, 0.7, 0.9])
print((-np.log(1 - u)).round(3))                                 # [0.105 0.357 0.693 1.204 2.303]
x = -np.log(1 - rng.random(1_000_000))
print(round(x.mean(), 3), round((x > 2).mean(), 4))              # 1.0 0.1354; exactly 1 and e^-2 = 0.1353

# a categorical distribution sampled through cumulative sums, as for a next token
p = np.array([0.5, 0.3, 0.2])
k = np.searchsorted(np.cumsum(p), rng.random(1_000_000), side='right')
print((np.bincount(k) / 1_000_000).round(3))                     # [0.499 0.301 0.199]

# the probability integral transform: F(X) is uniform
print(np.quantile(Z.cdf(rng.normal(size=1_000_000)), [0.1, 0.5, 0.9]).round(3))   # [0.1   0.499 0.9  ]

Next#

A distribution is a whole function; often two or three numbers are enough to work with: where it is centred, how wide it is, how heavy its tails are. The next note defines the expectation and the variance, shows that the expectation of $g(X)$ can be computed without finding the distribution of $g(X)$ at all, and derives the $1/\sqrt n$ rule for averages from the opening note.

References#

• J. K. Blitzstein, J. Hwang. Introduction to Probability, 2nd ed. CRC Press, 2019. Chapter 5 (universality of the uniform), Chapter 8 (transformations, change of variables).
• L. Devroye. Non-Uniform Random Variate Generation. Springer, 1986. Chapter II (the inversion method).
• D. Rezende, S. Mohamed. Variational Inference with Normalizing Flows. ICML, 2015.