Joint distributions and independence

Data rarely come one number at a time. A patient has a blood pressure and an age, a house a price and an area, a training example a feature vector and a label, a sentence one word and the next. To ask how one of these quantities depends on another, the probabilities of all their combinations are needed: the joint distribution. This note defines it, recovers from it the distribution of each variable alone and of one variable given another, defines independence for random variables, and introduces the covariance and the correlation, the numbers that summarize how two variables move together, and do so only partly.

Notation. $p_{X,Y}(x, y) = P(X = x, Y = y)$ is a joint PMF and $f_{X,Y}(x, y)$ a joint density; $p_X$, $f_X$ are the marginal distributions; $p_{Y \mid X}(y \mid x)$ and $f_{Y \mid X}(y \mid x)$ the conditional ones. $\operatorname{Cov}(X, Y)$ is the covariance and $\rho(X, Y)$ the correlation.

Joint and marginal distributions#

For discrete variables the joint PMF is a table: one cell for every pair of values, the cells adding up to 1. Roll two dice and let $X$ be the first and $Y$ the larger of the two. The table, its row and column sums, and one of its rows renormalized, are the three objects of this note:

Figure

Joint, marginal and conditional distributions of two dice

The distribution of $X$ alone is recovered by summing over the values of $Y$, and that of $Y$ by summing over $X$:

$$p_X(x) = \sum_y p_{X,Y}(x, y), \qquad p_Y(y) = \sum_x p_{X,Y}(x, y) .$$

They are called marginal because they are written in the margins of the table. The marginals do not determine the joint distribution: variables $X'$ and $Y'$ drawn independently with the same marginals as $X$ and $Y$ have the same row and column sums and a completely different table, with no zeros in it.

For continuous variables the table becomes, when it exists, a joint density $f_{X,Y} \ge 0$ that integrates to 1 over the plane; probabilities are volumes, $P((X, Y) \in A) = \iint_A f_{X,Y}\, dx\, dy$, and marginals integrate the other variable out, $f_X(x) = \int f_{X,Y}(x, y)\, dy$. A point uniform in the unit disk has $f_{X,Y} = 1/\pi$ inside the disk. Its first coordinate is not uniform: integrating over the vertical chord at $x$ gives $f_X(x) = \frac{2}{\pi} \sqrt{1 - x^2}$, larger near the centre, where the chords are long. Continuous marginals do not guarantee a joint density. For a standard normal $Z$, both $Z$ and $Z^2$ are continuous, but the pair $(Z, Z^2)$ lies on a parabola, a curve of zero area, so no function on the plane integrates to its probabilities. The formulas of this note with $f_{X,Y}$ assume that a joint density exists.

Conditional distributions#

Fixing the value of one variable and renormalizing what is left gives the distribution of the other:

$$p_{Y \mid X}(y \mid x) = \frac{p_{X,Y}(x, y)}{p_X(x)}, \qquad f_{Y \mid X}(y \mid x) = \frac{f_{X,Y}(x, y)}{f_X(x)} .$$

Both ratios are defined where the denominator is positive, $p_X(x) > 0$ or $f_X(x) > 0$. The continuous version is determined only up to the values of $x$ in a set of probability zero, which changes no probability computed from it. In the table, $X = 3$ leaves row 3, and $Y$ equals 3 with probability $3/6$ and each of 4, 5, 6 with probability $1/6$. The continuous version answers the question left open by the conditional probability note: conditioning on $X = x$, an event of probability zero, is defined through densities. For the point in the disk, given $X = x$ the coordinate $Y$ is uniform on the chord $[-\sqrt{1 - x^2}, \sqrt{1 - x^2}]$. Rearranged, the definition is the chain rule for distributions, $f_{X,Y}(x, y) = f_X(x)\, f_{Y \mid X}(y \mid x)$: a generative model often specifies a joint distribution exactly this way, a distribution of $x$ and a conditional of $y$ given $x$.

Independence of random variables#

$X$ and $Y$ are independent if every event about $X$ is independent of every event about $Y$, which is equivalent to the joint distribution being the product of the marginals:

$$p_{X,Y}(x, y) = p_X(x)\, p_Y(y) \quad \text{for all } x, y ,$$

or the same with densities. Equivalently, every conditional distribution of $Y$ given $X = x$ equals the marginal of $Y$: learning $X$ says nothing about $Y$. The first die and the larger value are dependent; already $P(X = 3, Y = 1) = 0$, while $P(X = 3)\, P(Y = 1) = \frac16 \cdot \frac1{36}$. The coordinates of a point in the disk are dependent too: if $X$ is close to 1, $Y$ must be close to 0.

For independent variables, expectations of products factorize: $E[g(X)\, h(Y)] = E[g(X)]\, E[h(Y)]$, since the double sum or integral splits into a product. This is the step used in the previous note to multiply moment generating functions.

Covariance and correlation#

The covariance measures whether two variables tend to be above their means together:

$$\operatorname{Cov}(X, Y) = E\big[(X - \mu_X)(Y - \mu_Y)\big] = E[XY] - E[X]\, E[Y] .$$

It exists when both variances are finite. It is positive when large values of $X$ come with large values of $Y$, negative when they come with small ones. It is linear in each argument, and $\operatorname{Cov}(X, X) = \operatorname{Var}(X)$. Expanding the square of a sum gives the rule promised in the previous note:

$$\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) + 2 \operatorname{Cov}(X, Y) ,$$

so variances add exactly when the covariance is zero, in particular for independent variables, whose covariance is $E[X]E[Y] - E[X]E[Y] = 0$.

The covariance has the units of $X$ times those of $Y$. Dividing by the standard deviations gives the unit-free correlation

$$\rho(X, Y) = \frac{\operatorname{Cov}(X, Y)}{\sigma_X \sigma_Y} \in [-1, 1] ,$$

defined when $0 < \operatorname{Var}(X) < \infty$ and $0 < \operatorname{Var}(Y) < \infty$, so not for a constant. It is $\pm 1$ only if $Y = aX + b$ exactly. For the dice, $E[Y] = 161/36$, $\operatorname{Cov}(X, Y) = 35/24$ and $\rho = 0.61$.

Independence implies zero correlation; the converse is false. If $Z \sim N(0, 1)$, then $\operatorname{Cov}(Z, Z^2) = E[Z^3] = 0$, although $Z^2$ is a function of $Z$. The correlation sees only the linear part of a dependence:

Figure

Correlation measures only linear dependence

Two more warnings. A correlation computed from pooled groups can differ in sign from the correlation inside each group, as in Simpson’s paradox of the opening note. And correlation is not causation: ice cream sales and drownings are correlated because both rise in summer.

Averaging correlated estimates#

Covariance explains why averaging does not always help. Take $n$ estimates of the same quantity, each with variance $\sigma^2$ and every pair with correlation $\rho$: models in an ensemble, measurements by the same instrument, annotations by overlapping annotators. The variance of their average is

$$\operatorname{Var}\Big(\frac1n \sum_{i=1}^n X_i\Big) = \frac{1}{n^2}\Big(n \sigma^2 + n(n-1) \rho \sigma^2\Big) = \rho \sigma^2 + \frac{(1 - \rho)\, \sigma^2}{n} .$$

The second term vanishes as $n$ grows; the first does not. With $\rho = 0.5$, averaging 10 estimates leaves $0.55\sigma^2$ and averaging 100 leaves $0.505\sigma^2$: the floor is $\rho\sigma^2$. Ensembles of nearly identical models gain almost nothing, which is why random forests decorrelate their trees by training each on a bootstrap sample and choosing every split among a random subset of the features. The same formula is why many examples from the same user, or near-duplicates in a test set, shrink the error bar far less than their count suggests.

Sums of independent variables#

The distribution of $X + Y$ for independent $X$ and $Y$ collects every way to reach each total:

$$P(X + Y = z) = \sum_x P(X = x)\, P(Y = z - x), \qquad f_{X+Y}(z) = \int f_X(x)\, f_Y(z - x)\, dx .$$

This operation is the convolution of the two distributions. Two dice give the triangular PMF of the sum; two independent $U(0, 1)$ give the triangular density on $[0, 2]$, with $P(U_1 + U_2 \le 0.5) = 0.5^2/2 = 0.125$. Convolving again and again makes the shape smoother and more bell-like, which is what the central limit theorem makes precise.

What changed#

	One variable at a time	Joint distribution
Describes	each variable separately	all combinations of values
Recovers		marginals by summing out, conditionals by renormalizing
Dependence	invisible	independence: joint = product of marginals; covariance and correlation summarize the linear part
Variance of a sum	added, assuming independence	$\operatorname{Var}(X) + \operatorname{Var}(Y) + 2\operatorname{Cov}(X, Y)$
Average of correlated estimates	$\sigma^2/n$	$\rho\sigma^2 + (1 - \rho)\sigma^2/n$

Limitations#

• Joint distributions grow fast. Ten binary variables have $2^{10}$ cells, a hundred have more than $10^{30}$; real models assume structure, such as conditional independence in naive Bayes or graphical models, or a parametric family such as the multivariate Gaussian.
• Correlation is a summary of linear dependence. It can be zero when one variable completely determines the other, and it is sensitive to outliers; mutual information or rank correlations capture more.
• Neither correlation nor any joint distribution reveals causation. The same joint distribution of observational data is compatible with different causal structures.

What you should remember#

• The joint distribution gives the probability of every combination of values; marginals sum or integrate the other variables out; conditionals renormalize a slice.
• $X$ and $Y$ are independent if the joint distribution is the product of the marginals; then $E[g(X)h(Y)] = E[g(X)]E[h(Y)]$.
• $\operatorname{Cov}(X, Y) = E[XY] - E[X]E[Y]$; $\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) + 2\operatorname{Cov}(X, Y)$; $\rho = \operatorname{Cov}/(\sigma_X\sigma_Y) \in [-1, 1]$.
• Independent implies uncorrelated, not the other way round: $Z$ and $Z^2$ are uncorrelated.
• The average of $n$ estimates with pairwise correlation $\rho$ has variance $\rho\sigma^2 + (1 - \rho)\sigma^2/n$.
• The distribution of a sum of independent variables is the convolution of their distributions.

Code#

The table of the figure in exact fractions, and the other numbers by simulation:

import numpy as np
from fractions import Fraction

# two dice: X the first, Y the larger; the joint PMF as a 6 x 6 table of fractions
J = np.full((6, 6), Fraction(0), dtype=object)
for a in range(1, 7):
    for b in range(1, 7):
        J[a - 1, max(a, b) - 1] += Fraction(1, 36)
pX, pY = J.sum(axis=1), J.sum(axis=0)
print([str(p) for p in pX], [int(p * 36) for p in pY])   # ['1/6', ...] [1, 3, 5, 7, 9, 11]
print([str(v / pX[2]) for v in J[2]])                    # ['0', '0', '1/2', '1/6', '1/6', '1/6']

# covariance and correlation of X and Y
v = np.arange(1, 7)
EX, EY = (pX * v).sum(), (pY * v).sum()
EXY = sum(J[i, j] * v[i] * v[j] for i in range(6) for j in range(6))
cov = EXY - EX * EY
varX, varY = (pX * v**2).sum() - EX**2, (pY * v**2).sum() - EY**2
print(EY, cov, round(float(cov) / float(varX * varY) ** 0.5, 3))   # 161/36 35/24 0.608

# uncorrelated but dependent: Z and Z^2
rng = np.random.default_rng(0)
z = rng.normal(size=1_000_000)
print(round(float(np.corrcoef(z, z**2)[0, 1]), 3))                 # 0.0

# the average of n estimates with variance 1 and pairwise correlation rho
for rho in (0.0, 0.5, 0.9):
    print(rho, [round(rho + (1 - rho) / n, 3) for n in (1, 10, 100)])
# 0.0 [1.0, 0.1, 0.01]   0.5 [1.0, 0.55, 0.505]   0.9 [1.0, 0.91, 0.901]
shared, own = rng.normal(size=(100_000, 1)), rng.normal(size=(100_000, 10))
estimates = np.sqrt(0.5) * shared + np.sqrt(0.5) * own              # pairwise correlation 0.5
print(round(float(estimates.mean(axis=1).var()), 3))               # 0.547, against 0.55

# the sum of two independent uniforms has a triangular density
print(round(float((rng.random((2, 1_000_000)).sum(axis=0) <= 0.5).mean()), 4))   # 0.125

Next#

With two variables the covariance is one number; with $d$ variables it is a $d \times d$ matrix. One family of joint distributions is completely described by its means and that matrix, keeps its form under linear maps, marginalization and conditioning, and sits under linear regression, Gaussian processes and the latent spaces of generative models: the multivariate Gaussian.

References#

• J. K. Blitzstein, J. Hwang. Introduction to Probability, 2nd ed. CRC Press, 2019. Chapter 7 (joint, marginal and conditional distributions, covariance and correlation), Chapter 8 (convolutions).
• W. Feller. An Introduction to Probability Theory and Its Applications, Vol. 1, 3rd ed. Wiley, 1968. Chapter IX (joint distributions, covariance, correlation).
• L. Breiman. Random Forests. Machine Learning, 45(1), 2001. The correlation between trees and the error of the forest.