Precision and recall for generative models

The FID note ended with a warning: one number scores two different properties. A generator can fail by producing images that do not look like the data, or by producing only part of what the data contains, for example faces of one age or one pose. Both failures raise FID, and two models that fail in opposite ways can get the same score. This note splits the score in two, borrowing the words from classification: precision is the share of generated samples that look like real data, and recall is the share of the real data the model can produce.

Notation. $P$ is the real distribution and $Q$ the model, as in the rest of the course. Both are compared in a feature space: $\varphi(x)$ is the embedding of an image by a pretrained network. $X = \{x_1, \dots, x_N\}$ are real samples and $Y = \{y_1, \dots, y_M\}$ generated ones, already embedded. $B(c, r)$ is the ball of radius $r$ around $c$, and $\mathrm{NND}_k(x)$ the distance from $x$ to its $k$-th nearest neighbour in its own set.

Two failures, one number#

Take real data $P = \mathcal N(0, I)$ in two dimensions and models $Q_s = \mathcal N(0, s^2 I)$. FID between two Gaussians is $\lVert\mu_P - \mu_Q\rVert^2 + \mathrm{Tr}\big(\Sigma_P + \Sigma_Q - 2(\Sigma_P\Sigma_Q)^{1/2}\big)$, and here

$$\mathrm{FID}(P, Q_s) = 0 + \mathrm{Tr}\big((1 + s^2 - 2s)\, I\big) = 2(1 - s)^2 .$$

The value depends only on $\lvert 1 - s\rvert$, so $s = 0.3$ and $s = 1.7$ score the same, $2 \cdot 0.7^2 = 0.98$. Yet the first model is collapsed: its samples are plausible points of $P$, but it rarely reaches the outer two thirds of the data. The disc of radius 0.90 holds the inner third of $P$, and only $e^{-0.81/0.18} = e^{-4.5} \approx 1.1\%$ of the model’s samples fall outside it. The second is spread out: it reaches everything, but many of its samples land where real data almost never are. A metric that can say which of the two happened needs two numbers.

Precision and recall of distributions#

The idealized definition compares supports, the regions where each distribution puts mass:

$$\text{precision} = Q\big(\mathrm{supp}\,P\big), \qquad \text{recall} = P\big(\mathrm{supp}\,Q\big).$$

Precision is the probability that a generated sample falls where real data can occur; recall is the probability that a real sample falls where the model can generate. Taken literally, exact supports cannot tell the two Gaussians above apart. A nondegenerate Gaussian has positive density on all of $\mathbb R^d$, so $\mathrm{supp}\,P = \mathrm{supp}\,Q_s = \mathbb R^2$, and precision and recall are both exactly 1 for every $s$. The definition is useful as a picture of something weaker: the region where each distribution puts most of its mass. Read that way, a collapsed model has precision near 1 and low recall, and a model that puts mass everywhere has recall near 1 and low precision. Three different objects carry this idea, and they should not be confused: the exact supports above; the mass decomposition of the PRD curve, which asks how much of each distribution is shared; and the finite-sample estimates, which cover only the region that the samples actually reach. The numbers later in this note are values of the estimators. The collapsed model gets a low estimated recall because almost all of its 2000 samples lie within radius 0.9, so the small balls around them miss most real points farther out, although its exact support is the whole plane. The supports, or the high-mass regions, have to be estimated from samples, in a feature space where distances mean something. The two methods below do this differently.

The PRD curve#

Sajjadi et al. (2018) define precision and recall through mixtures. $Q$ has precision $\alpha$ at recall $\beta$ with respect to $P$ if both share a common component $\mu$,

$$P = \beta\,\mu + (1 - \beta)\,\nu_P, \qquad Q = \alpha\,\mu + (1 - \alpha)\,\nu_Q ,$$

so that a fraction $\beta$ of the data is covered by the shared part and a fraction $\alpha$ of the model’s mass lies in it. For distributions on a finite set the attainable pairs form a curve, traced by $\lambda > 0$:

$$\alpha(\lambda) = \sum_\omega \min\big(\lambda P(\omega),\, Q(\omega)\big), \qquad \beta(\lambda) = \sum_\omega \min\Big(P(\omega),\, \frac{Q(\omega)}{\lambda}\Big).$$

To apply it to images, the real and generated embeddings (Inception features in the paper) are clustered together with k-means into 20 clusters, and $P$ and $Q$ become the two histograms over the clusters; the clustering is randomized, so the curve is averaged over several runs. Each curve is summarized by two numbers, the largest values along it of $F_b = (1 + b^2)\,\alpha\beta / (b^2 \alpha + \beta)$ for $b = 8$ and $b = 1/8$, with precision $\alpha$ and recall $\beta$. $F_8$ emphasizes recall and $F_{1/8}$ precision; the coefficient in the formula is $b^2 = 64$, so “eight times” describes the conventional relative importance, not a literal weight.

Small example. Two clusters, $P = (0.5, 0.5)$ and a collapsed model $Q = (1, 0)$. Then $\alpha(\lambda) = \min(0.5\lambda, 1)$ and $\beta(\lambda) = \min(0.5, 1/\lambda)$. For $\lambda \le 2$ the curve runs at recall 0.5 up to precision 1; beyond, precision stays 1 and recall falls. The model reaches precision 1, every sample is in a cluster the data use, but its recall never exceeds 0.5: half of the data, the second cluster, is out of its reach.

Nearest neighbours as a support estimate#

Clusters are coarse: with 20 of them, a model that covers each cluster but only part of it looks perfect. Kynkäänniemi et al. (2019) estimate each support directly as a union of balls. Around every real sample put a ball reaching to its $k$-th nearest real neighbour; the union of these balls is the estimate of where real data live. With

$$f(\varphi, \Phi) = \begin{cases} 1, & \varphi \in B\big(\varphi', \mathrm{NND}_k(\varphi')\big) \text{ for some } \varphi' \in \Phi, \\ 0, & \text{otherwise,} \end{cases}$$

precision is the share of generated samples inside the real estimate and recall the share of real samples inside the generated estimate:

$$\text{precision} = \frac{1}{M}\sum_{j=1}^M f(y_j, X), \qquad \text{recall} = \frac{1}{N}\sum_{i=1}^N f(x_i, Y).$$

The paper uses $k = 3$, 50,000 samples on each side and features from the second fully connected layer of a VGG-16 classifier. Ball radii adapt to the local density: in dense regions the balls are small, in sparse regions large.

Worked example: one dimension, k = 1#

Real samples $X = \{0, 1, 2, 3\}$, generated samples $Y = \{0.2, 0.8, 1.1, 7\}$, $k = 1$. Every real sample has a neighbour at distance 1, so each real ball has radius 1, and their union is $[-1, 4]$. Three generated samples lie inside it and 7 does not: precision is 3/4. The generated radii are the distances to the nearest generated neighbour: 0.6 for 0.2, 0.3 for 0.8 and for 1.1, and 5.9 for 7. Real 0 and 1 are covered by the small balls around 0.2 and 0.8. Real 2 is 0.9 away from 1.1, outside its ball of radius 0.3, and real 3 is far from all three, yet both lie within 5.9 of the outlier 7. Recall is 4/4. Remove the outlier and recall drops to 2/4: the model’s actual samples sit between 0.2 and 1.1 and say nothing about 2 or 3.

Figure

k-NN balls, and an outlier

The example shows the weakness of the estimate. An isolated sample has a large distance to its neighbours and so a large ball, and that ball claims a whole region of space for its distribution. A single real outlier inflates the real support and raises precision; a single generated outlier raises recall. With images, outliers are not rare: they are exactly the unusual samples a generator produces when it fails.

Density and coverage#

Naeem et al. (2020) keep the balls around real samples only, where outliers are rarer and the data are trusted, and count rather than test:

$$\text{density} = \frac{1}{kM}\sum_{j=1}^M \sum_{i=1}^N \mathbb 1\big[y_j \in B(x_i, \mathrm{NND}_k(x_i))\big], \qquad \text{coverage} = \frac{1}{N}\sum_{i=1}^N \mathbb 1\big[\exists\, j:\ y_j \in B(x_i, \mathrm{NND}_k(x_i))\big].$$

Density replaces precision: instead of asking whether a generated sample is in some real ball, it counts in how many, divided by $k$. When the model equals the data its expected value is exactly 1, whatever the distribution or dimension; values below 1 mean samples in sparse or empty regions, values above 1 samples crowded into the densest regions. Coverage replaces recall: the share of real samples whose own ball contains at least one generated sample. A generated outlier cannot raise it, because it can fall only into balls that some real sample owns. The paper recommends $k = 5$ with 10,000 samples on each side, which gives an expected coverage of about 0.97 for identical distributions.

In the one-dimensional example with $k = 1$, the generated samples 0.2, 0.8 and 1.1 fall into 2 real balls each and 7 into none, so density is $6/(1 \cdot 4) = 1.5$: the model piles its samples into the left part of the data. Coverage is 3/4: the balls of 0, 1 and 2 contain a generated sample and the ball of 3 does not. Removing the outlier leaves coverage at 3/4, where recall had jumped between 4/4 and 2/4.

The two Gaussians again#

The simulation behind the figure draws 2000 real points from $\mathcal N(0, I)$ and 2000 from each model, with seeded random numbers, and computes the scores with $k = 3$ for precision and recall and $k = 5$ for density and coverage. All of them are finite-sample estimates; with exact supports precision and recall would be 1 for all three models:

Model	FID	Precision	Recall	Density	Coverage
collapsed, $s = 0.3$	0.98	0.98	0.48	0.97	0.37
matched, $s = 1$	0	0.97	0.98	1.00	0.98
spread out, $s = 1.7$	0.98	0.88	0.99	0.81	0.90

Figure

The same FID, opposite precision and recall

Precision and recall separate the two models that FID could not: the collapsed one keeps its precision and loses half its recall, the spread-out one keeps its recall and loses precision. The loss of precision is smaller than the picture suggests, 0.88, because the real balls at the edge of the data are large and accept many of the far samples; density, 0.81, reports the same failure more clearly. Coverage reports the collapse most clearly, 0.37: only a third of the real points have a generated sample in their own neighbourhood.

Where the trade-off appears#

Many generators have a knob that trades one property for the other. The truncation trick samples the latent code of a GAN from a narrowed distribution: BigGAN resamples latent values outside a threshold (Brock, Donahue and Simonyan, 2019), StyleGAN pulls the intermediate latent toward its average. Narrower codes give more typical images, so precision rises while recall falls; Kynkäänniemi et al. (2019) trace this curve for StyleGAN, where FID gives a single number per setting of the knob. Precision and recall are therefore reported as a pair, or as a curve over the knob, never averaged into one number.

What changed#

	FID	PRD	k-NN precision / recall	Density / coverage
Output	one distance	a curve, summarized by $F_8$, $F_{1/8}$	two numbers	two numbers
Model of the data	one Gaussian in feature space	histogram over 20 clusters	union of k-NN balls	union of k-NN balls around real samples
Separates fidelity from coverage	no	yes	yes	yes
Sensitive to outliers	through means and covariances	little	yes, both sides	much less
Cost	linear in the samples	k-means	all pairwise distances	real-to-all distances

Limitations#

• Everything happens in a feature space. Inception or VGG features trained on ImageNet decide what counts as similar; Kynkäänniemi et al. (2023) show that FID responds to the ImageNet classes of the images, and the same caution applies here.
• The estimates depend on $k$ and on the sample size. Small $k$ or few samples underestimate the supports and lower both precision and recall even for a perfect model; the published defaults ($k = 3$ with 50,000 samples, $k = 5$ with 10,000) are calibrated for that.
• High-dimensional features. Nearest-neighbour distances concentrate as the dimension grows, so the balls describe the support less sharply than in the two-dimensional picture.
• Two numbers are not a ranking. A model with higher precision and lower recall is not better or worse; which matters depends on the use.
• Cost. All pairwise distances between 50,000 and 50,000 samples are $2.5 \cdot 10^9$ distances per evaluation.

Summary#

• One number cannot tell a collapsed model from a spread-out one: two Gaussians with $s = 0.3$ and $s = 1.7$ have the same FID, 0.98.
• Precision is the share of generated samples in the region where real data concentrate; recall is the share of real samples in the region the model concentrates on. With exact supports both are 1 for any two nondegenerate Gaussians, so the useful numbers are finite-sample estimates.
• PRD estimates both from histograms over clusters; the k-NN method estimates supports as unions of balls reaching to the $k$-th neighbour, and is fooled by outliers.
• Density counts real balls around generated samples and equals 1 in expectation for a perfect model; coverage is the share of real samples whose ball holds a generated sample. Both use balls around real samples only, which makes them robust to generated outliers.

Code#

No course notebook covers these metrics. A minimal implementation in NumPy, for embeddings X (real) and Y (generated) of shape [n, d]:

import numpy as np

def kth_radius(A, k):
    d = np.linalg.norm(A[:, None] - A[None], axis=-1)
    return np.sort(d, axis=1)[:, k]          # column 0 is the point itself

def in_balls(B, A, r):                       # [len(B), len(A)]: is b in the ball around a?
    return np.linalg.norm(B[:, None] - A[None], axis=-1) <= r[None]

def precision_recall(X, Y, k=3):
    return in_balls(Y, X, kth_radius(X, k)).any(1).mean(), in_balls(X, Y, kth_radius(Y, k)).any(1).mean()

def density_coverage(X, Y, k=5):
    hits = in_balls(Y, X, kth_radius(X, k))
    return hits.sum() / (k * len(Y)), hits.any(0).mean()

With X = rng.standard_normal((2000, 2)) and Y = 0.3 * rng.standard_normal((2000, 2)) it reproduces the first row of the table up to sampling noise of about 0.02. The full pairwise distance matrix is fine for a few thousand samples; for 50,000 the distances are computed in blocks.

Next#

Precision, recall, FID and IS all replace a person looking at images by a network looking at features. The next note returns to the person: how to ask people about generated images so that their answers are reliable, and how to measure that reliability.

References#

• M. S. M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, S. Gelly. Assessing Generative Models via Precision and Recall. NeurIPS, 2018.
• T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen, T. Aila. Improved Precision and Recall Metric for Assessing Generative Models. NeurIPS, 2019.
• M. F. Naeem, S. J. Oh, Y. Uh, Y. Choi, J. Yoo. Reliable Fidelity and Diversity Metrics for Generative Models. ICML, 2020. Density and coverage.
• M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, S. Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. NeurIPS, 2017. FID.
• A. Brock, J. Donahue, K. Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis. ICLR, 2019. The truncation trick.
• T. Kynkäänniemi, T. Karras, M. Aittala, T. Aila, J. Lehtinen. The Role of ImageNet Classes in Fréchet Inception Distance. ICLR, 2023.