Inception Score

A GAN has no likelihood, a VAE’s likelihood bound comes with blurry samples, and a WGAN critic’s estimate of $W_1$ depends on the critic, so none of the training losses can rank finished generators. We need metrics computed from the images instead. This note and the next two cover three standard ones. The Inception Score (IS) and the Fréchet Inception Distance (FID) score a set of generated images; LPIPS is a perceptual distance between pairs of images, used as a loss and to measure the diversity of samples. The Inception Score (Salimans et al., 2016) uses only generated images, a pretrained classifier, and the entropy and KL divergence from Entropy, cross-entropy and KL divergence.

Notation. $x$ is a generated image and $y \in \{1, \dots, n\}$ is one of the $n$ classes of the classifier. $p(y \mid x)$ is the classifier’s softmax output for image $x$, and $\mathbb E_x$ is the expectation over images produced by the generator. Logarithms are natural unless a base is shown.

What the Inception Score measures#

The Inception Score rewards two properties at once:

• Variety. The images are of many different kinds, for example many different breeds of dog, not the same breed over and over.
• Distinctness. Each image distinctly looks like something: a classifier can say with confidence what it shows.

Neither property is enough alone. A generator that draws the same sharp beagle every time has distinct images and no variety. A generator that draws shapeless blobs has nothing a classifier can recognize.

The algorithm#

1. Take lots of generated images $x_1, \dots, x_N$.
2. Run them through a pretrained classifier, usually Inception v3 trained on ImageNet (Szegedy et al., 2016). The network is built from Inception modules and ends in a softmax, so for every image we get a class distribution $p(y \mid x)$.
3. Calculate two entropies from these outputs.
   • $H(p(y \mid x))$ should be low: the classifier makes a confident prediction, so the image distinctly resembles a certain class. This term enters the score with a minus sign.
   • $H(p(y))$ should be high: the images are diverse, so on average the classes are close to uniformly distributed.

Figure

Computing the Inception Score

Per-image (conditional) entropy#

$$H(p(y \mid x)) = -\sum_{y} p(y \mid x)\,\log p(y \mid x) .$$

A peaked $p(y \mid x)$ (one class near probability 1) has entropy near $0$. A flat one has the maximal entropy $\log n$.

Marginal entropy#

The marginal class distribution is the classifier output averaged over the generated images:

$$p(y) = \mathbb E_x\big[p(y \mid x)\big],$$

and its entropy is

$$H(p(y)) = -\sum_{y} p(y)\log p(y).$$

Why the marginal is an average of the outputs. Let $X \sim P_g$ be a random generated image, where $P_g$ is the distribution of the generator’s samples, and let $Y$ be the class that the classifier assigns to it: given $X = x$, $Y$ is drawn from $p(y \mid x)$.

Step 1. By definition, $p(y) = P(Y = y)$.

Step 2. Condition on the image (law of total probability):

$$p(y) = \mathbb E_{X \sim P_g}\big[P(Y = y \mid X)\big].$$

If the images took finitely many values, this would be the familiar sum $\sum_x P(Y = y \mid X = x)\,P_g(X = x)$. For a generator $x = G(z)$ it is $\mathbb E_z\big[P(Y = y \mid X = G(z))\big]$, an average over the noise (Salimans et al., 2016, §4). No density of $P_g$ is needed, only samples. This matters because the GAN generator is implicit: we can sample from it, but we cannot evaluate a density.

Step 3. The conditional probability is the classifier output, $P(Y = y \mid X = x) = p(y \mid x)$, so

$$p(y) = \mathbb E_x\big[p(y \mid x)\big].$$

In practice the expectation is empirical: the distribution of $X$ is uniform over the $N$ samples we took, so each image gets weight $1/N$ and

$$p(y) \approx \frac1N \sum_{i=1}^{N} p(y \mid x_i).$$

From two entropies to one KL divergence#

We want the marginal entropy to be large and the average conditional entropy to be small, so we look at their difference. It turns out to be an average KL divergence: $H(p(y)) - \mathbb E_x H(p(y \mid x)) = \mathbb E_x D_{\mathrm{KL}}(p(y \mid x)\,\|\,p(y))$. The five steps below prove it, followed by the linearity argument that step 2 uses. A reader who prefers numbers first can check the identity on the three-class example and return to the proof later.

1. Expand both entropies.

$$\begin{aligned} &H(p(y)) - \mathbb E_x\big[H(p(y \mid x))\big] \\ &\quad= \Big(-\sum_y p(y)\log p(y)\Big) \\ &\qquad - \mathbb E_x\Big[-\sum_y p(y \mid x)\log p(y \mid x)\Big] \\ &\quad= -\sum_y p(y)\log p(y) \\ &\qquad + \mathbb E_x\Big[\sum_y p(y \mid x)\log p(y \mid x)\Big]. \end{aligned}$$

2. Replace $p(y)$ by $\mathbb E_x[p(y \mid x)]$ in the weight of the first sum. The $p(y)$ inside the logarithm stays as it is. The factor $\log p(y)$ does not depend on $x$, so it can move inside the expectation, and the sum over $y$ can be swapped with $\mathbb E_x$ (see below):

$$\begin{aligned} &= -\sum_y \mathbb E_x\big[p(y \mid x)\big]\log p(y) \\ &\qquad + \mathbb E_x\Big[\sum_y p(y \mid x)\log p(y \mid x)\Big] \\ &= \mathbb E_x\Big[\sum_y p(y \mid x)\log p(y \mid x)\Big] \\ &\qquad - \mathbb E_x\Big[\sum_y p(y \mid x)\log p(y)\Big]. \end{aligned}$$

3. Factor out the shared expectation.

$$= \mathbb E_x\Big[\sum_y p(y \mid x)\big(\log p(y \mid x) - \log p(y)\big)\Big].$$

4. Recombine the logarithms.

$$= \mathbb E_x\Big[\sum_y p(y \mid x)\log\frac{p(y \mid x)}{p(y)}\Big].$$

5. Recognize the KL divergence. The inner sum is $D_{\mathrm{KL}}(p(y \mid x)\,\|\,p(y))$, so

$$\begin{aligned} &H(p(y)) - \mathbb E_x\big[H(p(y \mid x))\big] \\ &\quad= \mathbb E_x\Big[D_{\mathrm{KL}}\big(p(y \mid x)\,\|\,p(y)\big)\Big]. \end{aligned}$$

The left-hand side is the entropy of $Y$ minus its conditional entropy given $X$, which is the mutual information $I(X; Y)$ between a generated image and its predicted class. So $\log \mathrm{IS} = I(X; Y)$: the logarithm of the score measures how much a generated image tells us about its label (Barratt and Sharma, 2018, §3.3, eqs. 2–3 and Appendix A).

Why the sum and the expectation can be swapped. $\mathbb E_x[\cdot]$ is a linear operator and $\sum_y$ is a finite, discrete sum over classes. For any linear operator and a finite sum,

$$\sum_y \mathbb E_x\big[f_y(x)\big] = \mathbb E_x\Big[\sum_y f_y(x)\Big],$$

where $y$ is a class and $x$ is an image. This is linearity of expectation: the expectation of a sum is the sum of the expectations. Step 2 uses it with $f_y(x) = p(y \mid x)\log p(y)$.

Figure

What each term of the identity measures

The KL form gives a second reading of the score. We compare each image’s class distribution with the averaged (marginal) one. If an image’s predicted class distribution differs from the average one, its prediction differs from those of the other images. The divergence is large when $p(y \mid x)$ is sharp and lies on a class that the marginal does not concentrate on.

The score#

The Inception Score is the exponential of this average divergence:

$$\begin{aligned} \mathrm{IS} &= \exp\Big(\mathbb E_x\Big[D_{\mathrm{KL}}\big(p(y \mid x)\,\|\,p(y)\big)\Big]\Big) \\ &\approx \exp\Big(\frac1N\sum_{i=1}^{N} D_{\mathrm{KL}}\big(p(y \mid x_i)\,\|\,p(y)\big)\Big). \end{aligned}$$

Higher is better: more confident per-image predictions and more diversity across images.

The divergence is always finite. A KL divergence $D_{\mathrm{KL}}(p(y \mid x)\,\|\,p(y))$ is finite only if $p(y) > 0$ wherever $p(y \mid x) > 0$, and here this always holds: $p(y)$ is the average of the $p(y \mid x_i)$, so it is positive for every class that any image gives positive probability. In floating point a softmax output can still underflow to exactly 0, and the code must then use the convention $0 \log 0 = 0$ instead of computing it.

In code the estimator is a few lines. With the classifier outputs stacked into an array of shape $(N, n)$, it reproduces the worked example below ($2.02$ for generator A, $1.52$ for D). SciPy’s rel_entr computes $a \log(a/b)$ with $0 \log 0 = 0$, so exact zeros give no NaN. It is a tiny estimator for illustration, not a reference implementation: published scores also fix the network weights, the image preprocessing, the number of samples and the splits.

import numpy as np
from scipy.special import rel_entr

def inception_score(probs):
    # probs[i] is p(y | x_i), shape (N, n)
    assert np.allclose(probs.sum(axis=1), 1)  # rows are distributions
    p_y = probs.mean(axis=0)  # marginal p(y)
    # D_KL(p(y | x_i) || p(y)) for every i, with 0 log 0 = 0
    kl = rel_entr(probs, p_y).sum(axis=1)
    return np.exp(kl.mean())

A = np.array([[0.9, 0.05, 0.05],
              [0.05, 0.9, 0.05],
              [0.05, 0.05, 0.9]])
print(inception_score(A))  # 2.022...

Log base and range#

The base of the logarithm can be anything, as long as the exponentiation matches it. With $\log_e$ the score uses $\exp(\cdot)$; with $\log_2$ it uses $2^{(\cdot)}$. Both choices give the same score: a KL value in bits is the value in nats divided by $\ln 2$, and $2^{a/\ln 2} = e^{a}$.

The range of the score follows from three facts proved in Entropy, cross-entropy and KL divergence:

• $D_{\mathrm{KL}} \ge 0$ (why KL is non-negative), so $\mathbb E_x[D_{\mathrm{KL}}] \ge 0$.
• Entropies of discrete distributions are non-negative, $H(p(y \mid x)) \ge 0$. By the identity, $\mathbb E_x[D_{\mathrm{KL}}] = H(p(y)) - \mathbb E_x[H(p(y \mid x))] \le H(p(y))$.
• The marginal entropy is at most $\log n$, with equality only when $p(y)$ is uniform (Gibbs’ inequality against the uniform distribution).

Together,

$$\begin{gathered} 0 \le \mathbb E_x\big[D_{\mathrm{KL}}\big] \le H(p(y)) \le \log n \\ \Longrightarrow\quad 1 \le \mathrm{IS} \le n . \end{gathered}$$

In the language of mutual information this is $0 \le I(X; Y) \le H(Y) \le \log n$ (Barratt and Sharma, 2018, §3.3 and Appendix B). The maximum of the exponent is $\ln n$ in nats and $\log_2 n$ in bits.

$\mathrm{IS} = 1$ exactly when (almost) every image gets the same class distribution, because a KL divergence is zero only when its two arguments are equal. The common distribution can be peaked (mode collapse) or flat (no image is recognizable). $\mathrm{IS} = n$ would need every $H(p(y \mid x)) = 0$ and a uniform marginal, that is one-hot predictions spread evenly over all $n$ classes. A softmax never outputs an exact one-hot vector, so the score only approaches $n$. For Inception v3 on ImageNet, $n = 1000$.

A worked example with three classes#

Take a classifier with $n = 3$ classes (beagle, corgi, husky) and three generated images per generator. The columns of the table are the per-image outputs $p(y \mid x_i)$ for $i = 1, 2, 3$, the marginal $p(y)$, its entropy $H(p(y))$, the mean conditional entropy $\mathbb E_x H(p(y \mid x))$, the mean divergence $\mathbb E_x D_{\mathrm{KL}}$ and the score. Entropies and divergences are in nats; the score has no unit.

Generator	Per-image outputs	Marginal	Marginal entropy	Mean conditional entropy	Mean KL	IS
A: confident, diverse	$(0.9, 0.05, 0.05),$ $(0.05, 0.9, 0.05),$ $(0.05, 0.05, 0.9)$	$(\frac13, \frac13, \frac13)$	$1.099$	$0.394$	$0.704$	$2.02$
B: mode collapse	$(0.9, 0.05, 0.05)$ three times	$(0.9, 0.05, 0.05)$	$0.394$	$0.394$	$0$	$1.00$
C: not confident	$(\frac13, \frac13, \frac13)$ three times	$(\frac13, \frac13, \frac13)$	$1.099$	$1.099$	$0$	$1.00$
D: partial collapse	$(0.9, 0.05, 0.05)$ twice, $(0.05, 0.9, 0.05)$	$(0.617, 0.333, 0.05)$	$0.814$	$0.394$	$0.420$	$1.52$

For generator A, every image has

$$\begin{aligned} H(p(y \mid x)) &= -\big(0.9\ln 0.9 + 2\cdot 0.05\ln 0.05\big) \\ &\approx 0.0948 + 0.2996 \\ &= 0.394 \text{ nats}, \end{aligned}$$

and the marginal is uniform, so $H(p(y)) = \ln 3 \approx 1.099$. The difference is $0.704$, and $\mathrm{IS} = e^{0.704} \approx 2.02$. Because the marginal is uniform, each per-image divergence is $D_{\mathrm{KL}}(p(y \mid x_i)\,\|\,p(y)) = \ln 3 - H(p(y \mid x_i)) = 0.704$, so their mean matches the difference of entropies. In general only the mean has to match. In D the three divergences are $0.245$, $0.245$ and $0.768$ nats, and only their mean, $0.420$, equals $0.814 - 0.394$.

Generators B and C both score the minimum $1$, for opposite reasons. B is confident but always draws a beagle, so $p(y)$ is as peaked as each $p(y \mid x)$. C is “diverse” on average, but no single image looks like anything. Generator D drops the husky class and lands in between. With perfectly one-hot, evenly spread predictions the score would reach its maximum $n = 3$.

Figure

Three classes, four generators

Limitations#

The score has well-known blind spots (Barratt and Sharma, 2018; the section numbers below refer to that paper):

• It never looks at real data. A generator that produces one perfect image per class gets the maximum score while ignoring everything else in the data distribution, including variety within a class. A generator that memorizes training images scores as well as the data itself (§4.2.3), and the data distribution need not maximize the score at all: in a one-dimensional example with two overlapping classes, several other distributions score higher than the true one (§4).
• It can be gamed. Optimizing the score, directly or indirectly through model selection (early stopping, hyperparameter or architecture search), drifts toward adversarial examples. Small perturbations of WGAN samples reach $\mathrm{IS} \approx 900$ out of a possible 1000 while looking no more like natural images (§4.2.2, Figure 1).
• It depends on the classifier. The score is tied to the 1000 ImageNet classes and to a specific network and implementation. Keras and Torch versions of Inception v3 with almost the same accuracy give scores that differ by 3.5% on ImageNet validation images and by 11.5% on CIFAR-10 images (§4.1.1). The original code even uses a version of the network with 1008 outputs rather than 1000 (footnote 2). For data outside ImageNet’s domain, the classes are not meaningful (§4.2.1).
• It depends on the sample size and protocol. The usual protocol uses 50 000 images split into 10 groups of 5000 and reports the mean and standard deviation of the per-group scores. The marginal $p(y)$ is estimated per group, and the score changes with the number of groups, so scores computed differently are not comparable (§3.4, §4.1.2).

Notebook#

The metrics notebook of the HSE-LAMBDA Deep Generative Models repository (pinned commit; cell numbers count every cell, markdown included, from 1) introduces IS in cells 2–4: the purpose of the score, two illustrations of confident versus diverse predictions, the empirical formulas $p(y) \approx \frac1N\sum_i p(y \mid x_i)$ and $\mathrm{IS} \approx \exp\big(\frac1N\sum_i D_{\mathrm{KL}}\big)$, a remark that the score depends on the classifier and may be affected by domain mismatch, and the reading that each sample’s class distribution is compared with the averaged one. Cell 6 summarizes IS, FID and LPIPS in one comparison table. The notebook does not compute IS; its hands-on part is about LPIPS.

Next#

IS judges generated images by themselves. The Fréchet Inception Distance uses the same network, but compares the generated images with real ones.

References#

• T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, X. Chen. Improved techniques for training GANs. NeurIPS, 2016. §4: the Inception Score.
• C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, Z. Wojna. Rethinking the Inception architecture for computer vision. CVPR, 2016.
• S. Barratt, R. Sharma. A note on the Inception Score. ICML Workshop on Theoretical Foundations and Applications of Deep Generative Models, 2018. §3.3: the mutual-information reading; §4: issues with the score.
• HSE-LAMBDA, Deep Generative Models course repository: metrics.ipynb.