Human evaluation of generative models

IS, FID, precision and recall all replace a person looking at images by a network looking at features. They are cheap and reproducible, and they inherit the blind spots of the network: a feature extractor trained to classify ImageNet does not see everything a person sees. The previous note ended there. This one returns to the person. The quality of a generated image is ultimately what people perceive, so human judgement is the reference that automatic metrics are validated against. It is also noisy, expensive and easy to collect badly. The note covers what to ask, how to turn the answers into numbers, and how many answers a conclusion needs.

Notation. $n$ judgements, $k$ of them of a given kind (for example errors), and the observed rate $\hat p = k / n$. In a pairwise comparison $A \succ B$ means that A is preferred to B, and $s_A$ is the score of option A.

What can be asked#

Four questions cover most human evaluations of generative models:

Question	What the rater sees	Number reported	Measures
Real or generated?	one image, or a real and a generated one	fool rate, or error rate on a balanced set	realism
Which of two is better?	outputs of two models for the same input	win rate, Bradley–Terry or Elo scores	relative preference
How good is it?	one output	mean rating on a fixed scale (MOS)	absolute quality
Does it match the input?	a prompt and an output	share of matching outputs, or a win rate	alignment to the condition

The first question asks the rater to detect something; the others ask for an opinion. Detection has a correct answer for every trial, which makes it easier to check and to calibrate.

Real or generated#

Show an image and ask whether it is real. The fool rate is the share of generated images judged real. If people cannot tell and guess with even odds, it approaches 50%. But it counts only generated images, so it also depends on how readily a rater says “real”: a rater who calls every image real reaches a fool rate of 100% without looking. A fake-only comparison with 50% therefore assumes unbiased guessing. Mixing in an equal number of real images and scoring the error rate over all of them removes this response bias: the always-”real” rater is then wrong on every generated image and right on every real one, an error rate of exactly 50%. The two numbers are different and should not be mixed. An early use is LAPGAN (Denton et al., 2015): its CIFAR-10 samples were mistaken for real about 40% of the time, against about 10% for a GAN baseline.

HYPE (Zhou et al., 2019) standardizes the protocol. Each rater sees 100 images, 50 real and 50 generated, in random order, and labels each one. HYPE∞ is the error rate over all of them with unlimited viewing time: misjudged generated images plus misjudged real ones, divided by 100. 50% means the rater is at chance; above 50% the generated images look more real than the real ones. HYPE time shows each image only briefly, between 100 and 1000 ms, and adapts the exposure with a staircase: correct answers shorten it, wrong answers lengthen it; the score is the exposure time raters need to tell generated from real, so longer is more realistic. A run uses 30 raters. In the paper, StyleGAN on FFHQ sampled with the truncation trick reaches a HYPE∞ of 27.6%, against 19.0% without truncation: truncation trades diversity for typical, realistic faces, the precision side of the trade-off from the previous note.

Worked example: is 38% different from chance?#

In a balanced test, 100 real and 100 generated images, raters make 76 errors in 200 judgements, so the error rate is $\hat p = 0.38$. The usual interval $\hat p \pm 1.96\sqrt{\hat p(1 - \hat p)/n}$ misbehaves near 0 and 1 and for small $n$; the Wilson interval (Wilson, 1927) inverts the score test instead:

$$\frac{\hat p + \frac{z^2}{2n}}{1 + \frac{z^2}{n}} \;\pm\; \frac{z}{1 + \frac{z^2}{n}} \sqrt{\frac{\hat p(1 - \hat p)}{n} + \frac{z^2}{4n^2}}, \qquad z = 1.96 .$$

Here the centre is $(0.38 + 0.0096)/1.0192 = 0.382$ and the half-width $1.923 \cdot \sqrt{0.001178 + 0.000024} = 0.067$, so the 95% interval is $[0.316, 0.449]$. It excludes 0.5: raters can tell these images from real ones. An exact binomial test of 76 out of 200 against $p = 0.5$ gives $p \approx 0.0008$. Both calculations treat the 200 judgements as independent trials with one common error probability. Judgements by the same rater, or about the same image, are correlated, so in a real study the interval is wider: it should come from resampling raters and images as clusters, or from a model with rater and item effects. The numbers here are an idealized baseline. The width matters as much as the centre. It shrinks like $1/\sqrt n$:

Figure

How many judgements an answer of ±0.05 takes

How many judgements a comparison needs#

To tell a model with a true balanced error rate of 0.45 from one that is indistinguishable from data (0.5), with a two-sided test at the 5% level and a probability of 0.8 of detecting the difference, the sample size is

$$n = \frac{\big(z_{0.975}\sqrt{p_0 q_0} + z_{0.8}\sqrt{p_1 q_1}\big)^2}{(p_1 - p_0)^2} = \frac{\big(1.96 \cdot 0.5 + 0.842 \cdot 0.497\big)^2}{0.05^2} \approx 783,$$

with $p_0 = 0.5$, $p_1 = 0.45$ and $q = 1 - p$. A study of 100 judgements per model can detect only large differences; small ones need hundreds to thousands. The formula again assumes independent judgements; the judgements of one rater, or about one image, are not independent, which raises the number further. So 783 is an idealized baseline, not a guaranteed study size.

Which of two is better#

Absolute questions are hard to answer consistently; comparisons are easier. Show two outputs for the same input, one from each model, and ask which is better. Many such votes over many pairs of models are summarized by the Bradley–Terry model (Bradley and Terry, 1952): every option has a score, and

$$P(A \succ B) = \frac{1}{1 + e^{-(s_A - s_B)}} .$$

The scores are fitted by maximum likelihood over all observed votes, so models that were never compared directly are placed on one scale through the models they share. The Elo scale is the same model in other units: with ratings $R = 400\, s / \ln 10$,

$$P(A \succ B) = \frac{1}{1 + 10^{-(R_A - R_B)/400}} .$$

Worked example. Model A wins 60% of its comparisons against B. Then $10^{d/400} = 0.6 / 0.4 = 1.5$ and $d = 400 \log_{10} 1.5 = 70.4$ Elo points. If B beats C by the same margin, A is $140.9$ points above C, and the model predicts that A beats C with probability $1.5^2/(1.5^2 + 1) = 0.692$, although no one compared them. This transitivity is an assumption of the model: if votes contradict it, as with rock–paper–scissors preferences, the fit is poor and the scores mislead.

Figure

The Bradley–Terry model on the Elo scale

Public arenas apply this at scale: users submit a prompt, see two anonymous outputs, vote, and the scores are refitted. Chiang et al. (2024) describe Chatbot Arena for language models; GenAI-Arena (Jiang et al., 2024) does the same for image and video generation. The same model, fitted to the preferences of labellers, is the reward model of RLHF, and the LLM evaluation note discusses arenas for text.

How good, and does it match#

Absolute ratings. The rater scores one output on a fixed scale, usually 1 to 5, and the mean opinion score (MOS) averages the ratings. The protocol comes from speech quality testing (ITU-T P.800) and is standard for speech synthesis. Its weakness is the scale itself: raters use it differently, some never give a 5, and anchors drift during a session, so MOS values are comparable only within one study with the same raters and instructions.

Alignment to the condition. For conditional models the question is not only whether the output looks good, but whether it is what was asked for. Text-to-image benchmarks fix a set of prompts that probe known weaknesses and ask raters to compare two models on image quality and on prompt alignment separately: DrawBench has 200 prompts in 11 categories (Saharia et al., 2022), PartiPrompts more than 1,600 (Yu et al., 2022). Counting objects, spatial relations and rendered text are where models most often disagree with the prompt.

Making judgements reliable#

Most of the work is in the design, not in the statistics:

• Blind and randomized. Raters do not know which model produced an output; the order of trials and the left–right position in pairs are randomized.
• Random samples. Outputs are drawn with random seeds, not picked by the authors. Cherry-picked samples measure the authors’ taste.
• Checks. Trials with a known answer (an obvious artifact, a real image shown twice) catch inattentive raters.
• Several raters per item and a measured agreement between them, such as Cohen’s κ for two raters (see the LLM evaluation note); low agreement means the question is ambiguous.
• Fixed instructions and exposure. What “realistic” or “matches the prompt” means is defined with examples, and viewing time is controlled when it matters, as in HYPE time.
• Intervals, not only means. Every rate or win probability is reported with its confidence interval and the number of judgements behind it.

What changed#

	Automatic metrics (IS, FID, precision, recall)	Human evaluation
Reference	features of a pretrained network	human perception
Cost per model	minutes of compute	hours of paid work
Reproducible	exactly, given the code and samples	statistically, given the protocol
Sees	what the features encode	what people notice, including artifacts features miss
Noise	sampling of images	sampling of images and of raters, rater disagreement
Typical use	every training run	final comparisons, validating the metrics

Limitations#

• Expensive and slow. It cannot follow every training run; it validates automatic metrics and makes final comparisons.
• Protocol-dependent. Numbers from different instructions, interfaces or rater pools are not comparable.
• Raters differ and drift. Experience, fatigue and the order of trials change answers; agreement has to be measured, not assumed.
• Realism is not everything. An error rate near 50% says nothing about diversity: a model that reproduces training images perfectly cannot be told from the data.
• Pairwise models assume transitivity. Bradley–Terry scores are meaningful only if preferences are roughly consistent across pairs.

Summary#

• Four questions: real or generated, which of two, how good, and does it match the input.
• Real-or-generated tests report a fool rate over generated images, which depends on the rater’s bias toward “real”, or an error rate over a balanced set, where 50% is chance whatever the bias. HYPE fixes the protocol: 50 real and 50 generated images per rater, unlimited time (HYPE∞) or adaptive exposure (HYPE time).
• A rate needs an interval: 200 independent judgements give about ±0.07, and telling 0.45 from 0.5 takes about 783; correlated judgements from the same raters need more.
• Pairwise votes are summarized by the Bradley–Terry model; a 60% win rate is 70.4 Elo points.
• Reliability comes from the design: blind, randomized, random samples, checks, several raters and measured agreement.

Code#

No course notebook covers human evaluation. The two computations of the worked examples:

import math

def wilson(k, n, z=1.96):
    p = k / n
    d = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / d
    half = z / d * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return centre - half, centre + half

print(wilson(76, 200))                   # (0.316, 0.449)
print(400 * math.log10(0.6 / 0.4))       # 70.4 Elo points for a 60% win rate

Next#

The evaluation block ends here. The models so far either bound the likelihood (VAE) or avoid it (GAN). Normalizing flows compute it exactly.

References#

• E. Denton, S. Chintala, A. Szlam, R. Fergus. Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks. NeurIPS, 2015.
• S. Zhou, M. L. Gordon, R. Krishna, A. Narcomey, L. Fei-Fei, M. S. Bernstein. HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models. NeurIPS, 2019.
• E. B. Wilson. Probable Inference, the Law of Succession, and Statistical Inference. Journal of the American Statistical Association, 22(158), 1927.
• R. A. Bradley, M. E. Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/4), 1952.
• W.-L. Chiang et al. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. ICML, 2024.
• D. Jiang, M. Ku, T. Li, Y. Ni, S. Sun, R. Fan, W. Chen. GenAI Arena: An Open Evaluation Platform for Generative Models. NeurIPS Datasets and Benchmarks, 2024.
• ITU-T. Recommendation P.800: Methods for subjective determination of transmission quality. 1996.
• C. Saharia et al. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. NeurIPS, 2022. Imagen and DrawBench.
• J. Yu et al. Scaling Autoregressive Models for Content-Rich Text-to-Image Generation. TMLR, 2022. Parti and PartiPrompts.