LLM evaluation

The evaluation note at the start of the course assumed a test set with labels and a model that outputs one of them. The systems of this section break both assumptions: the output is text, the “model” is a pipeline of retrieval, generation and tool calls, and the quantity of interest (was the answer helpful, faithful, safe, fast enough, cheap enough) has no label. This note lays out the instruments that exist for each layer and what each can and cannot tell you, with one experiment that calibrates how much a model’s judgement is worth.

Notation. A prompt $x$, a system output $y$, a score $s(x, y)$; human raters $A, B$; a judge model $J$. Pearson correlation $r$ between two sets of scores; Cohen’s $\kappa$ for categorical agreement.

Figure

What measures what in an LLM system

Task-level benchmarks#

The first layer asks what the base model can do: knowledge (MMLU), commonsense completion (HellaSwag), science questions (ARC), resistance to common falsehoods (TruthfulQA), and so on. Each is a dataset with a fixed prompt format and a scoring rule, and the value of the layer comes from running every model through the same format and rule. The lm-evaluation-harness (Gao et al., 2024) standardizes this: a task is a YAML file that names the dataset, says how to build the prompt from a record (doc_to_text), how to read the target (doc_to_target), the choices if any, the few-shot setup and the metrics; every model implements three calls, loglikelihood(context, continuation), loglikelihood_rolling(text) and generate(prompt); and results are aggregated across sub-tasks into one report.

Two scoring modes matter for reading results. Log-likelihood scoring never generates: for a multiple-choice question, each option is appended to the prompt and the option with the highest model probability wins (variants normalize by length or by an unconditional baseline). It is deterministic, cheap and measures something slightly different from “can the model answer”, namely “does the model find the right continuation more probable”. Generative scoring samples an answer and compares it with the target by exact match, F1, BLEU or ROUGE, which depends on decoding settings and on parsing the answer out of the text.

What a benchmark score cannot say: whether the model will do your task (benchmarks are proxies), whether the number is stable under prompt changes (a different template moves MMLU by points), and whether the model has seen the test set (contamination: web-scale pretraining corpora contain the benchmarks, and a memorized answer is not a capability). A score reported without the harness version, the prompt format, the number of shots and the decoding settings is not reproducible.

The scoring rule alone can move a number a long way. With the same prompt (the question, four lettered options and Answer:) on ten MMLU geography questions, SmolLM2-360M scores 0.4 when the harness compares the probabilities of the letters A–D and 0.1 when it compares the probabilities of the full option texts. Ten items give a standard error of about 0.15, so this shows the mechanism, not a measurement.

Retrieval evaluation#

The second layer is the one the semantic search note built: a golden set of (query, relevant passage) and Hit@$k$ with MRR. It is cheap and exact, and it is measured first. When the golden set marks every passage that is sufficient evidence, recall at the context size bounds the share of answers that are correct and grounded in the retrieved text; with one nominated passage per question it is a proxy for that bound, since another passage may hold the same evidence and the model may answer from its parameters.

A model as judge#

The third layer scores free text. For open-ended answers there is no reference and no rule; the alternative to asking people is asking a model, with a rubric (Zheng et al., 2023). It is fast and cheap, and its value depends entirely on how well it agrees with people, which can be measured.

The experiment#

A dataset of question–answer pairs with two independent human ratings each on a four-point scale (bad, could be improved, acceptable, excellent). A 1 B-parameter instruction-tuned model is used as the judge on 28 sampled pairs where the two humans agreed.

Comparison	Items	Pearson $r$
Human rater A vs human rater B	all pairs of the dataset	0.563
Judge, naive prompt (score 0–10 as a float, no rubric) vs humans	28 agreed pairs, 7 per level	−0.169
Judge, improved prompt (1–4 integer scale, rubric text per level, written evaluation before the score) vs humans	the same 28	0.192

Three readings, in order of importance.

The naive judge shows no evidence of signal. On 28 items a correlation of −0.17 cannot be told apart from zero: its 95 % interval runs from about −0.51 to 0.22. The experiment gives no reliable evidence of a positive association, which is not proof that there is none, and the sign is noise. Asking a small model for a float on a 0–10 scale produces numbers that look like scores and are not.

Prompt structure is not enough here. A small integer scale with a described meaning for each level, and a required written evaluation before the number, moves the correlation to 0.19. On 28 items that change is within the noise: the 95 % interval around 0.19 runs from about −0.20 to 0.53, and the one around −0.17 overlaps it. Taken at face value it is also well below the 0.56 between raters, but that number comes from different items, so the gap is not a measured one. A 1 B model is not a usable judge for this task. With a capable judge the same experiment comes out differently: a Mixtral-8x7B judge reached 0.567 with the naive prompt and 0.843 with the improved one (Roucher). The prompt advice holds for a capable judge, and the point stands: a judge’s agreement with people has to be measured, on your task, before its scores are used.

The human–human number is not a ceiling. It is tempting to read 0.56 as the best any judge could reach, and two things block that reading. The samples differ: 0.56 is the agreement of two raters on all pairs, while the judges were scored on 28 pairs chosen because the raters agreed and balanced over the four levels, an easier and less noisy target. And agreement between two noisy raters does not bound agreement with the truth or with their consensus. Under the simplest noise model, each rating the true score plus independent noise of equal variance, a judge that outputs the true score correlates $\sqrt{0.563} \approx 0.75$ with a single rater and about 0.85 with the average of two, both above 0.56. Noisy labels do limit what a comparison can show, but putting a number on the limit needs a noise model and the same items for humans and judges: score the judge and each rater on the same held-out items, report individual and consensus labels, and give intervals.

Practice#

What raises judge quality, from the literature and the experiment: a small integer scale (1–4 or 1–5) rather than a wide or continuous one; a rubric that defines each level; evaluation text before the score; a reference answer when one exists; few-shot examples of scored answers; an additive rubric (one point per criterion met) when the judgement decomposes; structured output so that the score is parsed rather than regexed out of prose; temperature 0; and, for pairwise comparisons, presenting both orders to cancel position bias. Known systematic biases of judge models include a preference for longer answers (verbosity bias) and for their own style (self-preference), both documented in Zheng et al. (2023). Cost control depends on what is judged. Relevance judging of retrieved chunks can be limited to the reranked top candidates, the ones that reach the context; faithfulness and answer quality are judged per generated response, on a sample of responses large enough for the interval needed.

Agreement for categorical labels#

When the judge’s output is a category (relevant / not relevant, pass / fail), Pearson is the wrong statistic. Cohen’s $\kappa$ measures agreement between two labellers beyond what chance would produce:

$$\kappa = \frac{p_o - p_e}{1 - p_e},$$

with $p_o$ the observed proportion of agreements and $p_e$ the agreement expected from each labeller’s marginal frequencies. Toy example: on 100 items, rater A marks 60 % relevant and rater B 70 %, and they agree on 80 items. Then $p_e = 0.6 \times 0.7 + 0.4 \times 0.3 = 0.54$ and $\kappa = (0.80 - 0.54)/0.46 = 0.565$. The 80 % raw agreement looked good; nearly half of it was expected by chance. $\kappa = 1$ is perfect, 0 is chance, below 0 is systematic disagreement; the usual (and arbitrary) reading has 0.4–0.6 as moderate and above 0.8 as near-perfect (Landis and Koch, 1977). $\kappa$ falls when one labeller uses a category much more often than the other, even at high raw agreement, so report both.

Human evaluation#

People are the reference for everything above and expensive, so the design of a human evaluation is about getting reliable labels from few judgements.

• Pairwise preference is more reliable than absolute scores: show two outputs for the same prompt, ask which is better. Chatbot Arena collects such votes from users and fits a Bradley–Terry model (the same one as the reward model of the instruction tuning note) to rank models.
• Rubrics and training for raters, and a measured inter-rater agreement ($\kappa$ or correlation) reported with the result; the 0.56 above is a typical number for a four-point helpfulness scale.
• Sampling. A few hundred prompts, stratified by type, judged by two or more raters each, with disagreements adjudicated. Confidence intervals, since the sample is small.
• Blinding. Raters must not know which system produced which output.

The system, not the model#

The layers above score a model or a stage. What ships is a workflow, and it is evaluated on three more axes:

End-to-end task success. Did the user get what they needed? For a RAG assistant: fraction of golden questions answered correctly and faithfully. For an agent: fraction of tasks completed, at what number of steps. Failures are attributed to the stage that caused them (retrieval miss, bad context, unfaithful generation, wrong tool call), which is why each stage keeps its own metric.

Latency and cost. Time to first token and tokens per second for interactive use; total wall-clock for batch use; tokens in and out per request, which is money and, for agents, grows with every step (the 9,500-token weather lookup of the previous note). A model that scores two points higher and doubles the cost per query has not necessarily improved the product.

Reliability. The rate of unparseable outputs (constrained structured output greatly reduces syntax failures for completions that finish, while truncation, timeouts, refusals, unsupported schema features and well-formed but wrong values remain separate failures to count), of tool errors, of timeouts, of refusals on legitimate requests; variance across repeated runs of the same prompt at the deployed temperature; and behaviour on adversarial and out-of-distribution inputs, including injection attempts through retrieved or tool-returned text.

These are measured on a fixed regression suite that runs on every change to a prompt, a model version, an index or a tool, with the same discipline as a test suite for code, and continued in production by sampling live traffic for human or judge review.

What changed#

	Classifier evaluation	LLM-system evaluation
Ground truth	labels	references (sometimes), rubrics, preferences
Scorer	exact comparison	metrics per stage; model judges validated against people
Unit	one model	a pipeline; failures attributed by stage
Reproducibility	split + metric	+ prompt format, shots, decoding, harness version
Non-quality axes	rarely	latency, cost, reliability, safety

Limitations#

• Benchmarks saturate and leak. A score near the ceiling or on a contaminated set says little.
• Judges inherit the generator’s weaknesses. Verbosity and self-preference biases, and low agreement with people for small judges, as measured above.
• Human labels are noisy (0.56 between raters here), which limits what any comparison with them can show; how much depends on a noise model, and agreement with a consensus can exceed the agreement between two raters.
• Small samples. 28 items give a correlation estimate with a wide interval (about ±0.37 here); neither judge’s correlation is distinguishable from zero, and the gap to the raters’ 0.56 is not a matched comparison either.
• Metrics drift from goals. A system optimized for a judge score optimizes for the judge’s biases.

What you should remember#

• Benchmarks are standardized prompts plus scoring rules; report format, shots, decoding and harness version; watch for contamination.
• Retrieval recall at the context size bounds the correctly grounded answers of a RAG system when all sufficient evidence is annotated (with one golden passage per question it is a proxy); measure it first.
• A model judge is only as good as its measured agreement with humans: with a 1 B judge on 28 agreed items, naive 0–10 float prompt $r = -0.17$ and structured 1–4 rubric $r = 0.19$, neither distinguishable from zero; the raters’ $r = 0.56$ on the full dataset is context, not a ceiling. Use small integer scales, rubrics, reasoning first, references, structured output, temperature 0, both orders for pairs.
• Cohen’s $\kappa$ corrects categorical agreement for chance; report it with raw agreement.
• Evaluate the workflow: end-to-end success attributed by stage, latency, cost, reliability, on a regression suite that runs on every change.

Notebook#

Two notebooks. Cell numbers count every cell, markdown included, from 1. Quoted numbers come from the run saved with the course; cells changed since then for current library versions have no saved output.

week11_pt1_Evaluation_with_Harness.ipynb: cells 1–3 describe the six-stage pipeline (dataset, doc_to_text, doc_to_target, the three-call LM interface, metrics, aggregation). Every run uses SmolLM2-360M with --limit 10 or less, so the scores illustrate the mechanics, not the model. Cells 4–11 define BoolQ and CoLA tasks in YAML, group them, and run them (the BoolQ run in cell 8 failed on a Hub timeout; CoLA scores 0.7). Cells 12–24 cover Jinja prompt templates, log-likelihood versus generative modes, custom !function hooks and dataset preprocessing on MMLU high-school geography: 0.4 scoring the letters (cell 14), 0.1 scoring the option texts (cell 17), and 0.3 with the same prompt built by a function or by preprocessing (cells 22 and 24). Cells 26–37 build a small intent-classification dataset and evaluate it on five items by log-likelihood (accuracy 0.2, cell 35) and generatively (exact match 0, cell 37).

week11_pt2_LLM_as_a_Judge.ipynb: cells 4–5 load Llama-3.2-1B-Instruct as the judge; cell 7 loads the feedback dataset and maps the four ratings to 1–4; cell 9 prints the human–human Pearson correlation 0.563; cell 11 samples 7 agreed-upon items per score level; cells 13–16 run the naive 0–10 prompt, parse the float, rescale to 1–4 and print $r = -0.169$; cells 19–24 run the improved prompt (1–4 scale, rubric, evaluation before score, and a promised reward of GPUs that the authors add as a joke) and print $r = 0.192$; cell 27 lists further improvements (reference answers, few-shot examples, additive scales, structured generation). The commentary in cells 17, 25 and 27 is the tutorial’s (Roucher), written for its Mixtral-8x7B judge (0.567 naive, 0.843 improved), and does not match these numbers: it calls the naive judge “close to human-level agreement”, the change an improvement “by nearly 30%”, and the remaining disagreements “minor”, and with a 1 B judge the printed numbers support none of the three.

The Cohen’s $\kappa$ definition and the judge cost rules are in week09_RAG.ipynb, cells 86–88.

Next#

Every generator in this course so far wrote text left to right, one token per step. The last note of the course, on non-autoregressive generation and text diffusion, gives up that order: start from a fully masked sequence, predict every position at once with a bidirectional model, keep the confident predictions, and repeat, which trades the sequential loop for a fixed number of parallel refinement steps.

References#

• L. Gao et al. The Language Model Evaluation Harness. EleutherAI, v0.4, 2024.
• D. Hendrycks et al. Measuring Massive Multitask Language Understanding. ICLR, 2021. MMLU.
• L. Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS, 2023. Judge agreement with humans, position and verbosity biases.
• Z. Li, A. Mohammad, B. Jin, S. Guo, S. Wang, J. Zhu, H. Chen, M. Zhang. Leveraging Large Language Models for NLG Evaluation: A Survey. arXiv, 2024.
• J. Cohen. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1), 1960.
• J. R. Landis, G. G. Koch. The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1), 1977.
• Z. Li, X. Wang, et al. FeedbackQA (McGill-NLP). The dataset of doubly-rated answers used in the judge experiment.
• O. Sainz et al. NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark. Findings of EMNLP, 2023.
• A. Roucher. Using LLM-as-a-judge for an automated and versatile evaluation. Hugging Face Open-Source AI Cookbook. The tutorial the judge notebook adapts.