Corpora and evaluation

The two previous notes turned text into token ids. Before training anything on them we need a protocol for saying whether a model works, because every later note compares methods by a number, and a number is only meaningful if the data it was computed on and the way it was computed are right. This note fixes the protocol: how the corpus is split, which failures of the split silently inflate results, which metric fits which task, and what an evaluation can and cannot conclude.

Notation. A labelled corpus is a set of pairs $(x_i, y_i)$. A classifier outputs $\hat y_i$. For a class $k$, $\mathrm{TP}_k$ counts examples with $y = k$ and $\hat y = k$, $\mathrm{FP}_k$ those with $y \ne k$ and $\hat y = k$, and $\mathrm{FN}_k$ those with $y = k$ and $\hat y \ne k$.

Three splits and what each is for#

A model is fitted on the training set. Every choice that is not fitted by gradient descent (learning rate, number of epochs, vocabulary size, prompt wording, the decision to stop) is made by looking at the validation set. The test set is touched once, at the end, to report the number.

The reason for three rather than two is that selection on a set biases the estimate on that set. If we try twenty learning rates and keep the one with the best validation accuracy, the validation accuracy of the winner is an optimistic estimate of its accuracy on new data, because part of its advantage is luck on those specific examples. The test set, never used for a decision, is the only unbiased estimate left.

Figure

The evaluation protocol and the ways it leaks

The datasets used in this course follow the convention as far as their publishers allow:

Corpus	Train	Validation	Test	Used in
AG News (4 topics)	120,000	—	7,600	classification
Multi30k (De→En)	29,000	1,014	1,000	translation
SST-2 (sentiment)	67,349	872	1,821 (labels hidden)	fine-tuning
CoNLL-2003 (NER)	14,041 sentences	3,250	3,453	NER
SQuAD 1.1 (QA)	87,599	10,570	—	question answering

Two of them hide or lack the test labels, so their results are reported on the validation set, which is honest as long as nothing is tuned on it. AG News ships with no validation split. Most of the classification experiments in this course report test accuracy after a fixed number of epochs with no selection, which is honest only because nothing was tuned on the test set; the BERT fine-tuning run in the transfer learning note instead carves 10 % of the training set out as a validation set and reports on that.

Leakage: when the split is not a split#

Leakage is any path by which information about the evaluation examples reaches the model or the person tuning it. Four forms are common in text:

• Duplicates across splits. Web corpora and question collections contain near-duplicates. If the same question appears in training and test, the model is rewarded for memorization. Deduplicate before splitting, with a fuzzy criterion rather than exact string equality.
• Preprocessing fitted on everything. A tokenizer, a TF-IDF vocabulary or a word-embedding model trained on train + test has seen the test words. The TF-IDF note fits the vectorizer on the training texts only, then transforms both. The same rule applies to the Word2Vec model in the CNN note: it is trained on the training texts.
• Temporal leakage. If the task is to predict something about future text, a random split lets the model train on later documents than it is tested on. Split by time.
• Test-set tuning. Running the test set after every change and keeping the best result is leakage through the researcher. It is the reason the validation set exists.

A fifth form is specific to pretrained models and harder to detect: the pretraining corpus may contain the benchmark. A model that has read the test set of a benchmark during pretraining reports a score that measures recall, not ability. The LLM evaluation note returns to this.

Accuracy, and when it lies#

For a classifier the obvious number is accuracy, the fraction of correct predictions:

$$\mathrm{Acc} = \frac{1}{N}\sum_{i=1}^N \mathbb 1[\hat y_i = y_i].$$

It is the right number when the classes are balanced and every error costs the same. AG News has 30,000 training examples per class, so accuracy is used throughout its experiments. SST-2 is 44 % negative and 56 % positive, close enough.

Accuracy fails when one class dominates. In the NER corpus most tokens are labelled O (not an entity), so a model that predicts O everywhere is right most of the time and useless. It also hides which class fails. The zero-shot classifier in the transfer learning note reports 45.6 % accuracy on the four AG News topics, which looks like a weak but working model, until the accuracy is broken down by class: 96 % on Business, 51 % on Sports, 32 % on Sci/Tech and 3 % on World. The model has not learned the World class at all; it maps nearly all of it to other labels. One number hid a complete failure on a quarter of the data.

Precision, recall and F1#

The repair is to count errors per class and in two directions. Of the examples the model labelled $k$, how many were really $k$ (precision), and of the examples that were really $k$, how many did the model find (recall)?

$$P_k = \frac{\mathrm{TP}_k}{\mathrm{TP}_k + \mathrm{FP}_k}, \qquad R_k = \frac{\mathrm{TP}_k}{\mathrm{TP}_k + \mathrm{FN}_k}.$$

Each can be made perfect trivially at the expense of the other: predict $k$ for everything and recall is 1, predict $k$ only when certain and precision is 1. The F1 score is their harmonic mean,

$$F_1 = \frac{2 P R}{P + R} = \frac{2\,\mathrm{TP}}{2\,\mathrm{TP} + \mathrm{FP} + \mathrm{FN}},$$

which is close to the smaller of the two: a model with $P = 1$ and $R = 0.1$ has $F_1 \approx 0.18$, not $0.55$. That is the property we want from a single number; the arithmetic mean would reward the trivial strategies.

With several classes there are two ways to combine. Macro averaging computes $F_1$ per class and averages, so every class counts equally regardless of size. Micro averaging sums the TP, FP and FN counts over classes first, which weights every example equally and, for single-label classification, equals accuracy.

Worked example#

A toy classifier on 100 documents with three topics; rows are true classes, columns predictions:

true \ predicted	A	B	C	total
A	50	5	5	60
B	5	20	5	30
C	5	3	2	10
total	60	28	12	100

Accuracy is $(50 + 20 + 2)/100 = 0.72$. Per class:

Class	TP	FP	FN	$P$	$R$	$F_1$
A	50	10	10	0.833	0.833	0.833
B	20	8	10	0.714	0.667	0.690
C	2	10	8	0.167	0.200	0.182

Macro $F_1 = (0.833 + 0.690 + 0.182)/3 = 0.568$; micro $F_1 = 0.72 =$ accuracy. The small class C is nearly missed, accuracy barely notices, and macro $F_1$ does. Which number to report depends on whether class C matters as much as class A; that is a decision about the task, not about statistics.

Spans rather than tokens#

For sequence labelling the unit of evaluation should be the thing we want extracted. NER systems predict a tag per token, but a half-recognized entity (New tagged, York missed) is not half right. The seqeval convention used in the NER note therefore computes precision, recall and $F_1$ over whole entity spans: a span counts as a true positive only if its boundaries and type both match.

Intrinsic and downstream evaluation#

Some models are not classifiers and have no accuracy. A language model assigns probabilities to text, an embedding model places words in a vector space, a retriever returns a ranked list. Each has an intrinsic metric that measures the model’s own objective:

Model	Intrinsic metric	Note
Language model	Perplexity	n-grams
Word embeddings	Analogy accuracy, similarity correlation	word embeddings
Retriever	Precision@k, mean reciprocal rank	semantic search
Translation / summarization	BLEU / ROUGE against references	translation, summarization

An intrinsic metric is cheap and comparable across models, and it measures the wrong thing. Lower perplexity does not guarantee better generated text; a better analogy score does not guarantee a better classifier built on the embeddings. Downstream evaluation trains or runs the model inside the task we actually care about and measures that task’s metric. It is the number that matters, and it is expensive, task-specific and noisy. The usual practice is to use intrinsic metrics to iterate and downstream metrics to decide.

Pitfalls that survive a correct split#

• Implementation differences. BLEU has several implementations that tokenize and smooth differently and give different numbers on the same outputs. Comparisons are valid only within one implementation; the translation note fixes one and uses it throughout.
• Averaging over batches. Averaging a per-batch accuracy over batches of unequal size (a last partial batch, or variable numbers of non-padding tokens) weights examples unequally. Sum the counts, then divide once.
• Reporting the best epoch on the test set. That is validation, not test.
• Confidence without a confidence interval. On the 872-example SST-2 validation set one example is 0.11 %; a difference of 0.5 % between two runs is within the noise of initialization and data order. Repeat runs with different seeds before calling a difference real.
• Metric saturation. When every model scores near 100 %, the benchmark no longer ranks them.

Limitations#

A correct evaluation tells you how a model behaves on data drawn like the test set. It does not tell you how it behaves on different data, on inputs chosen by an adversary, or on the long tail that a 7,600-example test set does not contain. A benchmark result is a necessary condition, not a sufficient one; the LLM evaluation note looks at what remains when the model is a system rather than a classifier.

What you should remember#

• Train to fit, validate to choose, test once to report. Selection on a set biases the estimate on that set.
• Leakage comes through duplicates, preprocessing fitted on test data, time, and the person tuning the model.
• Accuracy is fine for balanced classes with equal error costs. Otherwise report per-class precision, recall and $F_1$, and choose macro or micro deliberately.
• $F_1$ is the harmonic mean, so it tracks the worse of precision and recall.
• Intrinsic metrics are for iterating, downstream metrics for deciding. Use one implementation of a metric, sum counts before dividing, and repeat runs before trusting small differences.

Notebook#

The relevant cells are spread over the notebooks that introduce each dataset. Cell numbers count every cell, markdown included, from 1. Quoted numbers come from the run saved with the course; cells changed since then for current library versions have no saved output.

• week01_word_embeddings.ipynb, cells 33–38: AG News train and test sizes and the class histogram; cells 44–48: the TF-IDF vectorizer is fitted on training texts only and accuracy is reported on both splits (0.94 train, 0.918 test).
• week06_BERT.ipynb, cells 36–39: overall zero-shot accuracy 0.456 and the per-class breakdown quoted above; cell 60 splits 10 % of the training set off as a validation set before fine-tuning.
• week07_nlp_tasks.ipynb, cells 28–30: entity-level precision, recall and $F_1$ with seqeval, after dropping the -100 labels that mark special and continuation tokens.
• week04_lstm_attention_transformer.ipynb, cell 36: the choice of one BLEU implementation; cell 42 contains the per-batch accuracy averaging that the pitfall above refers to, with a comment acknowledging it.

Next#

With units and a measurement protocol in place, the course starts modelling. The first model deliberately ignores word order: it counts which tokens a document contains and feeds the counts to a linear classifier. The next note builds it, measures it on AG News, and establishes the baseline that neural models in later notes have to beat.

References#

• D. Jurafsky, J. H. Martin. Speech and Language Processing, 3rd ed. draft. Chapter 4 (§4.7, evaluation: precision, recall, F-measure; §4.8, test sets and statistical significance).
• C. D. Manning, P. Raghavan, H. Schütze. Introduction to Information Retrieval. Cambridge University Press, 2008. §8.3 (precision, recall, F) and §13.6 (evaluation of text classification, micro and macro averaging).
• N. Nakayama. seqeval: A Python framework for sequence labeling evaluation. Entity-level metrics for IOB-tagged sequences.
• M. Post. A Call for Clarity in Reporting BLEU Scores. WMT, 2018. Why BLEU implementations disagree.
• X. Zhang, J. Zhao, Y. LeCun. Character-level Convolutional Networks for Text Classification. NeurIPS, 2015. The AG News split used throughout the course.