Text classification

Classification is the task every representation in this course was tested on, so most of its content is already written. This note is the summary that the course owes it: a clean formulation, the metrics, the full table of results on the two datasets used, the two ways the task is done today, and the pitfalls that are specific to classification rather than to evaluation in general.

Notation. Input text $x$, label $y \in \{1, \dots, K\}$; a model outputs $p(y \mid x)$ and predicts $\hat y = \arg\max_y p(y \mid x)$.

Task formulation#

Map a text to one of $K$ predefined labels. Variants differ in the output structure:

Variant	Output	Loss	Examples
Binary	one of two	binary cross-entropy	sentiment (SST-2), spam
Multi-class	one of $K$	softmax cross-entropy	topic (AG News, $K = 4$), intent
Multi-label	any subset of $K$	$K$ binary cross-entropies	tags, toxicity types
Ordinal	one of $K$ ordered	cross-entropy or regression	star ratings

The input is a document of arbitrary length; every model needs a fixed-size representation of it first (counts, a pooled vector, a special token’s state), and the differences between approaches are mostly differences in how that representation is built.

Metrics#

From the evaluation note: accuracy for balanced classes with equal error costs; per-class precision, recall and $F_1$ otherwise, macro-averaged when every class matters equally. AG News is balanced (30,000 per class) and SST-2 nearly so (44/56), so both datasets are reported with accuracy; a production classifier with a rare class of interest should not be.

Results across the course#

The numbers below come from the notes linked in the first column, one run each. They do not share one protocol: the training-set size, the split they were measured on, and whether anything was chosen by looking at that split differ from row to row, so the table records each. Two numbers can be compared directly only when they share the evaluation set and neither was selected on it.

AG News, four topics, 120,000 training and 7,600 test articles, accuracy:

Model	Trained parameters	Train size	Evaluated on	Selected on the evaluation set	Accuracy
TF-IDF + logistic regression, word counts	$4 \times 29{,}350$	120,000	test	nothing (fixed settings)	0.918
TF-IDF, 1- to 3-gram counts	$4 \times 101{,}284$	120,000	test	nothing (fixed settings)	0.920
Stacked CNN, frozen Word2Vec, $d = 100$	61 K	120,000	test	nothing (last of 5 epochs)	0.899
TextCNN, frozen Word2Vec	122 K	120,000	test	nothing (last of 5 epochs)	0.901
TextCNN, trainable embeddings	2.9 M	120,000	test	nothing (last of 10 epochs)	0.911
4-block Transformer encoder from scratch, WordPiece 4 K, $d = 32$	158 K	120,000	test	best of 20 epochs	0.906
BERT, zero-shot cloze, MLM head	0	none	test	nothing / the label word	0.456 / 0.613
BERT [CLS] + logistic regression, frozen	$4 \times 768$	120,000	test	nothing	0.902
BERT, fine-tuned, all layers	110 M	108,000	12,000 held out from train	nothing (3 epochs)	0.949

SST-2, binary sentiment, GPT-2 small, 50,000 training phrases. Every regime is evaluated on the 872-phrase validation split, which serves as the test set, after a fixed number of steps, one run each. These rows share a protocol and can be compared directly, within the noise of 872 examples: near 0.9 one standard error is about 0.010.

Regime	Trained parameters	Accuracy
Linear probing	1.5 K	0.737
Prompt tuning	10.8 K	0.663
LoRA, $r = 8$	296 K	0.899
Full fine-tuning	124 M	0.890
LP-FT	124 M	0.900

Three readings, each as strong as its protocol. First, a tuned count baseline is a strong classifier for topic: no model trained from scratch on the 120,000 articles beat it on the test set, not even the Transformer whose number is the best of 20 test evaluations and therefore optimistic. Second, the one number above 0.92 is fine-tuned BERT’s 0.949, measured on a held-out tenth of the training set rather than on the test set. It points the same way as the published results on pretraining, but it is not a matched comparison: quoting a three-point gain needs fine-tuned BERT scored on the same 7,600 test articles. Third, on SST-2, where the rows do share a protocol, the training regime of one 124 M-parameter network spans 24 points; the top three regimes differ by about a point, which is within noise.

The Transformer-era approaches#

Encoder with a classification head. A pretrained encoder, a linear layer on the [CLS] vector (or the mean of token vectors), fine-tuned with a small learning rate or with LoRA. This is the default for a task with labelled data: one forward pass per document, no decoding, the highest accuracy per parameter, and a model that is cheap to serve and to evaluate. LoRA often comes close to full fine-tuning. In this course it did so for GPT-2 on SST-2 (0.899 against 0.890), not for an encoder, so the choice is first a cost decision and then something to validate on the task and the model.

Language model with constrained output. Describe the classes in a prompt and ask an instruction-tuned decoder for the label. It works without any labelled data and competes with fine-tuned encoders only for large models; it costs a generation per document; and its raw output is text, which must be parsed. The parsing problem has a clean solution: constrained decoding restricts the sampled tokens at each step to those that can still complete a valid output. With a schema Literal["Positive", "Negative", "Neutral"] the model can produce nothing else, and with a JSON schema it produces a parseable object with typed fields; the RAG note uses the same mechanism for structured extraction. Without constraints, the model “classifies” with explanations, hedges and formatting that break a parser on some fraction of inputs.

Figure

Constrained decoding for classification

	Fine-tuned encoder	Prompted decoder
Labelled data	Required (hundreds to thousands)	None, or a few examples in the prompt
Cost per document	One encoder pass	One generation
Accuracy	Highest with data	Depends on model size and prompt
Output	Logits	Text; constrain it
Changing the label set	Retrain the head	Edit the prompt
Explanations	No	Can be asked for (not necessarily faithful)

What improved, and what did not#

With the same labels, fine-tuned BERT reached 0.95 on a held-out part of the AG News training set, against 0.92 for TF-IDF on the test set; a comparison on the same test articles is still needed to quote the size of the gain. The gain is larger when labels are scarce: with a few hundred examples, count models and from-scratch networks collapse and fine-tuned encoders stay usable (Howard and Ruder, 2018). What did not change: the task still needs a label set that matches the data (the zero-shot experiment lost a class to a label name), the metrics still hide class-level failures unless asked, and a 95 % model still errs on 1 in 20 documents, which for some applications is too many and for others is irrelevant.

Evaluation pitfalls specific to classification#

• Per-class blindness. 0.456 overall hid 0.03 on one class. Always print the confusion matrix.
• Truncation. Encoders truncate at 128–512 tokens; a label that depends on the end of a long document is unreachable. Check where the signal is before choosing the limit.
• Label leakage through artefacts. A source tag, a URL or a date in the text can predict the label without reading it; the leave-one-token-out probe of the CNN note finds such shortcuts.
• Threshold choice. For binary and multi-label tasks, accuracy at the 0.5 threshold is one point on a precision–recall curve; report the curve or the threshold chosen on validation.
• Prompt sensitivity. A prompted classifier’s accuracy changes with wording, label names and example order; evaluate the prompt that will be deployed, with the sampler it will use.
• Test contamination. A pretrained model may have seen the benchmark; a prompted one may have seen it with labels.

Limitations#

A classifier answers one fixed question. Changing the question means new labels and a new head, or a new prompt and a new evaluation. Tasks whose answer is a span, a set of entities or a sentence need the models of the following notes, and tasks whose answer depends on information outside the text need retrieval, which the LLM systems section builds.

What you should remember#

• Classification maps text to one of $K$ labels; binary, multi-class and multi-label variants differ in the output layer and loss.
• AG News: TF-IDF 0.918 on the test set, fine-tuned BERT 0.949 on a held-out tenth of the training set, not a matched comparison; nothing from scratch beat the count baseline on the test set. Record the evaluation split and what was selected on it next to every number. SST-2: the training regime of one model spans 0.663 to 0.900.
• With labels, fine-tune an encoder (full or LoRA). Without labels, prompt a decoder with constrained output.
• Report per-class metrics, check truncation and artefacts, and evaluate the exact prompt and sampler that will be used.

Notebook#

The results in the tables come from the notebooks of the notes linked there, each number from the run saved with the course; the fine-tuned BERT number was saved by an earlier version of its cell, as the transfer learning note explains. For the constrained-output example: week09_RAG.ipynb, cells 94–99 (cell numbers count every cell, markdown included, from 1), where an instruction-tuned 3 B model is first asked for a 1–5 rating in free text and then constrained with outlines to a Literal of three sentiments and to a pydantic Rating schema with an integer field bounded to $[1, 5]$. The saved outputs of those cells are empty; the mechanism is documented in the library’s own examples.

Next#

A label per document is the simplest output. Many tasks compare two texts instead: duplicate questions, entailment, how close two sentences are in meaning. The next note covers these pair tasks and the two ways a Transformer can read a pair, together or one text at a time.

References#

• J. Howard, S. Ruder. Universal Language Model Fine-tuning for Text Classification. ACL, 2018. Pretraining gains as a function of labelled-set size.
• X. Zhang, J. Zhao, Y. LeCun. Character-level Convolutional Networks for Text Classification. NeurIPS, 2015. AG News and classical baselines.
• R. Socher et al. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. EMNLP, 2013. SST.
• B. T. Willard, R. Louf. Efficient Guided Generation for Large Language Models. arXiv, 2023. Constrained decoding as used by outlines.
• S. Gururangan et al. Annotation Artifacts in Natural Language Inference Data. NAACL, 2018. Label leakage through dataset artefacts.
• D. Jurafsky, J. H. Martin. Speech and Language Processing, 3rd ed. draft. Chapters 4–5 (classification) and 11 (fine-tuning for classification).