Transfer learning in NLP

The course has one running benchmark: four-way topic classification of AG News, where TF-IDF scored 0.918 and every model trained from scratch scored lower. A pretrained encoder changes the question from “which architecture” to “how to use what was learned on 3 billion words”. This note tries the three standard answers on the same model and data, in increasing order of cost, and gets the first result above the baseline, with a caveat about splits that the comparison below spells out.

Notation. A pretrained encoder $f_\theta$ maps a token sequence to vectors $h_1, \dots, h_n$; $h_{\text{CLS}}$ is the vector at the first position. A task head $g_\phi$ maps a vector to logits. “Frozen” means $\theta$ is not updated.

Baseline: the pretrained model as a black box#

The checkpoint is a set of weights and a tokenizer. Three things can be done with it without training it from scratch: ask it questions in its own language (prompting), read its outputs as features for another model, or continue training it on the task. Each transfers a different part of what pretraining learned.

Figure

Three ways to use one pretrained encoder

Zero-shot classification with the MLM head#

What transfers: the model’s ability to fill a blank with a plausible word, shown in the BERT note.

Append a template to each article, This news is about [MASK]., and read the MLM distribution at the mask restricted to four candidate words, one per class: world, sports, business, technology. The prediction is the candidate with the highest probability. No parameter is trained; the “training” is choosing the template and the words.

Setting	Accuracy
Labels world / sports / business / technology	0.456
per class: World 0.03, Sports 0.51, Business 0.96, Sci/Tech 0.32
Labels politics / sports / business / technology, chosen on the test articles	0.613

The first result looks like a weak classifier; the per-class numbers say it is a broken one. Nearly every World article is labelled something else, because world is not a word that completes “This news is about ___” for an article about an election. Counting the model’s top-100 fillers over the 1,900 World test articles, the most frequent are us, it, america, me, you, politics and news; set aside the pronouns, and the natural label is politics or a country. Replacing world by politics raises accuracy from 0.456 to 0.613 with no other change. The method works to the extent that the class name is the word the model would naturally use, and that is a property of the template and the vocabulary, not of the articles.

The 0.613 is test-informed. The new word was found by reading the fillers of the World test articles and the per-class test failures, and it was then scored on the same test set. A choice made by looking at the evaluation data is optimistic, by an amount this run cannot measure. The clean protocol chooses the template and the label words on a development sample (a few hundred labelled training articles are enough to count fillers) and scores the test set once. Here 0.456 is the only zero-shot number made without looking at test labels; 0.613 shows how much the label word matters, not what a zero-shot classifier can be expected to reach.

Zero-shot is the cheapest route and the weakest: 0.613 is far below TF-IDF. Its value is in needing no labelled training data (a fair choice of label words still needs a small labelled development sample), and in tasks whose labels are ordinary words. For a decoder-only model the same idea becomes prompting with a few examples (the GPT note), which is stronger but still a prediction from the pretrained distribution rather than from the task’s data.

Frozen features and a linear model#

What transfers: the contextual vectors.

Run every article through the frozen encoder, take $h_{\text{CLS}}$ (768 numbers), and train multinomial logistic regression on the 120,000 vectors. This is the TF-IDF pipeline with the 29,350-dimensional sparse count vector replaced by a 768-dimensional dense one.

Features	Linear model test accuracy
TF-IDF, 29,350-dim	0.918
Frozen BERT [CLS], 768-dim	0.902

The encoder’s vector carries the topic without ever having been trained to: a linear read-out recovers 90 % of the labels. It does not beat the count baseline, for a reason the CNN note already met: a feature set tuned to the task (every word, with a weight) is hard to beat with a general-purpose feature set, and [CLS] was pretrained to predict sentence adjacency, not topic. Mean-pooling the token vectors usually does better than [CLS] for a frozen encoder, but it is not part of this comparison.

Cost: one forward pass per document, once, then a model that trains in seconds and runs without the encoder at inference if the vectors are cached. This is the route when the encoder must stay fixed (shared between tasks, or served once for many consumers), and it is the route that sentence-embedding retrieval in the semantic search note takes.

Fine-tuning the whole encoder#

What transfers: the weights themselves, as an initialization.

Put a new linear head on [CLS], and train all parameters, encoder and head, on the labelled data with a small learning rate. The pretrained weights are the starting point and the task loss moves them.

Model	Trainable parameters	Validation accuracy
BERT-base + linear head, 3 epochs, lr $2 \times 10^{-5}$, batch 16	110 M	0.949

Held out: 10 % of the training set (12,000 articles), since AG News has no validation split. Three passes over 108,000 articles at a learning rate 50× smaller than from-scratch training give 0.949 on those 12,000 held-out articles. That is three points above TF-IDF’s 0.918 and about four above the best from-scratch model, but those numbers are test accuracy on the 7,600 test articles. Different examples, so the margin is indicative, not a paired comparison: quoting a gain needs both models scored on the same untouched test articles. The small learning rate is the point: the goal is to adjust the pretrained representation, not to overwrite it, and larger rates erase it (catastrophic forgetting) before the head has learned anything.

Route	Trains	Data needed	Evaluated on	AG News accuracy
Zero-shot cloze	nothing	a template	test (label word chosen on test)	0.456 / 0.613
Frozen features + linear	768 × 4 weights	labels	test	0.902
Full fine-tuning	110 M	labels	10 % of train, held out	0.949
TF-IDF + linear (no pretraining)	29,350 × 4	labels	test	0.918

What is being transferred#

The three routes transfer, respectively, the pretrained output distribution, the pretrained representation, and the pretrained weights as initialization. Fine-tuning wins because it uses the most: every layer can adapt, and the gradient of the task loss flows into the attention patterns and feed-forward features that pretraining built. The frozen route cannot change what [CLS] encodes; the zero-shot route cannot change anything.

The gain is empirical, and it has been measured many times since Howard and Ruder (2018) and Devlin et al. (2019): fine-tuned encoders beat from-scratch models by a margin that grows as the labelled set shrinks. With 120,000 labels the margin on AG News is about three points here, between a validation and a test score; with 1,000 labels, from-scratch models collapse and fine-tuned ones hold up.

Limitations#

• Full fine-tuning produces a full copy. Every task gets its own 110 M (or 7 B) parameters. The PEFT note addresses the cost.
• Instability. Fine-tuning runs vary with the seed, especially on small datasets; the practice is several runs and a validation-based choice (Dodge et al., 2020).
• Forgetting. A fine-tuned model is worse at everything except its task; the pretrained generality is spent.
• Zero-shot depends on wording. Template and label-word choice changed test accuracy by 16 points here, with the word chosen on the test set; that sensitivity does not go away with larger models, it moves to the prompt.
• Frozen [CLS] is a weak sentence vector. Models meant to produce embeddings are trained for it, with a contrastive objective, not read off a masked-language model.

What you should remember#

• Zero-shot via the MLM head: rephrase the task as a cloze, restrict the vocabulary to label words; accuracy depends on the label words (0.456 → 0.613 by renaming one class; the new name was chosen on the test articles, so 0.613 is optimistic).
• Frozen features: [CLS] + logistic regression gives 0.902, below TF-IDF’s 0.918; the pretrained vector carries the topic but was not trained for it.
• Full fine-tuning at a small learning rate gives 0.949 on a held-out tenth of the training set, the first result above the baseline’s 0.918 on the test set; the splits differ, so the three points are indicative. The gain comes from the pretrained initialization.
• Report per-class metrics: the 0.456 hid a class at 3 %.
• Each fine-tuned task costs a full copy of the model, which the next section reduces.

Notebook#

week06_BERT.ipynb. Cell numbers count every cell, markdown included, from 1. Quoted numbers come from the run saved with the course; cells changed since then for current library versions have no saved output.

• Cells 31–36 build the prompted test set (text + " This news is about [MASK]."), run the fill-mask pipeline with targets set to the four label words, and report 0.456.
• Cells 38–39 compute per-class accuracy; cells 41–44 filter the World articles and count the pipeline’s top-100 fillers; cells 46–49 rename the class to politics and report 0.613.
• Cells 52–57 switch to the feature-extraction pipeline, take the first output vector (out[0, 0]) of each article as $h_{\text{CLS}}$, fit LogisticRegression on the training vectors (the solver stops at its iteration limit, which the warning records) and report 0.902 on the test set. Mean-pooling of the token vectors is not run.
• Cell 60 fine-tunes AutoModelForSequenceClassification with 4 labels through Trainer: 90/10 split of the training set with seed 42, truncation at 128 tokens, 3 epochs, learning rate $2\times10^{-5}$, weight decay 0.01, batch 16; in the notebook as first taught this cell (then cell 61) saved an evaluation accuracy of 0.9486 on the held-out 10 %. That version’s code passes accuracy.computes, a method that does not exist, as compute_metrics, so the saved output must come from an earlier, working version of the line; the current notebook defines the metric function, and this cell has no saved output.

Next#

Fine-tuning is the best of the three routes and the most expensive: a full set of weights per task, trained with care. The next section of the course is about doing it cheaper and doing it better. The fine-tuning note takes the question to a decoder-only model and compares training everything, training only the head, and training the head first and everything second.

References#

• J. Howard, S. Ruder. Universal Language Model Fine-tuning for Text Classification. ACL, 2018. ULMFiT: fine-tuning a pretrained language model for classification.
• J. Devlin, M.-W. Chang, K. Lee, K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL, 2019. §4 (fine-tuning procedure and results).
• T. Schick, H. Schütze. Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference. EACL, 2021. Pattern-exploiting training: the cloze formulation of classification used above.
• J. Dodge, G. Ilharco, R. Schwartz, A. Farhadi, H. Hajishirzi, N. Smith. Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping. arXiv, 2020.
• N. Reimers, I. Gurevych. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP, 2019. Why frozen [CLS] is a poor sentence vector and what to train instead.
• M. E. Peters, S. Ruder, N. A. Smith. To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks. RepL4NLP, 2019. Feature extraction versus fine-tuning.