Summarization

A summary is a short text that keeps what matters in a long one. Two definitions of “keeps” give two tasks: an extractive summary is made of pieces of the input, and an abstractive summary is new text about it. The first is a labelling problem like those of the previous notes; the second is generation, and it is where the encoder–decoder family of the model-families note has its natural use. The metric is the same for both and favours the first.

Notation. A document $d$, a reference summary $r$, a system summary $s$; $n$-grams of a text are counted with multiplicity.

Task formulation and data#

Input: a document (a news article, a dialogue, a paper). Output: a summary, with the length set by the dataset’s convention. Two datasets appear in this note:

Dataset	Input	Reference	Train size
XSum (Narayan, Cohen and Lapata, 2018)	BBC article	its one-sentence opening summary, written by the journalist	204,045
SAMSum (Gliwa et al., 2019)	a short chat dialogue	a one- or two-sentence third-person summary	14,732

Both references are abstractive: they rephrase, and the XSum sentence is deliberately not in the article. That choice matters for what a metric can measure.

ROUGE#

ROUGE (Lin, 2004) counts $n$-gram overlap between the system summary and the reference. The original is recall-oriented (“how much of the reference is covered”), the usual report is the $F_1$ form:

$$\mathrm{ROUGE\text{-}}n_{\text{recall}} = \frac{|\text{$n$-grams}(s) \cap \text{$n$-grams}(r)|}{|\text{$n$-grams}(r)|}, \qquad \mathrm{ROUGE\text{-}}n_{\text{prec}} = \frac{|\text{$n$-grams}(s) \cap \text{$n$-grams}(r)|}{|\text{$n$-grams}(s)|},$$

and $F_1$ is their harmonic mean. ROUGE-1 and ROUGE-2 use unigrams and bigrams; ROUGE-L uses the longest common subsequence between the two texts, which rewards in-order matches without requiring adjacency, and rougeLsum applies it per sentence. Stemming is usually on. Scores are multiplied by 100.

Toy example: reference the cat sat on the mat, system the cat lay on a mat. Unigram overlap: the, cat, on, mat = 4 of 6 reference unigrams and 4 of 6 system unigrams, so ROUGE-1 $F_1 = 66.7$. Bigram overlap: the cat only (the reference’s the mat became a mat), 1 of 5, ROUGE-2 $F_1 = 20.0$. A summary that copied the sentence would score 100 on both. One that said a feline rested on the rug, correct and in other words, scores 33.3 and 20.0, half the near-copy’s ROUGE-1 and the same ROUGE-2, entirely on the strength of on the:

Figure

What ROUGE counts

This is the property to keep in mind: ROUGE measures lexical overlap with one reference. It is cheap, reproducible and correlates with human judgements across systems on news summarization; it does not detect a fabricated detail that reuses the article’s words, and it penalizes a correct paraphrase.

Extractive summarization#

Baseline. For news, take the first $k$ sentences (Lead-3). On datasets with a lead-biased writing style it is hard to beat; on XSum, whose references are written to not be the lead, it is weak. The classical learned approach scores sentences by position, length, term frequency and similarity to the document centroid and picks the top ones (Nenkova and McKeown, 2011).

Neural extractive summarization is sequence labelling at the sentence level: encode the document, represent each sentence (BERTSum inserts a [CLS] before every sentence and reads its vector, Liu and Lapata, 2019), and classify each as in or out of the summary. Labels come from the reference by an oracle: greedily add the sentences that most increase ROUGE against the reference. The output is a set of whole sentences, grammatical by construction.

A cruder version, tried on SAMSum, shows what goes wrong when the unit is too small: label each token of the dialogue 1 if the substring covered by some reference-summary token occurs in the dialogue, 0 otherwise, then train a linear head on a frozen BERT to predict the labels and concatenate the positive tokens. On a six-line chat whose reference is Laura is going to buy a printer., the output is

Laura new : Laura: thinking about Laura: _ : new ? Jamie buy hand one Laura: be

The procedure fails before the model is trained. Token-level labels built by string matching mark every occurrence of Laura, new and buy wherever they appear, so the positive set is a bag of words, not a span, and the “summary” is the bag read in order. Extraction has to select units that are readable on their own (sentences, or at least clauses), and the oracle has to score candidate units, not match tokens. The classifier itself (1,538 trained parameters on frozen features, evaluation loss 0.12) is doing what it was asked; what it was asked for was wrong.

Abstractive summarization#

The model is the encoder–decoder Transformer of the translation note, pretrained. T5 (Raffel et al., 2020) frames the task as text-to-text with a prefix: the input is summarize: + article, truncated to 1,024 tokens; the target is the reference, truncated to 128. The loss is cross-entropy on the target tokens; padding in the targets is set to $-100$ so it is ignored. Generation uses beam search or greedy decoding, and evaluation decodes the generated ids to text and scores them with ROUGE against the decoded references.

Results on 1,000 XSum validation articles with t5-small (60 M parameters):

Model	ROUGE-1	ROUGE-2	ROUGE-L	Mean length
t5-small, no XSum fine-tuning	16.8	2.1	13.3	19.9 tokens
t5-small after 500 steps of fine-tuning (batch 8, lr $2 \times 10^{-5}$)	not evaluated; mean training loss 3.13
Qwen2.5-0.5B-Instruct, prompted, 10 articles	19.1	3.7	13.2	—

Two cautions on reading the table. The pretrained T5 already produces something summary-like because it was multi-task pretrained with the same prefix; its 16.8 ROUGE-1 is the starting point, at a validation loss of 3.83. The 500-step run (4,000 articles, 2 % of the training set) reports a mean training loss of 3.13 over its steps, which is not directly comparable with the validation loss, and the fine-tuned model was not scored. The prompted 0.5 B model’s numbers come from ten articles and are not a measurement of anything beyond “it produces a summary”. For reference, the XSum paper’s best abstractive model scored 31.9 / 11.5 / 25.8 on the test set, and fine-tuned encoder–decoders of a few hundred million parameters reach the mid-40s on ROUGE-1 (Lewis et al., 2020). XSum ROUGE-2 is low for every system because the references are one abstractive sentence.

Language models as summarizers#

An instruction-tuned decoder summarizes from a prompt, with no fine-tuning, in any style the prompt asks for. Against a fine-tuned encoder–decoder it trades: no training data needed, controllable length and focus, better fluency and paraphrase; worse adherence to a dataset’s conventions (hence lower ROUGE against its references), higher cost per document, and the same faithfulness risk as any generator. Human evaluations have found large instruction-tuned models preferred to fine-tuned systems on news summarization even where ROUGE says otherwise (Goyal, Li and Durrett, 2022), which is a statement about the metric as much as about the models.

What changed#

	Extractive (classical / neural)	Abstractive (encoder–decoder)	Prompted LM
Output	sentences of the input	new text	new text
Grammatical	by construction	learned	learned
Faithful	by construction (but can mislead by omission)	can fabricate	can fabricate
Training	sentence labels from an oracle	reference summaries	none
ROUGE on abstractive references	low ceiling	highest when fine-tuned	lower; style mismatch

Evaluation pitfalls#

• ROUGE rewards copying. Extractive outputs and near-copies score well; paraphrase scores badly. Report it with a faithfulness check.
• Fabrication is invisible to overlap. A summary that adds a wrong number using the article’s words loses nothing on ROUGE. Faithfulness metrics (entailment models, QA-based checks) or a judge model, with the caveats of the LLM evaluation note, are needed.
• One reference. Many good summaries exist; a single reference makes the metric noisy, and implementations differ in tokenization and stemming.
• Length. Longer summaries raise recall-based scores; fix or report length.
• Tiny evaluation sets (10 articles) measure nothing.

Limitations#

• Extractive methods cannot compress across sentences or resolve pronouns; abstractive ones can invent.
• Input length is limited here by the run, not by a fixed table of positions. Inputs were truncated to 1,024 tokens; T5’s relative position buckets accept longer inputs, but attention memory grows with the square of the length and T5 was pretrained on 512-token inputs, so longer inputs need memory and validation, and long documents still need chunking and a second pass.
• Domain and style are learned from the references; a model fine-tuned on one-sentence BBC summaries writes one-sentence BBC summaries.

What you should remember#

• Extractive: select units of the input (sentences, via an oracle); abstractive: generate with an encoder–decoder or a prompted decoder.
• ROUGE-$n$ is $n$-gram overlap $F_1$ with the reference; ROUGE-L uses the longest common subsequence. It rewards copying and cannot see fabrication.
• Token-level extraction with string-matched labels produces word salad; the unit must be readable.
• T5 does summarization as text-to-text with a summarize: prefix, targets truncated to 128 tokens and padding ignored in the loss.
• Prompted language models summarize without training and score lower on ROUGE than fine-tuned models while often being preferred by people.

Notebook#

Two notebooks. Cell numbers count every cell, markdown included, from 1. Quoted numbers come from the run saved with the course; cells changed since then for current library versions have no saved output.

week07_nlp_tasks.ipynb, cells 70–81: the token-level extractive attempt on SAMSum, from the string-matching label construction (cell 72) through the frozen-BERT classifier (cell 74, 1,538 trainable parameters), three epochs of training (cell 79, evaluation loss 0.123) and the demo output quoted above (cell 81).

the summarization notebook (part 2 of the tasks notebook): cells 19–30 load XSum and the ROUGE metric; cells 41–48 preprocess with the summarize: prefix, 1,024-token inputs and 128-token targets; cells 52–62 set up Seq2SeqTrainer with predict_with_generate, a 1,000-example validation subset and a compute_metrics that sentence-splits before scoring; cell 63 evaluates the pretrained checkpoint (the first row of the table); cell 65 trains 500 steps (training loss 3.129), and the saved run does not evaluate the fine-tuned model; cells 70–76 prompt Qwen2.5-0.5B-Instruct on the first ten validation articles with up to 512 new tokens and score them (the last row).

Next#

Classification, tagging, span extraction and summarization each took one model and one head. The tasks that matter most in practice today are built from several: a retriever that finds relevant text, a generator that writes from it, tools the generator can call, and an evaluation of the whole chain. The next section of the course assembles those systems, starting with the embeddings and search note, where the contextual vectors of the encoder become an index.

References#

• C.-Y. Lin. ROUGE: A Package for Automatic Evaluation of Summaries. ACL Workshop on Text Summarization, 2004.
• S. Narayan, S. B. Cohen, M. Lapata. Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. EMNLP, 2018. XSum.
• B. Gliwa, I. Mochol, M. Biesek, A. Wawer. SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization. EMNLP Workshop, 2019.
• A. Nenkova, K. McKeown. Automatic Summarization. Foundations and Trends in Information Retrieval, 5(2–3), 2011.
• Y. Liu, M. Lapata. Text Summarization with Pretrained Encoders. EMNLP, 2019. BERTSum.
• C. Raffel et al. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR, 21(140), 2020. T5.
• M. Lewis et al. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. ACL, 2020. XSum results in Table 3.
• T. Goyal, J. J. Li, G. Durrett. News Summarization and Evaluation in the Era of GPT-3. arXiv, 2022.
• Hugging Face. Summarization. The fine-tuning recipe used in the notebook.