Encoder, decoder and encoder–decoder models

Two notes have now trained the same block in two arrangements with two objectives: the bidirectional encoder of the BERT note with masked tokens, and the causal decoder of the GPT note with next tokens. The encoder–decoder of the translation note was trained on a task, not pretrained, but it has its own pretraining objective too. This note lines the three up and makes the comparison that the architecture names alone do not: objective, what the model computes, what it is good at, and what it costs.

Notation. Input tokens $x$; for encoder–decoders, the target $y$; a contextual vector $h_t$ per position; the mask pattern of self-attention is full (every position sees every position) or causal (position $t$ sees $\le t$).

Baseline: the architecture name as the only description#

“Encoder-only”, “decoder-only” and “encoder–decoder” describe which positions can attend to which, and whether there is a second sequence. That determines what the model can compute. What it does compute is set by the pretraining objective, and the two are coupled: a bidirectional stack cannot use the next-token objective (the answer leaks), and a causal stack gets no benefit from masking.

The three families#

Figure

Three families as pairs of mask and objective

Encoder-only: full mask, masked-token objective#

Every position sees the whole input. Pretraining hides 15 % of the tokens and predicts them (MLM). The output is one contextual vector per input token; there is no mechanism to produce a sequence of new tokens. Representatives: BERT, RoBERTa, DeBERTa, and the sentence-embedding models of the semantic search note.

What it is good for: anything that maps an input to a label, a label per token, or a span of the input. Classification, named entity recognition, extractive question answering, retrieval embeddings. The NLP tasks section uses an encoder for all of these, with a different small head each time.

Decoder-only: causal mask, next-token objective#

Position $t$ sees $\le t$. Pretraining predicts every token from its prefix. The output at $t$ is a distribution over the next token, so the model generates by sampling and feeding back. Representatives: the GPT series, Llama, Mistral, Qwen, Gemma.

What it is good for: generation, and, through prompting, any task that can be written as text in and text out. The representation at position $t$ is one-directional, so for a pure understanding task the model has to see the whole input before any output position can summarize it; the summary lives at the last position.

Encoder–decoder: full mask on the source, causal mask on the target, cross-attention between them#

The encoder reads $x$ bidirectionally; the decoder generates $y$ causally while attending to the encoder output. Pretraining objectives are span corruption (T5, Raffel et al., 2020): drop random spans from the input, replace each by a sentinel token, and train the decoder to emit the dropped spans in order; or denoising (BART, Lewis et al., 2020): corrupt the input by masking, deletion, permutation and infilling, and reconstruct the original. Both are “reconstruct what was damaged”, using the encoder for the damaged text and the decoder for the repair.

What it is good for: tasks with a distinct input and a generated output. Translation, summarization, question generation. The summarization note fine-tunes a T5 model; T5 frames every task, including classification, as text-to-text (summarize: …, translate English to German: …).

	Encoder-only	Decoder-only	Encoder–decoder
Self-attention mask	full	causal	full (source) + causal (target)
Second sequence	no	no (prompt is part of the one sequence)	yes, via cross-attention
Pretraining objective	masked tokens	next token	corrupted spans / denoising
Trains on	15 % of positions	all positions	the dropped spans
Output	vector per input token	next-token distribution	next-token distribution for $y$
Representation of $x$	bidirectional	left-to-right	bidirectional
Can generate	no	yes	yes
Examples	BERT, RoBERTa, DeBERTa	GPT, Llama, Mistral	T5, BART, mT5

How the objectives differ in what they teach#

MLM gives the richest per-token representation, because each prediction uses both sides, and the poorest generation, because it never produces a sequence. It trains on a minority of positions per pass.

Next-token prediction trains on every position and gives a model that generates. The representation of early tokens is impoverished by never seeing later ones, but the model compensates when it is allowed to read the whole input first and answer afterwards, which is the normal prompting pattern.

Span corruption sits between them: the encoder learns bidirectional representations, the decoder learns to generate, and the sentinel mechanism trains the decoder to produce short, targeted outputs rather than whole documents. It costs two stacks and the cross-attention.

An encoder’s output is a representation of the input; a decoder’s output is a representation of what comes next. This is the practical rule for choosing: when the quantity of interest is a property of the input (its class, its entities, its similarity to another text), an encoder’s vectors are the right feature, and they are cheaper, because one forward pass produces all of them with no decoding loop. When the quantity of interest is new text, a decoder is required.

Why decoder-only models came to dominate#

Three reasons, all empirical.

Scaling. Next-token prediction uses every token of every document as a training signal and needs no masking or span machinery; it scaled to trillions of tokens with the least engineering, and the scaling laws were measured on it (Kaplan et al., 2020).

Unification. A sufficiently large decoder does classification, extraction and translation by prompting, with no new head. One model replaces a zoo of fine-tuned encoders, at a cost in inference compute, because a decoder has to generate the label token by token where an encoder read it off a linear layer.

In-context learning. Given a few examples in the prompt, a large decoder performs the task without parameter updates (Brown et al., 2020); encoders have no comparable mechanism because they have no output sequence to put the answer in.

What the dominance does not mean: encoders are not obsolete. For a fixed task with labels and a latency budget, a common rule of thumb is that a fine-tuned 110 M-parameter encoder is often as accurate as a prompted decoder, much cheaper to run (one forward pass, no generated tokens) and easier to evaluate. That is experience, not a measurement made in this course: the transfer learning note compares fine-tuned BERT with BERT prompted through its own masked-language-model head on AG News, not with a prompted decoder, and no latency or cost ratio was measured. A number for the cost ratio needs the same task, hardware and quality target for both. Every retrieval system in the LLM systems section runs on encoders.

Choosing, in practice#

Need	Pick	Why
Label an input, or each token of it, with labelled data available	Encoder, fine-tuned	Best accuracy per parameter; one pass; no decoding
Embed texts for search or similarity	Encoder (sentence-embedding variant)	Bidirectional vectors, trained for cosine comparison
Generate free text, chat, follow instructions	Decoder (usually)	The common choice for open-ended completion and chat; instruction-tuned encoder–decoders also take a prompt and generate
Transform an input into an output (translate, summarize) with training pairs	Encoder–decoder or decoder	Encoder–decoder is the classical fit; large decoders match it by prompting
Do many tasks with no labels	Decoder, instruction-tuned	Prompting and in-context learning

The decision interacts with the training regime of the next section: fine-tuning favours small encoders, parameter-efficient tuning and prompting favour large decoders.

Limitations#

• The table is about pretrained checkpoints, not possibilities. Any arrangement can be trained with any objective that does not leak; a decoder can be fine-tuned to produce embeddings, and encoders can be stacked into generators.
• “Bidirectional” is not free. An encoder must re-encode when the input changes; a decoder with a KV cache appends.
• The comparison is empirical and moving. Which family wins on a task depends on model size, data and the year; the structural facts (what each mask allows, what each objective trains) are the part that stays.

What you should remember#

• Encoder: full mask, MLM, vector per token, no generation. Decoder: causal mask, next token, generates. Encoder–decoder: both, joined by cross-attention, trained by span corruption or denoising.
• An encoder represents the input; a decoder represents what comes next. Pick by whether the answer is a property of the input or new text.
• Decoders dominate because next-token prediction scales most simply, one model serves many tasks by prompting, and in-context learning needs an output sequence.
• Encoders remain the efficient choice for labelled classification, tagging, extraction and retrieval embeddings.

Notebook#

No single notebook covers this comparison; the three families appear in the notebooks of their own notes. For the architectures side by side: week06_BERT.ipynb cell 9 (encoder), week08_modern_LLM.ipynb cell 18 (decoder), and the summarization notebook (part 2 of the tasks notebook) cells 41–52 (T5, with the summarize: task prefix and AutoModelForSeq2SeqLM). Cell numbers count every cell, markdown included, from 1. Quoted numbers come from the run saved with the course; cells changed since then for current library versions have no saved output.

Next#

A pretrained model of any family is a starting point, and the question is how to use it for a task: read its vectors as features, prompt it without training, or train it further. The next note measures all three on the same AG News problem with the same encoder, from 61 % zero-shot to 95 % fine-tuned, and sets up the adaptation methods of the following section.

References#

• C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR, 21(140), 2020. T5, span corruption, and the architecture comparison in §3.2.
• M. Lewis et al. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. ACL, 2020.
• J. Devlin, M.-W. Chang, K. Lee, K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL, 2019.
• T. B. Brown et al. Language Models are Few-Shot Learners. NeurIPS, 2020.
• J. Kaplan et al. Scaling Laws for Neural Language Models. arXiv, 2020.
• T. Wang et al. What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?. ICML, 2022. A controlled comparison of the three families and their objectives.