GPT and autoregressive language modeling

The encoder of the previous note fills in blanks and cannot continue a text. Generation needs the factorization of the n-gram note, $P(w_{1:T}) = \prod_t P(w_t \mid w_{<t})$, with a model that conditions on the whole prefix, or as much of it as its context window holds. A Transformer decoder with a causal mask is exactly that model, and training it on a large corpus is what the GPT series did (Radford et al., 2018, 2019; Brown et al., 2020). This note covers the objective, the architecture in two concrete sizes, the behaviour of the resulting base model, and the decoding step that every use of it goes through.

Notation. Tokens $w_{1:T}$; logits $z_t \in \mathbb R^{|V|}$ at position $t$; $P(w \mid w_{<t}) = \operatorname{softmax}(z_t)_w$. Temperature $\tau$.

Baseline: the recurrent language model#

The RNN note already trains a next-token predictor by teacher forcing. Its context is the whole prefix, through a state that forgets, and it processes positions one after another. Replacing the recurrence with causal self-attention keeps the objective and removes both constraints, at the price of a fixed context window (1,024 positions for GPT-2 small): attention reads every earlier position inside the window directly, but the history is not unbounded.

The objective#

$$\mathcal L = -\frac1T\sum_{t=1}^{T} \log P(w_t \mid w_{<t}),$$

the per-token cross-entropy, whose exponential is the perplexity. The input is the sequence, the target is the sequence shifted left by one, and two mechanisms work together. Teacher forcing supplies the gold prefix: the prediction at position $t$ is conditioned on the true $w_{<t}$, not on the model’s own earlier outputs, as in the recurrent model. The causal mask keeps the later gold tokens out of reach: position $t$ cannot attend to $w_{t+1}$ or anything after it, so one forward pass over the whole gold sequence computes all the predictions at once without leakage.

Worked example (three positions). For the sequence the cat sat down, the input is the cat sat and the target cat sat down:

Position	Input token	May attend to	Predicts
1	the	the	cat
2	cat	the cat	sat
3	sat	the cat sat	down

Every row reads gold tokens only (teacher forcing), and no row reads its own target or anything after it (the mask).

All positions are therefore trained in parallel on the same forward pass. This is the practical advantage over the recurrent model that the Transformer note promised: training cost per sequence is one pass of $T$ positions in parallel, not $T$ sequential steps.

Nothing else is needed. No masking rule, no segment pairs, no auxiliary task: every token in the corpus is a training example, which is why decoder-only pretraining is more sample-efficient than MLM and why it scaled first.

Two models by the numbers#

Two decoder-only models of different generations make the architecture concrete.

GPT-2 small (Radford et al., 2019): byte-level BPE vocabulary of 50,257, learned positions up to 1,024, 12 pre-norm blocks with $d = 768$, 12 heads and $d_{\text{ff}} = 3{,}072$. The attention projections are fused into one $768 \times 2{,}304$ matrix producing $Q, K, V$ together. Parameters: $12 \times 12 \times 768^2 \approx 85$ M in the blocks plus $50{,}257 \times 768 \approx 38.6$ M in the embedding, which is also the output matrix (tied), plus $1{,}024 \times 768$ positions: 124 M in total, the count the fine-tuning note starts from.

Llama 3.2 1B: vocabulary 128,256; 16 pre-norm blocks with $d = 2{,}048$; RMSNorm; rotary positions; a gated FFN with three $2{,}048 \times 8{,}192$ matrices; and grouped-query attention, with a $2{,}048 \times 2{,}048$ query projection next to $2{,}048 \times 512$ key and value projections: 32 query heads of width 64 share 8 key–value heads, which cuts the memory of cached keys and values by four at generation time. Per block: 10.5 M attention + 50.3 M FFN ≈ 60.8 M; 16 blocks ≈ 973 M; embeddings $128{,}256 \times 2{,}048 \approx 263$ M, tied with the output; about 1.24 B in all. The 1B model has ten times GPT-2 small’s parameters and a vocabulary 2.5 times larger, and the design changes (pre-norm, RMSNorm, RoPE, gated FFN, GQA) are the ones the architecture note listed as modern defaults.

What a base model does#

A pretrained decoder is a text continuer. Given Find the value of n in the equation n + 2 = 6., a 1 B base model produces a plausible continuation of a web page containing that sentence: it restates the problem, wanders through some irrelevant rules of algebra, finds n = 4, then keeps going with “If n is an integer, what is the value of n? n = 4”, “n = 5”, “n = 6”, and so on until the token budget runs out, because such lists are what follows such sentences in its training data. Given the same problem written as a completion, … Answer: n = , it emits 4 and then continues with a numbered list of equations. The model has the knowledge; it does not have the behaviour of answering and stopping. That behaviour is added by the instruction tuning note, and the base and tuned checkpoints there have identical architectures.

Decoding: turning probabilities into text#

The model outputs a distribution at each step; something has to pick a token from it, append the token, and repeat. The choice shapes the text as much as the model does.

Figure

The generation loop and where the decoding choices act

Greedy and beam search#

Greedy takes $\arg\max_w P(w \mid w_{<t})$. It is deterministic and cheap, and on open-ended prompts it loops: a 135 M-parameter model asked to continue a story produces “Lily’s mom, Mrs. Jenkins, was very kind and helpful” and then repeats the sentence, because once a phrase has been generated it is the most probable continuation of itself. Beam search (the translation note) finds higher-probability sequences and makes the problem worse for open-ended text: the most probable long text is bland and repetitive (Holtzman et al., 2020). It remains the right tool when there is a correct output, as in translation.

Temperature#

Divide the logits by $\tau$ before the softmax:

$$P_\tau(w) = \frac{\exp(z_w / \tau)}{\sum_{w'} \exp(z_{w'} / \tau)}.$$

$\tau = 1$ is the model’s distribution; $\tau \to 0$ approaches greedy; $\tau > 1$ flattens. For three logits $(2, 1, 0)$:

$\tau$	$P_\tau$
0.5	$(0.867, 0.117, 0.016)$
1	$(0.665, 0.245, 0.090)$
2	$(0.506, 0.307, 0.186)$

Temperature rescales all tokens, including the implausible tail, so high temperatures produce incoherent text as easily as varied text.

Truncated sampling: top-$k$ and top-$p$#

Sample, but only from the head of the distribution. Top-$k$ keeps the $k$ most probable tokens and renormalizes; it is predictable and ignores how peaked the distribution is, so $k = 40$ is too many when one token has 95 % of the mass and too few when the distribution is flat. Top-$p$ (nucleus sampling, Holtzman et al., 2020) keeps the smallest set $S$ with $\sum_{w \in S} P(w) \ge p$; the set adapts to the shape of the distribution, which is why $p \approx 0.9$ is the usual default for open-ended generation. Both remove the long tail that temperature alone would sometimes sample.

Penalties and contrast#

A repetition penalty $\rho > 1$ divides the logit of every token already in the context by $\rho$ if positive and multiplies it if negative (Keskar et al., 2019); OpenAI-style frequency and presence penalties instead subtract a term proportional to the token’s count. Both suppress loops at the cost of sometimes suppressing a token that should repeat (a name).

Contrastive search (Su et al., 2022) scores candidates by $\lambda \log P(w) - (1 - \lambda)\max_{j<t}\cos(h_w, h_j)$: probability minus the similarity of the candidate’s hidden state to earlier ones, so repetition of representation is penalized, not just of tokens. Typical sampling (Meister et al., 2023) keeps tokens whose surprise $-\log P(w)$ is close to the entropy of the distribution, discarding both the predictable and the bizarre. DoLa (Chuang et al., 2024) decodes from the difference between the logits of a late layer and an early layer, on the hypothesis that early layers carry generic predictions and late layers specific ones; on the same 135 M-parameter model and story prompt as above, without the repetition penalty that the method’s documentation recommends, it degenerates into “Max is 11 and 1 and 1 and 1…”.

Method	Knob	Deterministic	Typical use
Greedy	—	Yes	Short factual answers; loops on long text
Beam	$B$, $\alpha$	Yes	Translation, summarization with references
Temperature	$\tau$	No	Combined with truncation
Top-$k$	$k$	No	Legacy default
Top-$p$	$p$	No	Open-ended generation
Repetition penalty	$\rho$	—	Added to any of the above
Contrastive search	$\lambda$ or $\alpha$, $k$	Yes	Coherent long text without sampling

Making generation faster#

Each generated token requires a forward pass over the prefix. Two mechanisms recover most of the waste.

KV caching. Keys and values of earlier positions do not change when a token is appended; cache them and compute only the new position’s query, key and value. Memory per token is $2 \times L \times d_{kv}$ numbers, which is why grouped-query attention, which shrinks $d_{kv}$, matters for serving.

Speculative decoding (Leviathan, Kalman and Matias, 2023). A small draft model proposes $k$ tokens; the large target model verifies them in one forward pass over the extended prefix; accepted tokens are kept and the first rejected one is replaced by the target’s own choice. With greedy decoding the output is identical to the target model’s; with sampling, an acceptance rule makes the output distribution identical. With a 1.7 B target and a 135 M draft, 128 greedy tokens take 41 s instead of 101 s on the same hardware. The texts generated with and without the draft agree for the first sentence and then diverge. In exact arithmetic they would be identical; the likely cause is that the verification pass scores several tokens at once in 16-bit floating point, its logits differ slightly from those of one-token steps, and a near-tie between two continuations is enough to change a greedy choice. Serving engines add continuous batching (requests join and leave a running batch), paged KV memory and prefix caching (a shared long prompt is encoded once), which raise throughput further without changing the model.

What changed#

	Recurrent LM	Decoder-only Transformer
Context	Whole prefix, through a forgetting state	Every earlier position within the context window, by attention
Training over a sequence	$T$ sequential steps	One parallel pass with a causal mask
Scale reached	Tens of millions of parameters	$10^8$ to $10^{12}$
Use of the pretrained model	Perplexity, sampling	Completion, few-shot prompting, then tuning

Limitations#

• Left to right only. The representation of token $t$ never sees $t + 1$; for understanding tasks an encoder’s bidirectional vector is a better feature. The model-families note compares.
• Fluent is not correct. The objective rewards plausible continuations, so a base model fabricates with the same confidence it states facts.
• A continuer, not an assistant. Without further training the model does not follow instructions, stop, or refuse.
• Generation is sequential and memory-bound at inference even though training is parallel.
• Decoding choices are part of the system. The same model gives different text under different samplers; evaluations must fix them.

What you should remember#

• Objective: $-\frac1T\sum_t \log P(w_t \mid w_{<t})$ over every token; the causal mask trains all positions in one pass.
• GPT-2 small: 12 blocks, $d = 768$, 124 M, tied embeddings. Llama 3.2 1B: 16 blocks, $d = 2{,}048$, GQA, RoPE, gated FFN, ≈ 1.24 B.
• A base model continues text; prompting it as a completion works better than asking it a question.
• Decoding: greedy loops, beam search is for reference-matched tasks, temperature rescales, top-$p$ truncates adaptively, penalties and contrastive search target repetition.
• KV caching and speculative decoding speed generation without changing the output, up to floating-point effects.

Notebook#

Three notebooks. Cell numbers count every cell, markdown included, from 1. Quoted numbers come from the run saved with the course; cells changed since then for current library versions have no saved output.

week08_modern_LLM.ipynb: cells 6–11 load the Llama 3.2 1B base checkpoint in a text-generation pipeline with up to 512 new tokens and run the equation prompt as a question and as a completion; cells 18–20 print the module trees of the base and instruction-tuned checkpoints, from which the Llama 3.2 1B figures above are read, and confirm they are string-identical.

week10_PEFT.ipynb: cells 11–14 load GPT-2 small, add a [PAD] token, print the module tree, from which the GPT-2 small figures above are read, and the parameter count 124,442,112: GPT-2 small’s 124,439,808, plus 768 for the new token’s embedding and 1,536 for the two-class output layer.

week11_pt0_Decoding_and_Sampling.ipynb:

• Cells 4–7 state the temperature formula and define generate_once with max_new_tokens, temperature, top_k, top_p, typical_p and repetition_penalty passed to generate.
• Cells 9–12 run greedy (the Mrs. Jenkins loop), $\tau = 0.7$, top-$k$ = 40 and top-$p$ = 0.9 on “Continue the story: Once upon a time” with a 135 M instruction-tuned model and print the texts.
• Cells 13–15 explain and run typical sampling ($p = 0.2$) and top-$p$ with repetition penalty 1.2.
• Cells 16–19 run contrastive search (penalty_alpha=0.6, top_k=4) and DoLa (dola_layers="low"), both through custom-generation repositories; DoLa runs without a repetition penalty and produces the loop quoted above.
• Cells 20–23 pair a 1.7 B target with a 135 M draft via assistant_model and time 128 greedy tokens: 101.38 s without the draft (cell 22) and 40.79 s with it (cell 23), with the same seed and prompt. The texts diverge after the first sentence, as discussed above; cell 21, which loads the models and runs a first timing, has no saved output.
• Cells 24–33 describe vLLM (paged attention, continuous batching, prefix caching) and run a 360 M model with and without a shared long prompt; timings for those cells are not in the saved outputs.

Next#

Encoders fill in, decoders continue, and the translation model of the previous section did both with an encoder–decoder. The next note puts the three arrangements side by side with their objectives, asks which to pick for which task, and explains why decoder-only models came to dominate.

References#

• A. Radford, K. Narasimhan, T. Salimans, I. Sutskever. Improving Language Understanding by Generative Pre-Training. OpenAI, 2018. GPT-1.
• A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever. Language Models are Unsupervised Multitask Learners. OpenAI, 2019. GPT-2.
• T. B. Brown et al. Language Models are Few-Shot Learners. NeurIPS, 2020. GPT-3 and in-context learning.
• A. Holtzman, J. Buys, L. Du, M. Forbes, Y. Choi. The Curious Case of Neural Text Degeneration. ICLR, 2020. Nucleus sampling and the failure of likelihood maximization.
• N. S. Keskar, B. McCann, L. R. Varshney, C. Xiong, R. Socher. CTRL: A Conditional Transformer Language Model for Controllable Generation. arXiv, 2019. The repetition penalty.
• Y. Su, T. Lan, Y. Wang, D. Yogatama, L. Kong, N. Collier. A Contrastive Framework for Neural Text Generation. NeurIPS, 2022. Contrastive search.
• C. Meister, T. Pimentel, G. Wiher, R. Cotterell. Locally Typical Sampling. TACL, 11, 2023.
• Y.-S. Chuang, Y. Xie, H. Luo, Y. Kim, J. Glass, P. He. DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models. ICLR, 2024.
• Y. Leviathan, M. Kalman, Y. Matias. Fast Inference from Transformers via Speculative Decoding. ICML, 2023.
• W. Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP, 2023. vLLM.
• J. Ainslie et al. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. EMNLP, 2023. Grouped-query attention.