Sequence-to-sequence models

Classification maps a sequence to a label; language modelling continues a sequence. Translation needs both halves: read one sequence fully, then write a different one whose length is not known in advance. The recurrent cells of the previous note can do each half. This note joins them into the encoder–decoder of Sutskever, Vinyals and Le (2014) and Cho et al. (2014), trains it on 29,000 German–English pairs, and measures it.

Notation. Source tokens $x_1, \dots, x_S$ (German), target tokens $y_1, \dots, y_T$ (English). Encoder states $h_1, \dots, h_S$; decoder states $s_1, \dots, s_T$. The target is bracketed by a start symbol ([CLS] in the tokenizer used) and an end symbol ([SEP]).

The task and its data#

Multi30k (Elliott et al., 2016) pairs 29,000 image captions in German with their English translations, plus 1,014 validation and 1,000 test pairs. Captions are short: after WordPiece tokenization both languages fit in 32 tokens for almost all sentences, and sequences are truncated at that length. A typical pair:

Zwei junge weiße Männer sind im Freien in der Nähe vieler Büsche. Two young, White males are outside near many bushes.

Each language has its own pretrained WordPiece tokenizer (German 31,102 entries, English 30,522); the course’s tokenization note describes them.

Baseline: encoder–decoder with a context vector#

Encoder. A bidirectional LSTM reads the source embeddings. Its final forward and backward hidden states (each $H = 128$) are concatenated to $2H$ and projected back to $H$ by a linear layer; the cell states are treated the same way. The result initializes the decoder: $(s_0, c_0) = (W_h [\overrightarrow{h}_S; \overleftarrow{h}_1],\; W_c [\overrightarrow{c}_S; \overleftarrow{c}_1])$.

Decoder. A unidirectional LSTM, started from $(s_0, c_0)$, reads the target shifted right by one (start symbol first) and at each step emits logits over the English vocabulary:

$$P(y_t \mid y_{<t}, x) = \mathrm{softmax}(W_o s_t), \qquad (s_t, c_t) = \mathrm{LSTM}\big(y_{t-1}, (s_{t-1}, c_{t-1})\big).$$

The source enters only through $(s_0, c_0)$. Combined with the chain rule, the model defines $P(y \mid x) = \prod_t P(y_t \mid y_{<t}, x)$.

Figure

Encoder and decoder: what goes where

What happens, in the order of the numbers in the figure:

1. Encode. The encoder applies its recurrent cell to the source tokens one at a time, exactly as in the RNN note; here the cell is an LSTM and the encoder is bidirectional.
2. Hand over. The final states, projected to the decoder’s width, become the decoder’s initial state pair $(s_0, c_0)$, $2H = 256$ numbers. This fixed-size pair is all the decoder will ever see of the source.
3. Start. The decoder reads the start symbol, updates its state from $s_0$ to $s_1$ and outputs a distribution over the English vocabulary; the most probable word is the first word of the translation.
4. Feed back. The word just produced becomes the next input, and the step repeats. In training the gold previous word is fed instead (teacher forcing, below).
5. Stop. Generation ends when the decoder produces the end symbol or after 32 tokens.

The encoder and the decoder are each the recurrent network of the previous notes; what is new is the hand-over in the middle, in the amber outline, which is also the bottleneck: whatever the length of the source, it reaches the decoder through $(s_0, c_0)$ alone.

Training is teacher forcing, as for the recurrent language model: the decoder input is the gold target up to $t - 1$, the loss is cross-entropy against the gold token at $t$, summed over positions and ignoring padding. In code: logits for positions $1..T-1$ are compared with target ids at positions $2..T$ (logits[:, :-1] against ids[:, 1:]).

Generation is greedy decoding: start with the start symbol, take the argmax at each step, feed it back, stop at the end symbol or after 32 tokens. The decoder state is carried across steps, so generation costs one LSTM step per output token.

Sizes: $d = H = 128$. The whole model has 8.38 M parameters, of which 7.9 M are the shared embedding table and the output matrix; the recurrent and projection layers are 0.46 M. The model uses one embedding table for both languages and indexes it with ids from either tokenizer, a shortcut. The table is sized for the larger German vocabulary, so every id of either tokenizer is in range, but the two tokenizers were trained independently: the same id names unrelated tokens in the two languages, and one row has to serve both. The model can still learn, since the encoder and the decoder read the shared rows in different contexts, but each such row is a compromise between two tokens. Separate source and target tables, or one tokenizer trained on both languages, avoid the aliasing.

BLEU: measuring a translation#

There is no single correct translation, so accuracy against the reference is meaningless, and token-level perplexity does not evaluate the sentence the model actually produced. BLEU (Papineni et al., 2002) compares the n-grams of a candidate with those of one or more references:

$$\mathrm{BLEU} = \mathrm{BP} \cdot \exp\Big(\sum_{n=1}^{N} w_n \log p_n\Big), \qquad w_n = \tfrac1N,\; N = 4.$$

$p_n$ is the modified n-gram precision: the fraction of candidate n-grams that appear in the reference, where each reference n-gram can be matched at most as many times as it occurs there (so repeating the the the does not score). The geometric mean over $n = 1..4$ rewards both correct words and correct phrases. The brevity penalty

$$\mathrm{BP} = \exp\Big(-\Big[\frac{r}{c} - 1\Big]_+\Big)$$

with $r$ the total reference length and $c$ the total candidate length penalizes outputs shorter than the reference (precision alone would reward a one-word candidate that happens to be right). If the candidate is 20 % shorter than the reference, $r/c = 1.25$ and $\mathrm{BP} = e^{-0.25} = 0.78$; a candidate at least as long as the reference is not penalized. BLEU is computed over the whole corpus, not averaged per sentence, and ranges from 0 to 1 (often reported as 0–100).

Two cautions. BLEU implementations differ in tokenization and smoothing of zero $p_n$ and give different numbers on the same outputs, so comparisons are valid within one implementation only (Post, 2018); the course uses the Hugging Face evaluate implementation throughout. And BLEU measures n-gram overlap with the reference, not meaning: a correct paraphrase scores low and a fluent wrong sentence can score reasonably. The translation note returns to its use for model comparison.

Result#

After 10 epochs with Adam at learning rate $10^{-4}$ and batch size 32, the LSTM model reaches a test BLEU of 0.152. Reading the outputs explains the number:

German source	Reference	LSTM output
Ein Mann mit einem orangefarbenen Hut, der etwas anstarrt.	A man in an orange hat starring at something.	a man in a red shirt is standing on a street.
Ein Boston Terrier läuft über saftig-grünes Gras vor einem weißen Zaun.	A Boston Terrier is running on lush green grass in front of a white fence.	a young man is standing on a street with a large building.
Leute Reparieren das Dach eines Hauses.	People are fixing the roof of a house.	people are sitting on a street.

Every output is a grammatical English caption. The subject is often right (a man, people), and almost nothing else is: colours, objects and actions are replaced by the most frequent ones in the training set (red shirt, standing, street). The decoder has learned the English language model well and has learned very little about the specific source sentence it is given.

Why one vector is not enough#

The decoder’s only information about the source is the pair $(s_0, c_0)$, $2 \times 128 = 256$ numbers produced before the first target token is written. For a 20-token source that pair must hold every content word, in order, with enough precision that the decoder can retrieve the fourth adjective after it has already generated ten tokens and updated its state ten times. Two things go wrong:

• Capacity. The summary is fixed-size; the source is not. Longer sources are compressed harder. Cho et al. (2014) measured BLEU falling with sentence length for exactly this reason.
• Decay. The decoder’s state is overwritten at each step. Whatever the summary held about the end of the source has to survive the generation of the beginning of the target, through the same gated recurrence whose memory the previous note showed to be finite.

The model’s outputs are what a language model does when its conditioning signal is weak: it produces the most likely sentence of the right shape. The experiment in the next note isolates this: the same architecture with and without a path around the bottleneck differs by 0.13 BLEU.

What changed#

	Recurrent language model	Encoder–decoder
Input	A prefix of the same sequence	A separate source sequence
Conditioning	Previous tokens via the state	Source summary $(s_0, c_0)$ + previous tokens
Output length	Open-ended	Open-ended, stopped by an end symbol
Training	Teacher forcing	Teacher forcing on the target
Evaluation	Perplexity	BLEU against references

Limitations#

• The bottleneck. All source information passes through one fixed-size state pair. This is the limitation the next note removes.
• Exposure bias. Training feeds gold prefixes; generation feeds the model’s own, possibly wrong, prefixes. Errors compound and nothing in training prepares the model for them.
• Greedy decoding. Taking the argmax at each step does not give the most probable sequence; beam search in the translation note is the standard improvement.
• Sequential. Both halves are recurrent, so neither training nor inference parallelizes along the sequence.
• BLEU is a proxy. It rewards surface overlap and depends on the implementation.

What you should remember#

• Encoder–decoder: encode the source into a state, decode the target autoregressively from it; $P(y \mid x) = \prod_t P(y_t \mid y_{<t}, x)$.
• Teacher forcing with shifted targets and padding ignored, as for a language model; greedy decoding at inference.
• BLEU: geometric mean of modified n-gram precisions for $n = 1..4$ times a brevity penalty, corpus-level, implementation-dependent.
• Multi30k with a 128-unit bidirectional LSTM encoder: BLEU 0.152, fluent outputs with the wrong content.
• The failure is the fixed-size hand-over state, $(s_0, c_0)$ with 256 numbers here, not the decoder’s fluency.

Notebook#

week04_lstm_attention_transformer.ipynb. Cell numbers count every cell, markdown included, from 1. Quoted numbers come from the run saved with the course; cells changed since then for current library versions have no saved output.

• Cells 5–10 load Multi30k from the Hub (29,000 / 1,014 / 1,000 pairs) and build loaders with batch size 32; cells 13–35 set up the two tokenizers and fix MAX_LENGTH = 32.
• Cells 36–37 state the BLEU formula and load the evaluate implementation.
• Cell 39 defines Seq2SeqLSTM: bidirectional encoder, state projections, unidirectional decoder, and a greedy generate that always runs 32 steps (the output is cleaned of tokens after [SEP] by clean_text in cell 41).
• Cell 42 holds the training and evaluation loops: cross-entropy with the pad id ignored, the one-position shift between logits and targets, a per-batch token accuracy, and corpus BLEU on greedy generations.
• Cells 44–47 train for 10 epochs (the training cell has no saved output) and the markdown cell 48 states the resulting test BLEU, 0.152; cells 49–54 print the example translations quoted above with their sources and references, and cell 55 draws the same conclusion as this note.

Next#

The decoder needs access to the source while it writes, not a summary taken before it starts. The next note adds exactly that: at every decoder step, a weighted sum over all encoder states, with weights chosen by comparing the current decoder state to each of them. The bottleneck disappears, and the comparison operation it introduces turns out to be the building block of everything that follows.

References#

• I. Sutskever, O. Vinyals, Q. V. Le. Sequence to Sequence Learning with Neural Networks. NeurIPS, 2014.
• K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, Y. Bengio. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. EMNLP, 2014.
• K. Cho, B. van Merriënboer, D. Bahdanau, Y. Bengio. On the Properties of Neural Machine Translation: Encoder–Decoder Approaches. SSST, 2014. BLEU versus sentence length for the fixed-vector model.
• D. Elliott, S. Frank, K. Sima’an, L. Specia. Multi30K: Multilingual English-German Image Descriptions. VL, 2016.
• K. Papineni, S. Roukos, T. Ward, W.-J. Zhu. BLEU: a Method for Automatic Evaluation of Machine Translation. ACL, 2002.
• M. Post. A Call for Clarity in Reporting BLEU Scores. WMT, 2018.