BERT and masked language modeling

Every model in the course so far was trained on the data of its own task. The Transformer trained from scratch on 120,000 news articles in the architecture note did not beat a bag of counts. Unlabelled text, by contrast, is unlimited, and the Word2Vec note showed that a prediction task on it yields reusable representations. BERT (Devlin et al., 2019) applies the same idea to a full Transformer encoder: pretrain it to fill in blanks, then reuse the whole network.

Notation. Input tokens $w_{1:n}$, of which a subset $M$ is selected for prediction; the corrupted input is $\tilde w$; the encoder output at position $t$ is $h_t \in \mathbb R^d$.

Baseline: a language model objective on an encoder#

The natural self-supervised task is next-token prediction, which gives the decoder-only models of the next note. An encoder is bidirectional: position $t$ attends to every position, including $t + 1$. Train it to predict $w_{t+1}$ from $h_t$ and the network can copy the answer from the input through one attention head; the loss goes to zero and nothing is learned. Left-to-right prediction and bidirectional context are incompatible in the same stack.

Masked language modelling#

For each training sequence, 15 % of the token positions are sampled into $M$. For a position in $M$, the input token is replaced by [MASK] with probability 0.8, by a random vocabulary token with probability 0.1, and left unchanged with probability 0.1. The model predicts the original token at every position in $M$ through a softmax over the vocabulary, and the loss is the cross-entropy over those positions only:

$$\mathcal L_{\text{MLM}} = -\sum_{t \in M} \log P(w_t \mid \tilde w_{1:n}), \qquad P(\cdot \mid \tilde w) = \operatorname{softmax}(W_{\text{out}}\, g(h_t)),$$

where $g$ is a small transform (a linear layer, GELU and LayerNorm) and $W_{\text{out}}$ is tied to the input embedding matrix. The 80/10/10 rule exists because [MASK] never appears at fine-tuning time: if the model only ever saw real tokens at unmasked positions, it would learn to ignore them, and the random and unchanged cases force it to build a useful representation of every position.

For a 128-token sequence, about 19 positions are selected; roughly 15 become [MASK], 2 become a random token and 2 stay as they are. The model is trained on 19 predictions per 128 tokens read, a fraction that makes MLM slower to converge per token than next-token prediction, which trains on all 128.

Figure

Masked language modelling on one sentence

BERT also pretrains a second task, next-sentence prediction: the input is two segments, and a classifier on the first position predicts whether the second followed the first in the corpus. Later work found it unnecessary (Liu et al., 2019); it survives in BERT’s input format.

The input#

A BERT input is [CLS] segment A [SEP] segment B [SEP], padded to the batch length. Three embedding tables are summed: token (WordPiece, 30,522 entries for the uncased English model), position (a learned table of 512 rows, so 512 tokens is the hard maximum), and segment (token_type_ids, 2 rows, marking A or B). The sum is normalized and passed to the encoder. [CLS] is a token with no content whose output vector is trained, through next-sentence prediction, to summarize the pair; it is the conventional handle for sentence-level tasks.

The architecture by the numbers#

BERT-base is 12 post-norm encoder blocks with $d = 768$, 12 heads of width 64, $d_{\text{ff}} = 3{,}072$ and GELU:

Component	Parameters
Token embeddings $30{,}522 \times 768$	23.4 M
Position embeddings $512 \times 768$	0.4 M
12 blocks × $12 d^2$	84.9 M
Pooler (one linear layer on [CLS])	0.6 M
Total	≈ 110 M

The arithmetic is the one from the architecture note. BERT-large doubles the depth and widens to 1,024 (340 M). Pretraining used 3.3 billion words of books and Wikipedia; the resulting checkpoint is public and loads in one line.

From static to contextual embeddings#

The output $h_t$ is a vector for token $t$ in this sentence. It was computed by twelve rounds of attention over every other token, so it depends on them. The word bank gets different vectors in “river bank” and “bank account”; the Word2Vec vector of the embeddings note was one vector for both. This is the sense in which a pretrained encoder produces contextual embeddings, and it is the reason the same network is useful for tasks that static vectors could not serve:

• A token’s vector can be classified by a linear layer to tag it (the NER note) or to mark it as the start or end of an answer (the QA note).
• The vector of [CLS], or the mean of all vectors, gives one vector per sentence that can be compared with others by cosine similarity. Pooling alone does not give a useful geometry, though: raw BERT [CLS] vectors rank sentence pairs poorly, and a meaningful sentence space needs sentence-level, usually contrastive, training (sentence similarity note). That trained space is what the semantic search note builds retrieval on.
• The whole stack can be fine-tuned on a labelled task, which the transfer learning note measures.

Figure

Encoder-only model: one contextual vector per token, two kinds of head

What the vectors contain is an empirical question studied by probing: lower layers carry mostly lexical and syntactic information, upper layers more semantic and task-oriented information (Tenney, Das and Pavlick, 2019).

What the pretrained head can do#

The MLM head is a usable model on its own. Feed Elephants are playing in the zoo with monkeys. The story is about [MASK]. and read the distribution at the masked position:

Token	$P$
elephants	0.216
monkeys	0.109
animals	0.086
elephant	0.039
children	0.022

The top predictions are the entities of the sentence and their hypernym, read off from a model that was never told what a “story” is or asked a question. This is the mechanism behind zero-shot use of BERT, in which a classification task is rephrased as a fill-in-the-blank and the answer is read from the vocabulary; the transfer learning note runs it on AG News and finds both its promise and its limits. The fill-mask pipeline in the transformers library wraps tokenization, the model and the top-$k$ readout into one call and returns the same numbers.

What changed#

	Transformer trained on the task	Pretrained encoder
Training data	Task labels (thousands to millions)	Unlabelled text (billions of tokens)
Objective	Task loss	Fill in masked tokens
Token representation	Learned for this task	Contextual, reusable
Direction	Any	Bidirectional
Use	One task	Features, zero-shot, fine-tuning
Cost to the user	Full training	Download 110 M parameters

Limitations#

• No generation. The encoder predicts hidden tokens given the rest; it cannot continue a text. The decoder-only models of the next note can.
• Pretrain–fine-tune mismatch. [MASK] is seen only in pretraining; the 80/10/10 rule reduces but does not remove the gap.
• Sample inefficiency. 15 % of positions carry loss, so each sequence yields few training signals.
• 512 tokens. The learned position table fixes the maximum length.
• Static vocabulary and casing. bert-base-uncased cannot distinguish US from us; the cased variant can, at the cost of a larger effective vocabulary.
• Needs adaptation. Out of the box, the zero-shot route is weak; the model’s value appears after fine-tuning or when its vectors feed another model.

What you should remember#

• MLM: hide 15 % of tokens (80 % [MASK], 10 % random, 10 % unchanged), predict them with full bidirectional attention, loss on masked positions only.
• Input: [CLS] A [SEP] B [SEP] with token + position + segment embeddings; 512 positions maximum.
• BERT-base: 12 blocks, $d = 768$, 12 heads, ≈ 110 M parameters, 85 M of them in the blocks.
• Output vectors are contextual: one vector per token per sentence, usable for tagging, span extraction, sentence similarity and fine-tuning.
• The MLM head answers cloze prompts without any task training, which is zero-shot use.

Notebook#

week06_BERT.ipynb. Cell numbers count every cell, markdown included, from 1. Quoted numbers come from the run saved with the course; cells changed since then for current library versions have no saved output.

• Cells 3–6 load AG News and tokenize it with bert-base-uncased; the printed example shows the [CLS] (101) and [SEP] (102) ids around the text.
• Cell 9 loads AutoModel.from_pretrained('bert-base-uncased') and prints the module tree used for the table above: three embedding tables, 12 BertLayers with 768-wide attention and a 3,072-wide intermediate layer, and the pooler.
• Cell 12 loads AutoModelForSequenceClassification with 10 labels and shows the warning that the classifier weights are newly initialized: task heads are not part of the checkpoint.
• Cells 18–23 load the MLM head (AutoModelForMaskedLM), build the elephant prompt, take the softmax at the [MASK] position (index −3) and print the top 10 tokens quoted above.
• Cells 26–30 repeat the same with the fill-mask pipeline and get identical probabilities.

Next#

A pretrained encoder represents text; it does not write it. The next note trains the decoder arrangement on the plain next-token objective of the n-gram note, scaled up, and looks at what such a model does when prompted, how its probabilities are turned into text, and why the result is a text continuer rather than an assistant.

References#

• J. Devlin, M.-W. Chang, K. Lee, K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL, 2019. §3.1 (MLM and NSP), Appendix A.1 (the 80/10/10 rule).
• Y. Liu et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv, 2019. Dropping NSP, dynamic masking.
• I. Tenney, D. Das, E. Pavlick. BERT Rediscovers the Classical NLP Pipeline. ACL, 2019.
• M. E. Peters et al. Deep Contextualized Word Representations. NAACL, 2018. ELMo, the first widely used contextual embeddings, from bidirectional LSTM language models.
• J. Alammar. The Illustrated BERT, ELMo, and co.. 2018.
• Hugging Face. BERT model documentation. The AutoModelFor* heads used in the notebook.