Positional representations

The Transformer note added a positional signal $P$ to the token embeddings before the first block and deferred what $P$ is. This note starts from why the signal is necessary, which takes one line of algebra, then builds the encoding used in the original paper and the alternatives that replaced it.

Notation. $X \in \mathbb R^{n \times d}$ the embedded sequence; $\Pi$ an $n \times n$ permutation matrix; $\mathrm{pos} \in \{0, \dots, n - 1\}$ a position; $P \in \mathbb R^{n \times d}$ the positional matrix with rows $P_{\mathrm{pos}}$.

Baseline: no positions#

Apply self-attention to a permuted input $\Pi X$. The projections are per row, so $Q, K, V$ become $\Pi Q, \Pi K, \Pi V$. Then

$$\operatorname{softmax}\big((\Pi Q)(\Pi K)^\top / \sqrt{d_k}\big)\,\Pi V = \operatorname{softmax}\big(\Pi\, Q K^\top \Pi^\top / \sqrt{d_k}\big)\,\Pi V = \Pi\,\operatorname{softmax}\big(QK^\top/\sqrt{d_k}\big)\,V ,$$

using $\Pi^\top \Pi = I$ and the fact that a row-wise softmax commutes with permuting rows and columns together. The output of the permuted input is the permuted output: self-attention is permutation-equivariant. The feed-forward layer, LayerNorm and residuals act per position and are equivariant too. The argument holds for unmasked attention, or for a mask permuted along with the tokens, as a padding mask is. A causal mask is attached to positions, not to tokens, so it breaks arbitrary permutations: in a decoder the first token attends only to itself, whatever word it is. An encoder has no causal mask, so an encoder without positional input computes, for “dog bites man” and “man bites dog”, the same three vectors in a different order, and a classifier that pools them cannot tell the sentences apart. A recurrent network never had this problem, because it read tokens in sequence; the price of parallelism is that order must be put back by hand.

Sinusoidal encoding#

Vaswani et al. (2017) define, for position $\mathrm{pos}$ and dimension index $i$,

$$P_{\mathrm{pos},\,2i} = \sin\!\Big(\frac{\mathrm{pos}}{10000^{2i/d}}\Big), \qquad P_{\mathrm{pos},\,2i+1} = \cos\!\Big(\frac{\mathrm{pos}}{10000^{2i/d}}\Big), \qquad i = 0, \dots, d/2 - 1 .$$

Each pair of dimensions $(2i, 2i + 1)$ is a sine and cosine of the same angle $\omega_i \cdot \mathrm{pos}$ with $\omega_i = 10000^{-2i/d}$. The frequencies form a geometric progression: dimension pair 0 has wavelength $2\pi$ positions, the last pair, $i = d/2 - 1$, has wavelength $2\pi \cdot 10000^{1 - 2/d}$, which approaches $10000 \cdot 2\pi$ as $d$ grows: about 19,869 positions for $d = 16$ and 60,611 for $d = 512$. Low dimensions change quickly with position and resolve neighbours; high dimensions change slowly and encode coarse location.

Figure

Sinusoidal positional encoding: four of the sixteen dimensions

Worked example#

$d = 4$, so two frequency pairs: $\omega_0 = 1$ and $\omega_1 = 10000^{-1/2} = 0.01$.

pos	$\sin(\mathrm{pos})$	$\cos(\mathrm{pos})$	$\sin(0.01\,\mathrm{pos})$	$\cos(0.01\,\mathrm{pos})$
0	0	1	0	1
1	0.841	0.540	0.010	1.000
2	0.909	−0.416	0.020	1.000
3	0.141	−0.990	0.030	1.000

Rows are distinct, bounded in $[-1, 1]$, and the slow pair changes by $0.01$ per step, so positions 1 and 2 are closer in the slow dimensions than positions 1 and 30. The vector is added to the token embedding (scaled by $\sqrt d$ so that the two have comparable magnitude), and the same table is reused for every sequence; it is added once, before the first block.

Properties#

Relative shift is a rotation. For a fixed offset $k$, the pair at position $\mathrm{pos} + k$ is a linear function of the pair at $\mathrm{pos}$, independent of $\mathrm{pos}$:

$$\begin{pmatrix} \sin\omega(\mathrm{pos} + k) \\ \cos\omega(\mathrm{pos} + k) \end{pmatrix} = \begin{pmatrix} \cos\omega k & \sin\omega k \\ -\sin\omega k & \cos\omega k \end{pmatrix} \begin{pmatrix} \sin\omega\,\mathrm{pos} \\ \cos\omega\,\mathrm{pos} \end{pmatrix},$$

which is the angle-addition identity. The authors hypothesized that this lets a linear projection in attention pick out relative positions; the dot product $P_{\mathrm{pos}} \cdot P_{\mathrm{pos}+k}$ indeed depends on $k$ only. Whether the trained model uses this is empirical.

No parameters and no maximum length in principle. The formula evaluates at any position. In practice models are trained on sequences up to some length and do worse beyond it, because the attention patterns learned on short sequences do not transfer to combinations of positional vectors never seen in training.

Repetition within a dimension is not a collision. Any single sine repeats, so positions $\mathrm{pos}$ and $\mathrm{pos} + 2\pi/\omega_i$ share dimension $2i$. The other dimensions have incommensurable periods, so two positions in any practical range do not share the whole vector.

Learned absolute positions#

Replace the formula by a table $P \in \mathbb R^{n_{\max} \times d}$ of free parameters, nn.Embedding(max_len, d), trained with everything else. BERT does this with $n_{\max} = 512$, GPT-2 with 1,024, and the translation model of the next note with 32. It is simpler, performs about the same as sinusoids on tasks of the training length (Vaswani et al., 2017, Table 3, row E), and has a hard maximum: a position beyond $n_{\max}$ has no row. The table costs $n_{\max} \times d$ parameters, 0.4 M for BERT-base.

Relative and rotary positions#

Both schemes above encode absolute position and leave attention to infer relative distances. Later models encode the relative offset directly, because what matters for most linguistic relations is how far apart two tokens are, not where in the document they sit.

Relative position biases add a learned scalar $b_{j-i}$ to the score of query $i$ and key $j$ (T5, which shares one scalar among offsets grouped into buckets that widen with distance), or a fixed penalty proportional to $|j - i|$ (ALiBi, Press, Smith and Lewis, 2022), with no positional vector at the input at all.

Rotary position embeddings (RoPE, Su et al., 2024) move the sinusoids from the input to the attention scores. Each query and key of a head, of width $d_k$, is split into $d_k/2$ two-dimensional pairs, and pair $i$ at position $\mathrm{pos}$ is rotated by angle $\omega_i\,\mathrm{pos}$, with $\omega_i = 10000^{-2i/d_k}$. Because a dot product of two rotated vectors depends only on the difference of the rotation angles, the score $q_m \cdot k_n$ depends on the content of the two tokens and on $m - n$, not on $m$ and $n$ separately. The shift property of the sinusoidal scheme becomes exact and built into every attention layer. Llama 3.2 1B, the model of the GPT and instruction tuning notes, uses RoPE. The rotation has no parameters, and the base $10000$ can be increased to stretch a trained model to longer contexts.

Scheme	Where applied	Parameters	Max length	Encodes
Sinusoidal	Added to input	0	None in principle	Absolute, with shift structure
Learned table	Added to input	$n_{\max} d$	$n_{\max}$	Absolute
Relative bias / ALiBi	Attention scores	$O(n_{\max})$ or 0	Extrapolates somewhat	Relative offset
RoPE	Queries and keys	0	Trained length, extendable	Relative offset in the dot product

What changed#

	Recurrent model	Transformer + positions
Source of order	Sequential reading	An explicit positional signal
Cost	Implicit	$0$ to $n_{\max} d$ parameters
Parallel over positions	No	Yes
Length generalization	Any length, weak memory	Depends on the scheme

Limitations#

• Absolute schemes do not extrapolate. A learned table has a hard limit; sinusoids have a soft one.
• Added signals compete with content. In the input-addition schemes the positional vector shares the $d$ dimensions with the token vector; the model must learn to separate them.
• Long contexts stress every scheme. Which positional method generalizes best to lengths beyond training is an active empirical question, and the answer interacts with the data and the attention implementation.
• Positions are indices, not structure. None of the schemes knows about sentences, paragraphs or syntax; those have to be learned from the tokens.

What you should remember#

• Unmasked self-attention is permutation-equivariant: $\operatorname{Attn}(\Pi X) = \Pi\,\operatorname{Attn}(X)$. Order must be added; a causal mask breaks the symmetry, since it is fixed by position.
• Sinusoidal: $\sin$ and $\cos$ of $\mathrm{pos} \cdot 10000^{-2i/d}$, wavelengths in geometric progression from $2\pi$ to $2\pi \cdot 10000^{1 - 2/d}$, just under $10000 \cdot 2\pi$ for large $d$; a shift by $k$ is a fixed rotation of each pair.
• Learned tables are simpler and bounded by $n_{\max}$.
• RoPE rotates queries and keys so that scores depend on relative offset; no parameters, used by current decoder-only models.
• The scheme sets how the model handles lengths it was not trained on.

Notebook#

week05_transformer.ipynb. Cell numbers count every cell, markdown included, from 1. Quoted numbers come from the run saved with the course; cells changed since then for current library versions have no saved output.

• Cell 34 states the sinusoidal formulas and the wavelength range, and the dropout applied to the sum of embeddings and positions.
• Cell 35 implements PositionalEncoder: the frequencies are computed as $\exp(-2i \ln 10000 / d)$ for even $2i$, the table of shape [1, max_length, d] is filled with sines in even columns and cosines in odd columns and stored as a non-trainable parameter, and forward adds the first T rows to the input and applies dropout. Cell 36 points to rotary embeddings as the modern alternative.

week04_lstm_attention_transformer.ipynb, cell 58, uses a learned table nn.Embedding(max_len, hidden_size) with max_len = 32, indexed by arange(T) % max_len so that longer inputs wrap around rather than fail.

week08_modern_LLM.ipynb, cells 18–19, prints the module trees of the Llama 3.2 1B checkpoints; RoPE appears in them as the rotary_emb module.

Next#

With attention, the block and positions in place, the architecture is complete. The next note returns to the German–English task where the course left the LSTM at BLEU 0.152, trains a Transformer with the same parameter budget, measures what cross-attention, depth and beam search each contribute, and reads the attention maps.

References#

• A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin. Attention Is All You Need. NeurIPS, 2017. §3.5 (positional encoding), Table 3 row E (learned versus sinusoidal).
• J. Gehring, M. Auli, D. Grangier, D. Yarats, Y. N. Dauphin. Convolutional Sequence to Sequence Learning. ICML, 2017. Learned absolute positions in a non-recurrent model.
• P. Shaw, J. Uszkoreit, A. Vaswani. Self-Attention with Relative Position Representations. NAACL, 2018.
• O. Press, N. A. Smith, M. Lewis. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. ICLR, 2022. ALiBi.
• J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, Y. Liu. RoFormer: Enhanced Transformer with Rotary Position Embedding. Neurocomputing, 568, 2024. RoPE.
• A. Kazemnejad. Transformer Architecture: The Positional Encoding. 2019. The rotation-matrix derivation of the shift property.