Word embeddings

Every model so far has treated a token as an atom. The bag-of-words classifier gave cat and dog orthogonal coordinates; the n-gram model counted cat sat and dog sat as unrelated events. Whatever was learned about one word was useless for the other. This note asks what representation would let knowledge transfer between similar words, tries the obvious encodings in turn, and arrives at the dense vector that every later model in the course is built on.

Notation. $V$ is the vocabulary, $d$ the embedding dimension. The one-hot vector of token $i$ is $e_i \in \{0, 1\}^{|V|}$ with a single 1 at position $i$. The embedding table is $E \in \mathbb R^{|V| \times d}$, and $E_i$ denotes its $i$-th row.

Baseline: one-hot vectors#

A token is a vector of length $|V|$ with a 1 at its index. Every pair of distinct tokens has dot product 0 and Euclidean distance $\sqrt 2$: cat is exactly as far from dog as from therefore. The representation has no notion of similarity, and its size is the vocabulary size. A model with a first linear layer over one-hot inputs has $|V| \times h$ weights for a hidden size $h$; with $|V| = 3 \times 10^6$ (a web-scale word vocabulary) and $h = 100$, that is $3 \times 10^8$ parameters in one layer, and the activation for a sentence of 20 tokens is a $20 \times 3{,}000{,}000$ matrix that is almost entirely zeros.

Four attempts#

The problem has two parts: the vectors are too large, and they carry no similarity. Each attempt fixes one and reveals the next constraint.

1. One number per word. Replace the one-hot vector by its index, a single integer. Size is solved. But the index is arbitrary: cool might be 1,502 and awesome 7 and therefore 1,503. Words that mean the same are far apart, and a word’s neighbours in the encoding are meaningless.
2. Similar numbers for similar words. Assign indices so that synonyms are adjacent: cool 1,502, awesome 1,503. Now a single scalar has to encode every axis of similarity at once. cool is close to awesome on one axis (approval) and to cold on another (temperature); a line cannot place it next to both. Sarcasm and polysemy have no room at all.
3. A short vector of chosen features. Give each word a few hand-designed coordinates (sentiment, concreteness, part of speech). This is the right shape, but the features have to be invented and labelled by hand for every word, and the ones that matter for a task are rarely the ones we would think to write down.
4. A short vector of learned features. Keep the shape of attempt 3, and let the coordinates be learned from data. Nothing is designed; a $d$-dimensional vector per word is a parameter, and training moves it.

The embedding table#

Attempt 4 is a matrix $E \in \mathbb R^{|V| \times d}$. The vector of token $i$ is row $E_i$. In code this is a lookup, and in linear algebra it is a product with the one-hot vector:

$$E^\top e_i = E_i \in \mathbb R^{d},$$

which is the matrix-multiplication view that makes the first linear layer of a network over one-hot inputs equal to an embedding lookup: a $|V| \times h$ weight matrix multiplied by a one-hot vector just reads out one row. The layer nn.Embedding(|V|, d) is that matrix with the multiplication replaced by indexing.

Figure

An embedding layer is a row lookup

Shapes for a batch: token ids [B, T] (batch size, sequence length) become [B, T, d] after the lookup. For $|V| = 30{,}000$ and $d = 100$ the table has 3 M parameters, independent of sequence length; a sequence of 20 tokens produces a $20 \times 100$ activation instead of $20 \times 30{,}000$.

Where do the values come from? Three options, used in this order in the course:

• Learned with the task. The table is a parameter of a classifier and receives gradients from the classification loss. The CNN note does this and measures the effect.
• Pretrained on unlabelled text. The table is fitted on a large corpus with a self-supervised objective, then used as fixed input features. The next note builds the objective.
• Downloaded. Tables trained by others on billions of tokens, such as the GloVe vectors trained on Twitter used below.

Reading a vector space#

Once words are points, the geometry can be inspected. Three tools recur throughout the course.

Cosine similarity compares directions and ignores length:

$$\cos(u, v) = \frac{u \cdot v}{\|u\|\,\|v\|} = \frac{\sum_{j=1}^d u_j v_j}{\sqrt{\sum_j u_j^2}\,\sqrt{\sum_j v_j^2}} \in [-1, 1].$$

Lengths of embedding vectors vary with how often and in how many contexts a word occurs, for reasons unrelated to meaning, so the angle is the usual measure. A 32-dimensional model trained on 537,272 Quora questions lists the nearest neighbours of word as sentence (0.836), words, verb, phrase, sentences, pronounce, noun: words that occur in the same kind of question. “Similar” here means used in similar contexts, which is the only signal the training had.

Analogies test whether differences between vectors are meaningful. If the direction from man to woman encodes gender, then adding it to king should land near queen:

$$v_{\text{king}} - v_{\text{man}} + v_{\text{woman}} \approx v_{\text{queen}}.$$

With 100-dimensional GloVe vectors trained on Twitter, the nearest word to that sum (excluding the three inputs) is queen with cosine 0.705, followed by prince (0.667) and mother (0.644). The same arithmetic with latte − milk + water returns macchiato (0.608), cappuccino, warm, espresso, coffee: the drink axis survives the substitution, the specific answer is debatable. And head − foot + sock gives pants (0.621), butt, blow, with beanie (the hat one might expect) in sixth place. Analogies are a qualitative probe: when they work, they show that some directions in the space are consistent; when they return junk, they show which regularities the corpus did not contain. They are not a proof that the space is linear in meaning, and the standard analogy benchmarks are known to be sensitive to how the search excludes the input words (Linzen, 2016).

Projection to 2-D. To look at the space, reduce $d$ to 2. PCA finds the directions of largest variance: for a centred matrix $X \in \mathbb R^{n \times d}$ of $n$ word vectors it solves $\min_W \|X W W^\top - X\|_F^2$ over $W \in \mathbb R^{d \times 2}$ with orthonormal columns, and the solution is the first two right singular vectors of $X$. The plot is a linear shadow of the space: global structure is preserved, local neighbourhoods are often crushed. t-SNE (van der Maaten and Hinton, 2008) instead places points so that pairs that are close in $\mathbb R^d$ stay close in the plane, at the price of distorting large distances and depending on a perplexity setting. Distances between clusters in a t-SNE plot mean little, and the clusters themselves can mislead: at low perplexity, points drawn from a single Gaussian already form apparent clumps (Wattenberg, Viégas and Johnson, 2016). Trust a cluster only if it persists across perplexities and random seeds and its words are also close by cosine in $\mathbb R^d$. Both methods are for looking, not for measuring.

Worked example#

Three words in $d = 2$ (illustrative values):

$$v_{\text{cat}} = (0.9,\ 0.4),\qquad v_{\text{dog}} = (0.8,\ 0.5),\qquad v_{\text{therefore}} = (-0.3,\ 0.9).$$ $$\cos(\text{cat}, \text{dog}) = \frac{0.72 + 0.20}{\sqrt{0.97}\,\sqrt{0.89}} = \frac{0.92}{0.929} = 0.990, \qquad \cos(\text{cat}, \text{therefore}) = \frac{-0.27 + 0.36}{\sqrt{0.97}\,\sqrt{0.90}} = \frac{0.09}{0.934} = 0.096.$$

Two animal words point the same way; the connective is nearly orthogonal. Under one-hot encoding all three cosines would be 0.

From words to sentences, for now#

A classifier needs one vector per document, and an embedding table gives one per token. The simplest aggregate is the mean of the token vectors (skipping tokens outside the vocabulary, and returning zeros if none remain). With a 100-dimensional table that gives a 100-dimensional document vector, almost 300 times smaller than the 29,350-dimensional TF-IDF vector, and it already supports a nearest-neighbour search: embed a query question, compute its cosine against every stored question, return the top $k$. Averaging discards order and lets frequent words dominate, which is why the CNN and RNN notes replace it with models that read the sequence.

Limitations#

• One vector per type. bank has one vector whether the sentence is about rivers or money. The vector is a compromise between senses, and nothing in the representation knows which sense is meant. This is the limitation that contextual embeddings remove in the BERT note.
• Closed vocabulary. A word not in the table has no vector. Subword tokens from the tokenization note reduce the problem; character n-gram models remove it, at a cost.
• Similarity is co-occurrence. Antonyms (hot, cold) occur in the same contexts and end up close. Whether that is a bug depends on the task.
• The space reflects the corpus. Vectors trained on Twitter reproduce Twitter’s usage, including its biases and stereotypes, and analogies make those measurable (Bolukbasi et al., 2016).
• Averaging is lossy. A mean of word vectors is a bag of words in a smaller space; order and negation are still gone.

What you should remember#

• One-hot vectors are $|V|$-dimensional and pairwise orthogonal: no similarity, no transfer.
• An embedding table $E \in \mathbb R^{|V|\times d}$ maps each token to a learned row; nn.Embedding is a one-hot matrix product written as a lookup.
• Cosine similarity compares directions; nearest neighbours in a learned space are words used in similar contexts.
• Analogies and 2-D projections are qualitative probes of the geometry, not metrics.
• Static embeddings give one vector per word type regardless of context.

Notebook#

week01_word_embeddings.ipynb. Cell numbers count every cell, markdown included, from 1. Quoted numbers come from the run saved with the course; cells changed since then for current library versions have no saved output.

• Cell 58 trains a 32-dimensional Word2Vec model on the tokenized Quora questions (min_count=5, window=5); cell 59 reads out one vector and cell 60 lists the nearest neighbours of word quoted above.
• Cell 62 downloads the 100-dimensional glove-twitter-100 vectors; cells 64–66 run the three analogies, whose saved outputs are the numbers above.
• Cells 67–70 take the 1,000 most frequent words; cells 71–76 project them with PCA (the objective and its SVD solution are stated in cell 71) and draw an interactive scatter plot; cells 77–79 repeat with t-SNE (perplexity=3).
• Cells 80–85 define a phrase embedding as the mean of word vectors and plot about 1,000 questions; cells 86–91 implement nearest-neighbour search by cosine over the first 1,000 questions. The embedding and search functions were exercises; the notebook now has the solutions filled in, and these cells have no saved outputs.

Next#

The vectors used above were either downloaded or produced by a library call. The next note opens the library: it defines the prediction task that makes co-occurring words end up close, writes down the objective, and shows why the straightforward version is too expensive and how negative sampling repairs it.

References#

• T. Mikolov, K. Chen, G. Corrado, J. Dean. Efficient Estimation of Word Representations in Vector Space. ICLR Workshop, 2013. The analogy test and the linear-regularity observation.
• J. Pennington, R. Socher, C. D. Manning. GloVe: Global Vectors for Word Representation. EMNLP, 2014. The source of the Twitter vectors used above.
• T. Linzen. Issues in evaluating semantic spaces using word analogies. RepEval, 2016.
• T. Bolukbasi, K.-W. Chang, J. Zou, V. Saligrama, A. Kalai. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. NeurIPS, 2016.
• L. van der Maaten, G. Hinton. Visualizing Data using t-SNE. JMLR, 9, 2008.
• M. Wattenberg, F. Viégas, I. Johnson. How to Use t-SNE Effectively. Distill, 2016.
• D. Jurafsky, J. H. Martin. Speech and Language Processing, 3rd ed. draft. Chapter 6 (vector semantics and embeddings).