Word2Vec

The previous note found that library-trained embedding tables give similar vectors to words used in similar contexts, without explaining why. Words that occur in the same contexts tend to have similar meanings (Harris, 1954; Firth, 1957), and a model that must predict a word’s neighbours through a narrow bottleneck has to give words with the same neighbours similar bottleneck vectors. Word2Vec (Mikolov et al., 2013a, 2013b) turns that into two small networks, made trainable on billions of tokens by hierarchical softmax or negative sampling; this note develops negative sampling, the one used in practice.

Notation. The corpus is a token sequence $w_1, \dots, w_T$ over a vocabulary $V$; $d$ is the embedding dimension. For a position $t$ the centre word is $w_t$ and the context is the window $C_t = (w_{t+j})_{j \in J_t}$ with $J_t = \{j : 1 \le |j| \le m,\ 1 \le t + j \le T\}$, up to $2m$ neighbours with repeats kept; $m$ is the window radius. Each word $w$ has two vectors: an input (centre) vector $v_w \in \mathbb R^d$, a row of $W_{\text{in}} \in \mathbb R^{|V| \times d}$, and an output (context) vector $u_w \in \mathbb R^d$, a row of $W_{\text{out}} \in \mathbb R^{|V| \times d}$. $W_{\text{in}}$ is the embedding table $E$ of the previous note. $\sigma(x) = 1/(1 + e^{-x})$; $k$ is the number of negative samples and $P_n$ the distribution they are drawn from.

Baseline: fit the embeddings with a labelled task#

The previous note listed sources for the table. Training it as a layer of a classifier works (the CNN note measures it) but learns only about words in the labelled set and only what helps predict the label. Labelled text is scarce; unlabelled text is almost unlimited. What should a model predict when there is no label?

A network whose first layer is the table#

The key idea needs only a two-layer linear network. For an input word $c$ and a target word $o$:

1. take the one-hot vector $e_c \in \{0, 1\}^{|V|}$ of $c$;
2. multiply by the first layer: $h = W_{\text{in}}^\top e_c = v_c \in \mathbb R^d$, the row of $c$ (no bias, no non-linearity);
3. score every vocabulary word against $h$: $s = W_{\text{out}}\, h \in \mathbb R^{|V|}$, so $s_w = u_w \cdot v_c$;
4. turn the scores into a distribution with a softmax:

$$p(o \mid c) = \frac{\exp(u_o \cdot v_c)}{\sum_{w \in V}\exp(u_w \cdot v_c)}.$$

The network is fitted with the cross-entropy $-\log p(o \mid c)$ on (input, target) pairs cut from the corpus; then all but the first layer is thrown away, and the row $v_c$ is the embedding. In code the one-hot product is a lookup: a batch of ids $[B]$ becomes $[B, d]$, then $[B, |V|]$. The model has $2|V|d$ parameters.

Figure

The Word2Vec network: a row lookup, a dot product with every output vector, a softmax

Two tables are needed because with one, a word’s score against itself, $\|v_w\|^2$, would be large, while a word rarely occurs in its own window; two tables let $u_w \cdot v_w$ be small (Goldberg and Levy, 2014). Libraries return $W_{\text{in}}$ (wv in gensim).

Two prediction tasks: CBOW and skip-gram#

The network leaves open which words are inputs and which are targets. Word2Vec defines two choices on the same windows.

Continuous bag of words (CBOW) predicts the centre word from its context: the mean of the context input vectors is classified among the $|V|$ possible centre words:

$$h_t = \frac{1}{|J_t|}\sum_{j \in J_t} v_{w_{t+j}}, \qquad p(w_t \mid C_t) = \frac{\exp(u_{w_t} \cdot h_t)}{\sum_{w' \in V}\exp(u_{w'} \cdot h_t)}.$$

“Bag” because the mean ignores word order; each position gives one example.

Skip-gram reverses the direction: the centre word predicts each context word separately, with the softmax of the previous section, minimizing

$$\mathcal L_{\text{SG}} = -\frac{1}{T}\sum_{t=1}^T \sum_{j \in J_t} \log p(w_{t+j} \mid w_t).$$

Each position gives up to $2m$ (centre, context) pairs.

Figure

One window, two prediction tasks

Worked example: training pairs. In we drink hot coffee every morning ($T = 6$, $m = 2$) the centre coffee gives four skip-gram pairs and one CBOW example, in which the mean of the four context vectors predicts coffee (figure above). Position $i$ has $\min(m, i - 1) + \min(m, T - i)$ neighbours, so the sentence gives $2 + 3 + 4 + 4 + 3 + 2 = 18$ skip-gram pairs against 6 CBOW examples.

A neural language model (Bengio et al., 2003) also starts with an embedding table but predicts the next word, like the n-gram model, through a costly non-linear hidden layer. Word2Vec only wants the table, so it drops the non-linearity and uses context on both sides.

CBOW computes one output distribution per position and skip-gram up to $2m$, so CBOW is cheaper by about the window size, but in the comparison of Mikolov et al. (2013a), §3.1–3.2 and Table 3 (640-dimensional vectors), skip-gram was far better on semantic analogies (55% against 24%) and slightly worse on syntactic ones (59% against 64%). It is also reported to be better for rare words (word2vec project page), whose vectors CBOW only updates through a mean with up to $2m - 1$ others.

For each centre the effective radius is drawn uniformly from $\{1, \dots, m\}$ (Mikolov et al., 2013a, §3.2), so a neighbour at distance $j$ is used with probability $(m - j + 1)/m$ ($1, 0.8, 0.6, 0.4, 0.2$ for $m = 5$): near neighbours count more without an explicit weight.

Why the softmax is the problem#

Write the skip-gram loss of one pair $(c, o)$ as $-\log p(o \mid c) = -u_o \cdot v_c + \log \sum_{w \in V} \exp(u_w \cdot v_c)$. The derivative of the log-sum-exp with respect to $v_c$ is the softmax-weighted average of the $u_w$, and with respect to one $u_w$ it is $p(w \mid c)\, v_c$:

$$\begin{aligned} \frac{\partial\,(-\log p(o \mid c))}{\partial v_c} &= -u_o + \sum_{w \in V} p(w \mid c)\, u_w, \\ \frac{\partial\,(-\log p(o \mid c))}{\partial u_w} &= \big(p(w \mid c) - \mathbb 1[w = o]\big)\, v_c . \end{aligned}$$

The first line reads “observed minus expected”: a step pulls $v_c$ towards the context word that occurred and away from the one the model expects. The second is non-zero for every $w$, so one pair writes to all $|V|$ rows of $W_{\text{out}}$, and the normalizer alone needs $|V|$ dot products (Rong, 2014).

With $|V| = 200{,}000$ and $d = 300$ the model has $2|V|d = 1.2 \times 10^8$ parameters. One pair costs $|V| d = 6 \times 10^7$ multiply-adds for the scores and updates $|V| d + d = 60{,}000{,}300$ parameters, and a corpus of $10^9$ tokens with $m = 5$ gives up to $10^{10}$ pairs per pass. The model is small; the normalization makes training infeasible.

Figure

Parameters touched by one training pair as the vocabulary grows

Hierarchical softmax (Morin and Bengio, 2005) puts the vocabulary at the leaves of a binary (Huffman, Mikolov et al., 2013b, §2.1) tree, so a prediction takes about $\log_2 |V|$ binary decisions, 18 for 200,000 words. Negative sampling drops the normalization; it is the default in practice and the subject of the rest of this note.

Negative sampling#

Figure

Softmax versus negative sampling on the output side of skip-gram

For an observed pair $(c, o)$ and noise words $n_1, \dots, n_k \sim P_n$, the skip-gram negative-sampling (SGNS) loss is

$$\mathcal L_{\text{NS}}(c, o) = -\log \sigma(u_o \cdot v_c) - \sum_{i=1}^k \log \sigma(-u_{n_i} \cdot v_c).$$

$\sigma(u_w \cdot v_c)$ is the model’s probability that a pair is real and $\sigma(-x) = 1 - \sigma(x)$, so each term of the sum is the log-probability that a noise pair is recognized as fake. With the scores $s_o = u_o \cdot v_c$ and $s_i = u_{n_i} \cdot v_c$, and $\frac{d}{dx}\log\sigma(x) = 1 - \sigma(x)$, the gradients are

$$\begin{aligned} \frac{\partial \mathcal L_{\text{NS}}}{\partial v_c} &= \big(\sigma(s_o) - 1\big)\, u_o + \sum_{i=1}^k \sigma(s_i)\, u_{n_i}, \\ \frac{\partial \mathcal L_{\text{NS}}}{\partial u_o} &= \big(\sigma(s_o) - 1\big)\, v_c, \qquad \frac{\partial \mathcal L_{\text{NS}}}{\partial u_{n_i}} = \sigma(s_i)\, v_c . \end{aligned}$$

This is “observed minus expected” with the expectation over $|V|$ words replaced by $k$ sampled words, each weighted by how plausible the model finds it; all other rows have zero gradient.

CBOW uses the same loss with the context mean $h_t$ in place of the centre vector and the centre word as the positive:

$$\mathcal L_{\text{CBOW-NS}}(t) = -\log \sigma(u_{w_t} \cdot h_t) - \sum_{i=1}^k \log \sigma(-u_{n_i} \cdot h_t).$$

Its gradients are those above with $h_t$ for $v_c$. The chain rule through the mean would send $1/|J_t|$ of $h_t$‘s gradient to each context vector; word2vec.c (word2vec project page) and gensim (cbow_mean=1) add all of it, in effect a larger learning rate for the input table.

A pair updates $(k + 2) d$ parameters, $2{,}100$ for $d = 300$ and $k = 5$. Mikolov et al. (2013b), §2.2, report that $k$ between 5 and 20 works for small training sets and 2 to 5 for large ones; gensim uses $k = 5$.

Worked example#

Toy numbers: $d = 2$, $k = 2$, learning rate $\eta = 0.5$ (large, so that the movement is visible), centre coffee, observed context drink, negatives car and tax.

Vector	Value	Score with $v_{\text{coffee}}$	$\sigma(\text{score})$	Loss term
$v_{\text{coffee}}$	$(1.0, 0.5)$
$u_{\text{drink}}$ (observed)	$(0.8, 0.6)$	$1.10$	$0.750$	$-\log \sigma(1.10) = 0.287$
$u_{\text{car}}$ (negative)	$(-0.2, 0.9)$	$0.25$	$0.562$	$-\log \sigma(-0.25) = 0.826$
$u_{\text{tax}}$ (negative)	$(0.6, -0.4)$	$0.40$	$0.599$	$-\log \sigma(-0.40) = 0.913$

The loss is $0.287 + 0.826 + 0.913 = 2.026$, and the gradients are

• $\partial \mathcal L / \partial v_{\text{coffee}} = -0.250\,(0.8, 0.6) + 0.562\,(-0.2, 0.9) + 0.599\,(0.6, -0.4) = (0.047, 0.117)$;
• $\partial \mathcal L / \partial u_{\text{drink}} = -0.250\,(1.0, 0.5) = (-0.250, -0.125)$;
• $\partial \mathcal L / \partial u_{\text{car}} = 0.562\,(1.0, 0.5) = (0.562, 0.281)$ and $\partial \mathcal L / \partial u_{\text{tax}} = 0.599\,(1.0, 0.5) = (0.599, 0.299)$.

One step $x \leftarrow x - \eta\, \partial\mathcal L/\partial x$ gives $v_{\text{coffee}} = (0.977, 0.442)$, $u_{\text{drink}} = (0.925, 0.662)$, $u_{\text{car}} = (-0.481, 0.759)$ and $u_{\text{tax}} = (0.301, -0.550)$; the two figures below show the vectors and the scores moving, and the loss falls from $2.026$ to $1.611$. Four rows changed; with 200,000 words the other 199,997 output rows were not read.

Figure

One negative-sampling step, drawn in the plane

Figure

The two loss terms of negative sampling as functions of the score

For contrast, the full softmax over a toy six-word vocabulary with the same $v_{\text{coffee}}$: drink $(0.8, 0.6)$, hot $(0.7, 0.1)$, morning $(0.3, 0.5)$, car $(-0.2, 0.9)$, tax $(0.6, -0.4)$ and cup $(0.9, 0.2)$. The scores are $1.10, 0.75, 0.55, 0.25, 0.40, 1.00$, the probabilities $0.243, 0.171, 0.140, 0.104, 0.121, 0.220$, and the loss $-\log 0.243 = 1.414$. The gradient on $v_{\text{coffee}}$, $\sum_w p(w \mid c)\, u_w - u_{\text{drink}} = (-0.193, -0.277)$, needs all six rows and pushes down cup, a plausible context ranked second; negative sampling pushes a word down only when it is drawn.

Which words are negatives#

Negatives are drawn from the unigram distribution raised to the power $3/4$ and renormalized (Mikolov et al., 2013b, §2.2):

$$P_n(w) = \frac{\operatorname{count}(w)^{3/4}}{\sum_{w' \in V} \operatorname{count}(w')^{3/4}} .$$

With toy counts the 600, coffee 30 and espresso 3, sampling in proportion to the counts gives probabilities $0.948, 0.047, 0.005$, and the $3/4$ power gives $0.889, 0.094, 0.017$: espresso becomes 3.5 times more likely to be drawn.

Figure

Raising the unigram distribution to the power 3/4

Subsampling frequent words#

Pairs with the or a are numerous and uninformative. Before windows are formed, each occurrence of a word $w$ with relative frequency $f(w)$ is discarded with probability

$$P_{\text{discard}}(w) = 1 - \sqrt{\theta / f(w)}$$

for a threshold $\theta \approx 10^{-5}$, and never when $f(w) \le \theta$ (Mikolov et al., 2013b, §2.3). For toy frequencies $0.05$, $10^{-3}$ and $10^{-4}$ the discard probabilities are $0.986$, $0.9$ and $0.68$. Removing a token brings the words around it closer, so subsampling also widens the effective window. gensim uses a variant of this rule with threshold sample, default $10^{-3}$.

What negative sampling optimizes#

Negative sampling optimizes a different objective from the softmax, a simplification of noise-contrastive estimation (Gutmann and Hyvärinen, 2010; Mnih and Teh, 2012), which classifies with the corrected logit $u_o \cdot v_c - \log\big(k\, P_n(o)\big)$ and whose optimum is $\log P(o \mid c)$. Negative sampling uses the raw score (Mikolov et al., 2013b, §2.2), so its optimum is shifted: if $d$ is large enough for every score to be set independently, it is the pointwise mutual information shifted by $\log k$ (Levy and Goldberg, 2014, §3.1):

$$u_o \cdot v_c = \operatorname{PMI}(c, o) - \log k, \qquad \operatorname{PMI}(c, o) = \log \frac{P(c, o)}{P(c)\, P_n(o)} .$$

With the $3/4$ power this is a smoothed PMI: rare context words get a larger $P_n(o)$ and a smaller value (Levy, Goldberg and Dagan, 2015, “context distribution smoothing”).

Figure

The per-pair negative-sampling objective and its minimum

So SGNS factorizes the shifted PMI matrix into $W_{\text{in}} W_{\text{out}}^\top$ of rank $d$, and this compression puts words with similar PMI rows close together. Pairs that never co-occur have $\operatorname{PMI} = -\infty$ (dotted curve), so there it stays approximate for any $d$.

What a small model learns from Quora questions#

We train a model with $d = 32$, window radius 5 and minimum count 5 on about half a million Quora questions, tokenized with stopwords removed but punctuation such as ? kept. Everything else is a gensim default (table below), so this is CBOW with negative sampling, $k = 5$, and subsampling.

The vector of word has 32 coordinates between $-4.09$ and $4.18$ (norm 10.7), and its nearest cosine neighbours, listed in the previous note, are sentence (0.836), words (0.796) and verb (0.787). Singular and plural forms sit together (word/words, phrase/phrases) because the text was not lemmatized. The neighbours are topical as much as grammatical because, without stopwords and with ? subsampled, a radius-5 window covers more of a question than five raw tokens.

A second CBOW model, 100-dimensional with minimum count 4 and window radius 5, is trained on the AG News training articles only, so that no test words leak in; it supplies the input vectors of the next note.

Hyperparameters and what they do#

Parameter	Effect	Default in gensim	Quora and AG News models
vector_size $d$	Capacity; 50–300 is the usual range	100	32 (Quora), 100 (AG News)
window $m$	Radius; larger windows give more topical neighbours, smaller ones more syntactic ones (Jurafsky and Martin, §5.7)	5	5
min_count	Drops rarer words; sets $\lvert V \rvert$	5	5 (Quora), 4 (AG News)
sg	0 = CBOW, 1 = skip-gram	0	default
negative $k$	Negatives per pair; 0 with hs=1 gives hierarchical softmax	5	default
ns_exponent	Power of the counts in $P_n$	0.75	default
sample	Subsampling threshold for frequent words	$10^{-3}$	default
epochs, alpha	Passes over the corpus; initial learning rate	5, 0.025	default

With tuned window, subsampling, negatives and noise exponent, count-based methods and SGNS perform much more alike than the original comparisons suggested (Levy, Goldberg and Dagan, 2015).

What changed#

	Task-trained embeddings	Full-softmax Word2Vec	Word2Vec with negative sampling
Training signal	Labels of one task	The text itself	The text itself
Data available	Thousands of labelled examples	Billions of tokens	Billions of tokens
Vocabulary coverage	Words in the labelled set	Words in the corpus	Words in the corpus
Output per step	$K$ label classes	$\lvert V \rvert$-way softmax	$k + 1$ binary classifiers
Parameters touched per pair ($\lvert V \rvert = 200{,}000$, $d = 300$)	—	60,000,300	2,100 ($k = 5$)
What the vectors encode	Whatever separates the labels	Co-occurrence structure	Shifted PMI, compressed to rank $d$

The result is a table that any model can reuse.

GloVe and FastText#

Two later static embeddings have notes of their own. GloVe (Pennington, Socher and Manning, 2014), which gave the Twitter vectors of the previous note, fits $u_w \cdot v_c + b_w + b_c \approx \log X_{wc}$ to the co-occurrence matrix $X$ by weighted least squares; like SGNS it fits a function of co-occurrence counts with dot products, so the two end up close (GloVe note). FastText (Bojanowski et al., 2017) trains SGNS but represents a word as the sum of its character n-gram vectors (where → <wh, whe, her, ere, re>), so related words share parameters and an unseen word still gets a vector (FastText note).

Limitations#

• Static. One vector per word type: bank is a single compromise between the river and the money. The BERT note removes this.
• Closed vocabulary. Words below min_count or absent from the corpus get no vector; FastText and subword tokenization address this.
• Window-bound. Only neighbours within $m$ positions shape a vector, and CBOW ignores their order; negation and long-range dependencies do not enter.
• Similarity means substitutability. good and bad, hot and cold, Monday and Tuesday share contexts and end up close, the wrong geometry for sentiment.
• Data-hungry. Half a million questions suffice for frequent words such as word, not for rare ones; the analogies of the previous note use billions of tokens.
• Sensitive, not reproducible. Preprocessing, window, subsampling and $k$ change the neighbours, and multithreaded runs with one seed differ.

What you should remember#

• Skip-gram predicts the context from the centre, CBOW the centre from the context. The network is linear; its first layer $W_{\text{in}}$ is the embedding table, the output layer only a training signal.
• The full softmax updates all $|V|$ output rows per pair; negative sampling classifies the real pair against $k$ pairs drawn from the $3/4$-power unigram distribution and updates $(k + 2) d$ parameters.
• At its optimum SGNS sets $u_o \cdot v_c = \operatorname{PMI}(c, o) - \log k$: the vectors are a low-rank compression of co-occurrence statistics.
• gensim trains CBOW with $k = 5$ negatives by default; skip-gram needs sg=1.

Notebook#

week01_word_embeddings.ipynb. Cell numbers count every cell, markdown included, from 1. Quoted numbers come from the run saved with the course; cells changed since then for current library versions have no saved output.

• Data (cells 4–31). Cell 4 downloads the Quora file only if it is missing (the repository already has it); cell 6 reads data/quora.txt from disk. Cells 13–22 build data_tok: wordpunct_tokenize, lower-casing (an exercise in cell 15, with the solution filled in), NLTK stopword removal, punctuation kept. The lemmatized and stemmed variants of the 537,272 questions (cells 28–31) are not used by Word2Vec.
• Training (cells 53–58). Markdown introduces skip-gram and gensim (Alammar, 2019); cell 58 trains Word2Vec(data_tok, vector_size=32, min_count=5, window=5), which with the defaults is CBOW with 5 negatives and the summed context update (cbow_mean=1) described above, not skip-gram.
• Neighbours (cells 59–60). The vector of word and its nearest neighbours, the only saved outputs of this model: no loss curve, training time, vocabulary size or evaluation score was saved. The norm 10.7 quoted above is computed from the printed coordinates.
• Analogies (cells 61–66). Pretrained 100-dimensional GloVe Twitter vectors and the analogies reported in the previous note.

week02_NN_text_classification.ipynb, cells 13–15, trains a 100-dimensional CBOW model (min_count=4, window=5) on the AG News training texts only, so no test words leak in, and turns each article into a sequence of vectors for the next note.

Next#

Word2Vec learns from one window at a time and never looks at the corpus as a whole. The next note counts every pair of words once into a co-occurrence matrix and fits vectors to the logarithms of those counts directly, and shows that the two approaches end up factorizing nearly the same matrix.

References#

• Z. S. Harris. Distributional Structure. Word, 10(2–3), 146–162, 1954. DOI 10.1080/00437956.1954.11659520. The distributional hypothesis.
• J. R. Firth. A synopsis of linguistic theory 1930–1955. In Studies in Linguistic Analysis, Blackwell, 1957.
• Y. Bengio, R. Ducharme, P. Vincent, C. Jauvin. A Neural Probabilistic Language Model. JMLR, 3, 2003. Next-word prediction with an embedding layer.
• F. Morin, Y. Bengio. Hierarchical Probabilistic Neural Network Language Model. AISTATS, 2005. Hierarchical softmax.
• M. Gutmann, A. Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. AISTATS, 2010.
• A. Mnih, Y. W. Teh. A Fast and Simple Algorithm for Training Neural Probabilistic Language Models. ICML, 2012. Noise-contrastive estimation for language models.
• T. Mikolov, K. Chen, G. Corrado, J. Dean. Efficient Estimation of Word Representations in Vector Space. ICLR Workshop, 2013. CBOW and skip-gram, their training complexity (§3.1–3.2) and comparison (Table 3), the random window radius (§3.2).
• T. Mikolov, I. Sutskever, K. Chen, G. Corrado, J. Dean. Distributed Representations of Words and Phrases and their Compositionality. NeurIPS, 2013. Hierarchical softmax with a Huffman tree (§2.1), negative sampling, the range of $k$ and the $3/4$ power (§2.2), subsampling (§2.3).
• Y. Goldberg, O. Levy. word2vec Explained: deriving Mikolov et al.’s negative-sampling word-embedding method. arXiv, 2014. The negative-sampling objective and the reason for two tables.
• O. Levy, Y. Goldberg. Neural Word Embedding as Implicit Matrix Factorization. NeurIPS, 2014. SGNS as shifted-PMI factorization (§3.1).
• X. Rong. word2vec Parameter Learning Explained. arXiv, 2014. Softmax gradients for CBOW and skip-gram.
• J. Pennington, R. Socher, C. D. Manning. GloVe: Global Vectors for Word Representation. EMNLP, 2014.
• O. Levy, Y. Goldberg, I. Dagan. Improving Distributional Similarity with Lessons Learned from Word Embeddings. TACL, 3, 2015. Hyperparameters matter as much as the algorithm; context distribution smoothing (the $3/4$ power as smoothed PMI).
• P. Bojanowski, E. Grave, A. Joulin, T. Mikolov. Enriching Word Vectors with Subword Information. TACL, 5, 2017. FastText.
• D. Jurafsky, J. H. Martin. Speech and Language Processing, 3rd edition draft of August 19, 2026, chapter 5 “Embeddings”, §5.5 “Word2vec” and §5.7 “Semantic properties of embeddings”. Textbook treatment of skip-gram with negative sampling; the effect of the window size.
• T. Mikolov et al. word2vec. Project page and reference C implementation (word2vec.c), Google Code Archive, 2013. Practical comparison of skip-gram and CBOW (“better for infrequent words”); the CBOW update that adds the whole hidden-layer gradient to every context vector.
• R. Řehůřek, P. Sojka. Software Framework for Topic Modelling with Large Corpora. LREC Workshop on New Challenges for NLP Frameworks, 2010. The gensim library; defaults from its Word2Vec API documentation.
• J. Starmer (StatQuest). Word Embedding and Word2Vec, Clearly Explained!!!. Video, 2023. The explanation of the cost of one update and how negative sampling reduces it follows this video.
• J. Alammar. The Illustrated Word2vec. Blog post, 2019. An illustrated walk through skip-gram with negative sampling.