CNNs for text classification

The TF-IDF classifier of the bag-of-words note scores 0.92 on AG News while ignoring word order and treating every token as unrelated to every other. We now have embeddings in which related words are close. The question for this note is whether a network that reads the sequence of embeddings, and can therefore see short phrases, does better. The answer on this dataset is no, and the reasons are more instructive than a win would have been.

Notation. A document is a sequence of $T$ tokens with embeddings $x_1, \dots, x_T \in \mathbb R^{d}$, stacked into $X \in \mathbb R^{T \times d}$. A batch is a tensor of shape $[B, T, d]$ with $T$ the longest sequence in the batch. $K$ is the number of classes, 4 here.

From a sequence of vectors to one vector#

A classifier needs one vector per document. The embeddings note averaged the token vectors, which throws order away again. The alternative that this note develops keeps local order: move a small window along the sequence, compute a feature for every window position, and keep the strongest response of each feature over the document.

Batching variable-length sequences#

Before any model, the data has to be rectangular. Articles have different lengths; a batch is one tensor; so shorter sequences are padded. With pretrained vectors as input the padding is a zero vector, and the batch has shape $[B, T_{\max}, d]$ with $d = 100$. After preprocessing, half of the AG News training articles have between 22 and 30 tokens with a vector and 95 % have at most 39, so $T_{\max}$ in a shuffled batch of 32 is typically about 50 and the shorter articles are padded by 20 or more zero vectors.

One-dimensional convolution over time#

A convolution with kernel size $k$ and $d$ input channels holds a weight tensor $w \in \mathbb R^{k \times d}$ per output filter. At position $t$ it computes

$$y_t = \mathrm{ReLU}\Big(\sum_{j=0}^{k-1}\sum_{c=1}^{d} w_{j,c}\, x_{t+j,\,c} + b\Big), \qquad t = 1, \dots, T - k + 1.$$

One filter is a dot product between a $k \times d$ pattern and each window of $k$ consecutive token vectors; it responds when the window looks like the pattern. With $d_{\text{out}}$ filters the output is a $[B, d_{\text{out}}, T - k + 1]$ tensor (PyTorch puts channels second, so the input is permuted to $[B, d, T]$ first). The same filter is applied at every position, so the parameter count does not depend on $T$: a filter has $k \cdot d + 1$ parameters, and a layer with 100 filters of size 3 over 100-dimensional inputs has $100 \times (3 \times 100 + 1) = 30{,}100$.

Because the input vectors are word embeddings, a size-3 filter is a learned detector of a three-word pattern in embedding space, and because similar words have similar vectors, a pattern learned on stock market falls also fires, more weakly, on share prices drop. That is the transfer that the n-gram note lacked and the reason for using embeddings at all.

Max-pooling over time#

After the convolution each filter has produced a response at every position. Global max-pooling keeps, for each filter, the largest response anywhere in the document:

$$z_i = \max_{t}\, y_{i,t}, \qquad z \in \mathbb R^{d_{\text{out}}}.$$

The result is a fixed-size vector regardless of $T$: “did this pattern occur anywhere, and how strongly”. A linear layer maps $z$ to $K$ logits and the loss is the cross-entropy of the TF-IDF note. Max-pooling is also where the order information goes: the pooled vector records that a pattern fired, not where, so two documents with the same set of three-word patterns in a different arrangement are indistinguishable after pooling. This is why convolutional text classifiers are reported to work best on short texts, where “which phrases occur” is nearly the whole story.

Two architectures#

Stacked CNN. Two convolutional layers of 100 filters with $k = 3$, ReLU after each, global max-pool, linear to 4 classes. The second layer sees windows of first-layer features, so its receptive field is 5 tokens. Parameters: $2 \times 30{,}100 + (100 \times 4 + 4) = 60{,}604$.

TextCNN (Kim, 2014). Three parallel convolutions with $k = 3, 4, 5$ and 100 filters each, applied to the same input; each is max-pooled; the three 100-dimensional vectors are concatenated into 300 features; dropout (0.1) and a linear layer to 4 classes. Different kernel sizes detect phrases of different lengths in one layer. Parameters: $30{,}100 + 40{,}100 + 50{,}100 + (300 \times 4 + 4) = 121{,}504$.

Figure

TextCNN: phrase detectors and max-pooling

Both are tiny next to the embedding table they read (2.7 M parameters: 27,451 words at $d = 100$), which is frozen in the first two experiments.

Results on AG News#

Training: Adam with learning rate $10^{-4}$ and weight decay $10^{-3}$, batch size 32, 5 epochs; input vectors from a Word2Vec model trained on the preprocessed training articles only.

Model	Input	Epochs	Test accuracy	Test loss
TF-IDF + logistic regression	counts	—	0.918	—
Stacked CNN	frozen Word2Vec, $d = 100$	5	0.899	0.298
TextCNN	frozen Word2Vec, $d = 100$	5	0.901	0.288
TextCNN	trainable embeddings, $d = 100$	10	0.911	0.269

Three observations.

The CNNs lose to the count baseline. Topic in a news headline is carried by which words appear, and TF-IDF already has every word as a feature with a tuned weight. The CNN has to compress the same information through 100 filters and a max, and loses a little. The reported advantage of neural classifiers appears on tasks where phrases matter more than vocabulary, and on larger training sets.

Multiple kernel sizes help slightly. TextCNN over the stacked CNN is +0.2 points, within noise on 7,600 test examples, with twice the parameters. The mechanism (phrases of several lengths) is sound; the data does not reward it here.

A trainable table with a longer schedule scores highest. The third run changes three things at once: the embedding table receives gradients from the classification loss, it starts from random vectors instead of Word2Vec, and the learning rate and the number of epochs are doubled. Together they add a full point and more than halve the gap to the baseline, but the run does not say how much of the point comes from training the table; that would take frozen and trainable tables with the same initialization, rate and number of epochs. A plausible mechanism is that word vectors learned on the task move words that predict the same topic together, whether or not they co-occur, which is the geometry a classifier wants and not the one Word2Vec produces.

Trainable embeddings#

The input pipeline changes: documents are converted to integer ids with the Word2Vec vocabulary plus a [PAD] id, batches are padded with that id, and the model starts with nn.Embedding(|V| + 1, 100) whose output feeds the same TextCNN. The embedding table is now 2.7 M trainable parameters next to 0.12 M in the convolutions; the learning rate and the number of epochs are doubled so that the table has time to move from its random initialization. A common middle path, not run here, initializes the table from the pretrained vectors and then fine-tunes it, which keeps the coverage of pretraining and gains the task-specific geometry.

Seeing what the classifier uses#

A convolutional classifier is not interpretable from its weights, but it can be probed by perturbation: remove one token from a document, re-run the model, and record how the probability of the true class changes. A token whose removal lowers the probability was evidence for the class; one whose removal raises it was evidence against. Applied to the TextCNN with trainable embeddings on four training headlines, the probe finds wall and street the strongest evidence for Business in a story about Wall Street, and reuters the strongest against it; in a Business story about a technology company suing former employees, company pulls towards Business and technology away from it; in an Olympic swimming story, athens, olympic and swimmer pull towards Sports. The strongest evidence for Sci/Tech in a story about an EU review of a Microsoft deal is the news-agency tag ap, ahead of msft and microsoft: the probe also exposes what the classifier learned from artefacts of the dataset. The probe is cheap (one forward pass per token, batched) and local; it says what this model used for this document, not what the class means.

What changed#

	TF-IDF + linear	CNN over embeddings
Unit	Token counts	Windows of $k$ token vectors
Similar words	Unrelated features	Nearby inputs to the same filter
Order	None (n-grams as extra features)	Within a window of $k$; lost at pooling
Features	One weight per vocabulary entry	100–300 learned phrase detectors
Parameters	$K \times \lvert V \rvert$	Embeddings $\lvert V \rvert \times d$ + 0.06–0.12 M
AG News	0.918	0.899–0.911

Limitations#

• Receptive field. A filter sees $k$ tokens; stacking layers widens it slowly. Dependencies across a sentence are out of reach.
• Pooling discards position. After the max, the model knows which patterns fired, not where or in what order.
• Short texts only. The design assumes the document is a bag of phrases. Long documents need either hierarchical models or the sequence models of the next notes.
• Frozen vectors are the wrong geometry for the task. Co-occurrence similarity is not label similarity; the fix (training the table) costs labels and compute.
• No generation. The architecture produces one vector per document; it cannot predict the next token or produce text. The recurrent models of the next note can do both.

What you should remember#

• A 1-D convolution over a sequence of embeddings is a bank of learned n-gram detectors; its parameter count is independent of sequence length.
• Global max-pooling turns variable-length responses into a fixed vector and removes position.
• On AG News: TF-IDF 0.918; CNNs on frozen Word2Vec 0.899–0.901; TextCNN with trainable embeddings 0.911. A strong count baseline is hard to beat on topic classification.
• Here a trainable table (random start, doubled rate and epochs) beat frozen Word2Vec by a point; the run changes three things at once, so it does not isolate the effect of training the table. A common default is to initialize from pretrained vectors and fine-tune.
• Perturbation (drop a token, measure the change) is a cheap probe of what a text classifier relies on.

Notebook#

week02_NN_text_classification.ipynb. Cell numbers count every cell, markdown included, from 1. Quoted numbers come from the run saved with the course; cells changed since then for current library versions have no saved output.

• Cells 5–12 load AG News and preprocess every text (lowercase, \w+ tokens, stopwords removed, WordNet lemmas).
• Cell 13 trains the 100-dimensional Word2Vec model on the training texts; cells 14–15 map each text to a sequence of vectors; cell 17 plots the text-length histogram.
• Cells 19–24 build datasets and loaders with a collate_fn that pads with zero vectors; cell 24 prints a batch and its labels.
• Cells 26–31 define the stacked CNN, the training loop (nn.CrossEntropyLoss, Adam, Weights & Biases logging) and evaluation under torch.inference_mode(), and report 0.8987 test accuracy after 5 epochs.
• Cells 34–38 define TextCNN with kernel sizes 3, 4, 5 and report 0.9011.
• Cells 40–43 define and train a one-layer GRU classifier on the same inputs (the RNN note describes it) and print its test accuracy; the saved run has no output for them.
• Cells 45–57 rebuild the data as integer ids with a [PAD] index, define EmbeddingTextCNN, train for 10 epochs at learning rate $2\times10^{-4}$, and report 0.9109.
• Cells 59–66 implement the leave-one-token-out probe (explain), apply it to EmbeddingTextCNN and colour the tokens of four training headlines by their influence (blue for evidence for the class, red against); the examples above are read from those colours.

Next#

A convolution sees a window; a document is a sequence. The next note introduces a network that reads one token at a time while carrying a state, so that in principle every token can influence the representation of every later one. It can classify, and it can also predict the next token, which makes it the first neural language model of the course and brings back the perplexity measurement from the n-gram note.

References#

• Y. Kim. Convolutional Neural Networks for Sentence Classification. EMNLP, 2014. TextCNN.
• X. Zhang, J. Zhao, Y. LeCun. Character-level Convolutional Networks for Text Classification. NeurIPS, 2015. AG News and its baselines.
• Y. Zhang, B. Wallace. A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification. IJCNLP, 2017. What the kernel sizes, filter counts and pretrained vectors change.
• M. T. Ribeiro, S. Singh, C. Guestrin. “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. KDD, 2016. Perturbation-based explanations.
• The convolution and gradient background of this course is in the Generative Models note on training background.