What natural language processing is

A spam filter, a search box, a translator and a chat assistant all read text and act on it. Natural language processing (NLP) is the set of methods that let a program do that: turn text into a decision, or into other text. This note says what the problem is, why it is hard, why a course on the methods is worth taking when a chat model is one API call away, and how the course is built.

The problem: from text to decisions and back#

A text is a sequence of symbols, $x = (w_1, \dots, w_T)$, each $w_t$ taken from a vocabulary $V$. What a symbol is (a word, a piece of a word, a byte) is a choice the next note and the tokenization note make; here it only matters that text arrives as a sequence of discrete tokens of varying length.

Every task in the course has one of four shapes:

1. Text to a label, $p(y \mid x)$: one answer for the whole text. Spam or not, the sentiment of a review, the topic of a news item. The running example of the course is AG News, 120,000 news texts in four topics.
2. Text to a label per token, $p(y_1, \dots, y_T \mid x)$: an answer for every position. Named entities, parts of speech, the start and end of an answer span in a passage.
3. Text to text, $p(y_1, \dots, y_m \mid x)$: a new sequence conditioned on the input. Translation, summarization, rewriting.
4. Text continued, $p(w_1, \dots, w_T)$: no separate input, only the probability of the text itself, generated one token after another. This is a language model.

Figure

The four shapes of NLP tasks

The fourth shape is the most general. A label, a tag sequence or a translation can all be written as text that continues a prompt (Topic of this article: ), so a model that continues text well can in principle do the other three. That observation is why one pretrained language model now serves many tasks, and the GPT note and the model-families note follow it through. It is not a reason to do everything that way: a fine-tuned classifier is cheaper per document, easier to evaluate and often more accurate, as the text classification note shows.

What NLP is for#

• Search. Find the passages that answer a query, by matching words or meanings: the semantic search note.
• Classification at scale. Route support tickets, filter spam and abuse, tag news by topic, measure sentiment: millions of documents, one cheap decision each.
• Extraction. Pull names, dates, amounts and their relations out of free text into a table, or remove them, as in the de-identification of medical records: the NER and QA notes.
• Generation from a source. Translate, summarize, answer from a document: the translation and summarization notes.
• Assistants and agents. A language model that follows instructions, looks things up and calls tools: the instruction tuning, RAG and agents notes.

Most recorded human knowledge is text, and most of the work above is reading or writing it at a scale no person could.

One objective under most of it#

The chain rule of probability turns the language model into a sequence of classification problems, one per position:

$$p(w_1, \dots, w_T) = \prod_{t=1}^{T} p(w_t \mid w_1, \dots, w_{t-1}).$$

Training maximizes the log-probability of real text, which is the same as minimizing the average cross-entropy per token,

$$H = -\frac{1}{T} \sum_{t=1}^{T} \log p(w_t \mid w_{<t}), \qquad \mathrm{PPL} = e^{H}.$$

Perplexity $\mathrm{PPL}$ reads as an effective number of choices: a model that spreads its probability uniformly over 10,000 words has perplexity 10,000, and one that gives the observed next token probability 0.1 at every step has perplexity 10. The n-gram note derives it and measures it.

The methods of the course differ in how they compute $p(w_t \mid w_{<t})$. An n-gram model counts how often $w_t$ followed the last $n - 1$ tokens. A recurrent network compresses all of $w_{<t}$ into one state vector. A Transformer looks at every earlier token directly through attention. GPT is a Transformer trained this way on a very large corpus. A classifier uses the same cross-entropy with $K$ labels in place of $|V|$ tokens.

Why language is hard for a computer#

Text looks simple: a short string of familiar words. Five properties make it hard.

• Symbols have no geometry. As one-hot vectors, cat and dog are exactly as far apart as cat and carburetor. Nothing in the raw symbols says which words are similar; that has to be learned, which is what word embeddings do.
• The vocabulary is open and heavy-tailed. A few words make up most of any text, and most distinct words are rare. A model of whole words wastes parameters on words it has barely seen and has no entry at all for new ones, which is what subword tokenization fixes.
• Meaning depends on context. bank in river bank and bank account is one string with two meanings; I saw her duck has two readings. A fixed vector per word cannot tell them apart, which motivates the contextual vectors of BERT.
• Order matters, sometimes over long distances. not good and good, not use the same words. In The keys to the old cabinet are on the table, the verb agrees with keys, five words back. Bags of words ignore order, n-grams see only a few tokens, and recurrent networks, then attention, are the answers.
• Many outputs are right. A sentence has many correct translations and an article many good summaries, so scoring a system is a problem of its own: BLEU, ROUGE and judges in the evaluation and LLM evaluation notes.

Worked example: the vocabulary of AG News#

Lowercase the 120,000 training texts of AG News and split them into runs of letters and digits. The result has 4,759,159 tokens and 65,043 distinct words.

Most frequent words	Share of all tokens
10	19.3 %
100	38.1 %
1,000	66.6 %
10,000	93.9 %

At the other end, 20,925 words, a third of the vocabulary, occur exactly once. Frequency falls roughly as the inverse of rank, Zipf’s law (Zipf, 1949): the 10th, 100th and 1,000th words occur 44,506, 4,714 and 665 times, so rank times frequency stays between 0.45 and 0.67 million. A vocabulary of 10,000 words covers 94 % of the tokens; the remaining 55,000 words share the last 6 %, and each new text brings words that none of them cover.

The list also shows what raw text is like. The tenth most frequent “word” is 39. The source stored the apostrophe of Web's as the HTML entity ', the ampersand was lost, and #39; remains in 29,696 of the 120,000 texts; 44,316 of the 44,506 occurrences of 39 come from it. Nobody would guess this from a sample of the data, and every model trained on it learns it.

Why learn the methods when a chat model exists#

A capable language model answers questions about text out of the box. The methods of this course are still worth knowing, for four reasons.

1. An LLM system is built from these parts. The tokenizer decides what a request costs and which words the model handles badly. Embeddings decide what retrieval finds. Attention and positional encoding set how much context fits. The decoding strategy changes the answer for the same model. Quantization and parameter-efficient fine-tuning decide what runs on a given GPU. Each of them is a note in this course.
2. Simple baselines are strong. On AG News, TF-IDF with logistic regression reaches 0.918 accuracy on the test set with fixed settings. Three other results on the same data were measured under different protocols, so they are separate demonstrations, not a ranking. A small Transformer trained from scratch reaches 0.906 at an epoch picked on the test set, which makes it optimistic. BERT used zero-shot through its masked-language-model head reaches 0.456 with the natural label words and 0.613 with label words chosen on the test articles, optimistic again. BERT fine-tuned on the task reaches 0.949 on a held-out tenth of the training set, not on the test set. Even so the pattern is instructive: counting words gets most of the way, a large model used without training can be far behind, and the last points plausibly need a pretrained model trained on the task. The results of the classification note list each protocol. Knowing this is what lets you choose.
3. Cost and control. A fine-tuned encoder classifies a document in one forward pass with a known accuracy; a prompted model generates text that has to be parsed and checked. For a narrow task at scale the small model often wins on cost, latency and the ability to measure it.
4. Failures have mechanisms. Unseen words make a language model’s perplexity explode (an unsmoothed unigram model trained on a small corpus of jokes has a held-out perplexity on the order of $10^{46}$ in the n-gram note); a hand-written label map silently swaps two entity types; a small model judge’s agreement with people, 0.19 on 28 items, cannot be told apart from zero (the LLM evaluation note). Each is easy to diagnose once the mechanism is known and hard to notice otherwise.

The plan of the course#

The course follows one main line: most methods are introduced by a concrete failure of the one before it. A few steps are branches rather than repairs: a different way to do the same job, not a fix for the previous one.

Figure

The line of the course: most methods answer a failure of the previous one

1. Text foundations. What a string is and how it is cleaned, subword tokenization, corpora and evaluation.
2. Classical NLP. Bag of words and TF-IDF; n-gram language models, perplexity and smoothing.
3. Distributed representations. Dense word vectors, Word2Vec, GloVe and FastText.
4. Neural sequence models. CNNs for text, recurrent networks, LSTM and GRU, sequence-to-sequence models and the attention that removed their bottleneck.
5. Transformers. Self-attention, the Transformer block, positional representations, a translation model.
6. Pretrained language models. BERT, GPT and decoding, the model families, transfer learning.
7. Adaptation and efficient models. Fine-tuning, parameter-efficient fine-tuning (LoRA, prompt tuning), instruction tuning and alignment, quantization.
8. NLP tasks. Text classification, sentence pairs and similarity, named entity recognition, question answering, summarization.
9. Modern LLM systems. Embeddings and semantic search, retrieval-augmented generation, tools and agents, evaluation of LLMs.
10. Advanced generation. Generating text without a left-to-right order.

Each of the notes that follow has a worked example with numbers that can be checked by hand, and ends with a Notebook section, the practice part, which describes the course notebook where the method runs.

What you should remember#

• NLP turns text into a label, a label per token, other text, or a continuation; the four shapes cover the tasks of the course.
• A language model factorizes the probability of a text into next-token probabilities; training minimizes cross-entropy, and perplexity $e^{H}$ is its readable form.
• Language is hard because symbols have no geometry, the vocabulary is open and Zipf-distributed, meaning depends on context, order matters over long distances, and many outputs are right.
• On AG News, counting words reaches 0.918 on the test set and a fine-tuned BERT 0.949 on a held-out part of the training set; the protocols differ, so this is not a direct ranking. Measure a strong baseline first, under the same protocol as the model meant to beat it.
• The course goes from counts to LLM systems, most methods answering a failure of the previous one; text diffusion at the end is a branch, another order of generation.

Next#

Before any model, text has to become units a program can count. That is Text as data: bytes, Unicode, segmentation and cleaning.

References#

• D. Jurafsky, J. H. Martin. Speech and Language Processing, 3rd edition draft. 2025. Chapters 2–3: tokens and n-gram language models.
• J. Eisenstein. Introduction to Natural Language Processing. MIT Press, 2019.
• Y. Goldberg. Neural Network Methods for Natural Language Processing. Morgan & Claypool, 2017.
• C. D. Manning, H. Schütze. Foundations of Statistical Natural Language Processing. MIT Press, 1999. Ch. 1.4: Zipf’s law and word frequencies.
• G. K. Zipf. Human Behavior and the Principle of Least Effort. Addison-Wesley, 1949.
• X. Zhang, J. Zhao, Y. LeCun. Character-level convolutional networks for text classification. NeurIPS, 2015. The AG News benchmark.
• R. Bommasani et al. On the opportunities and risks of foundation models. arXiv:2108.07258, 2021. One pretrained model adapted to many tasks.