Text as data

Every model in this course consumes numbers. Text arrives as a string, and the first job is to decide which pieces of that string become the units a model sees. The decision looks clerical, but it fixes the vocabulary, the sparsity of every later representation, and some failure modes that no amount of training removes. This note walks from raw bytes to a list of cleaned units and ends with the question that drives the next note: what to do when the units are whole words and the vocabulary explodes.

Notation. A document $d$ is one string. A token is one unit cut from it. The vocabulary $V$ is the set of distinct tokens over a corpus, and $|V|$ is its size. We write $N(t, d)$ for the number of times token $t$ occurs in $d$.

The whole note on one fragment, from the bytes of a file to the units a bag-of-words model would count:

Figure

From bytes to cleaned units

What a string is#

A text file is a sequence of bytes. An encoding maps bytes to code points, the integers that Unicode assigns to characters. UTF-8 is variable width: ASCII letters take one byte, most Cyrillic letters two, most CJK characters three, and emoji four. So “length in bytes”, “length in code points” and “length in what a reader perceives as characters” are three different numbers.

The third one differs from the second because Unicode allows the same visible glyph to be written in more than one way. The letter é is either the single code point U+00E9 or the pair U+0065 (e) followed by U+0301 (combining acute accent). Two strings that look identical can therefore compare unequal, be counted as different words, and receive different vectors. Unicode normalization forms fix a canonical choice:

Form	What it does	Typical use
NFC	Compose into precomposed code points where one exists	Storage, comparison
NFD	Decompose into base + combining marks	Stripping accents
NFKC	NFC plus compatibility folding: fi → fi, full-width Ａ → A, superscript ² → 2	Search, tokenizer training
NFKD	NFD plus compatibility folding	Accent-insensitive search

Normalization is lossy in the K forms: ² and 2 become the same token, which is right for search and wrong for a formula. This is the first instance of the rule that runs through the whole note: a cleaning step is a modelling decision, not a free simplification. A modern subword tokenizer normalizes before anything else, and the two families of the next note choose differently: SentencePiece applies NFKC by default, while BERT’s WordPiece tokenizer decomposes with NFD, deletes the combining marks (strip_accents) and lowercases, so Café and cafe become the same token.

Segmentation: cutting the string into units#

The simplest units are words. Two definitions of “word” are in common use, and they disagree on punctuation:

import re
from nltk.tokenize import wordpunct_tokenize

text = "What TV shows or books help you read people's body language?"
re.findall(r"\w+", text)
# ['What', 'TV', 'shows', 'or', 'books', 'help', 'you', 'read',
#  'people', 's', 'body', 'language']
wordpunct_tokenize(text)
# ['What', 'TV', 'shows', 'or', 'books', 'help', 'you', 'read',
#  'people', "'", 's', 'body', 'language', '?']

The regular expression \w+ keeps runs of letters, digits and underscore and discards everything else, so the question mark disappears and the possessive splits into people and s. The punctuation-aware tokenizer keeps ' and ? as their own tokens. Neither is wrong. For a topic classifier the question mark carries no signal; for a language model that has to generate questions it is the most important token in the sentence.

Word segmentation also assumes that words are separated by spaces. Chinese, Japanese and Thai are not written that way, German and Finnish glue words into long compounds, and #deeplearning is one token in a tweet and two words to a reader. Treating these cases well is one of the motivations for the subword methods in the next note.

Cleaning: trading information for a smaller vocabulary#

After segmentation we have a list of tokens per document. Several transformations map many token forms onto one. Each one shrinks $|V|$, merges the statistics of forms that probably mean the same thing, and throws away whatever distinguished them.

Lowercasing. Cat, cat and CAT become one token. Loses the signal that distinguishes Apple from apple and US from us.

Stopword removal. Function words (the, is, of, not, …) are dropped from a fixed list. In a bag-of-words model they dominate the counts without carrying topic information. The standard English list in NLTK has 198 entries and includes not, no and nor. Removing them turns “not good” into “good”, which is a known way to break a sentiment classifier.

Stemming. A rule-based procedure strips affixes: studies → studi, running → run, кошками → кошк. The output need not be a word. It is fast and merges aggressively, which also merges forms that should stay apart (universe and university share the Porter stem univers).

Lemmatization. Each form is mapped to its dictionary entry, the lemma: studies → study, went → go, better → good, дети → ребёнок. This needs a part-of-speech decision and a morphological dictionary, so it is slower and language-specific, but the output is a real word and irregular forms are handled.

The two differ most on irregular morphology and on cost. The lemmatizer needs the part of speech: NLTK’s WordNet lemmatizer treats every word as a noun unless told otherwise, so went and better come back unchanged until it is called with pos="v" and pos="a". The table gives the lemma for the right part of speech:

Form	Porter stem	WordNet lemma
studies	studi	study
went	went	go
better	better	good
people	peopl	people

Stemming is a few string operations per token. Lemmatization is a dictionary lookup, sometimes preceded by part-of-speech tagging and followed by morphological analysis, so it is slower by a constant factor that depends on the implementation. In a pipeline that processes a large corpus once, either cost is negligible next to model training; in a service that must tokenize every request, the difference can matter.

Worked example#

Take the sentence above and run the common pipeline: lowercase, split on \w+, remove stopwords, then either lemmatize or stem.

Step	Output
Lowercase + split	what tv shows or books help you read people s body language
Remove stopwords	tv shows books help read people body language
Lemmatize	tv show book help read people body language
Stem	tv show book help read peopl bodi languag

Twelve tokens became eight; shows and books folded onto their singulars, so a document that says “book” and one that says “books” now share a feature.

On a whole corpus the size of the vocabulary is what these choices move. The 120,000 AG News training texts, split by scikit-learn’s default rule (runs of two or more word characters), contain 88,057 distinct types; lowercasing alone brings them down to 65,006. Dropping the types that occur in fewer than four documents (min_df = 4) leaves 29,350: 55% of the lowercased vocabulary, 35,656 types, is seen in at most three documents, too rarely for a model to learn anything from. None of the three counts removes stopwords or lemmatizes.

Why preprocessing depends on the task#

The pipeline above is good for one family of models: bag-of-words classifiers, where every distinct token is a separate feature and rare features are noise. For other tasks the same steps destroy the signal:

• Sentiment. Negation words are stopwords. “not done” and “done” collapse to the same bag. The TF-IDF note shows the standard partial fix, counting bigrams.
• Named entity recognition. Capitalization is the strongest single feature for finding names. The NER note uses a cased model for exactly this reason.
• Language modelling and generation. The model must reproduce every distinction its output needs, such as punctuation, casing and inflection, so none of those may be removed. A byte-level subword tokenizer keeps every character and lets the model learn which variations matter. Not every subword tokenizer is reversible: BERT’s uncased WordPiece lowercases and strips accents, and a tokenizer whose base vocabulary misses a character maps it to an unknown token.
• Retrieval. Stemming raises recall (run matches running) at the cost of precision; whether that trade is good depends on what users search for.

The practical rule: clean what demonstrably hurts (HTML tags, duplicated whitespace, encoding artefacts, exact duplicates), and keep everything that a reader would need to understand the text, unless an experiment shows the model does better without it.

Limitations#

• Normalization and lowercasing are irreversible. If the application has to show text back to the user, keep the original alongside the cleaned version.
• Stopword lists, stemmers and lemmatizers are per language and uneven in quality across languages. Morphologically rich languages (Russian, Turkish, Finnish) need lemmatization far more than English does, and get worse tools.
• Word-level units leave two problems open: any token not in the training vocabulary is unknown at test time, and a large vocabulary means a large embedding table and a long tail of forms seen once. The next note is about these two problems.

What you should remember#

• A text file stores bytes; decoding with an encoding turns them into a string of code points (a Python str); what a reader perceives as one character, a grapheme cluster, can be several code points. A normalization form picks one canonical spelling for visually identical strings. Compare and count only normalized text.
• Segmentation defines the units. Regex and punctuation-aware word tokenizers differ on punctuation and clitics; the right choice depends on whether punctuation is signal for the task.
• Lowercasing, stopword removal, stemming and lemmatization all shrink the vocabulary by merging forms, and each merge discards a distinction that some task needs.
• Stemming is rule-based and fast, lemmatization is dictionary-based and correct on irregular forms; prefer lemmatization for morphologically rich languages when the cost is affordable.
• Preprocessing is part of the model. Decide it per task and measure its effect.

Notebook#

The accompanying notebook is week01_word_embeddings.ipynb. Cell numbers count every cell, markdown included, from 1. Quoted numbers come from the run saved with the course; cells changed since then for current library versions have no saved output.

• Cells 4–6 load a file of Quora questions, one per line, and print line 50, the example used above.
• Cells 9–13 define the two word tokenizers (\w+ and wordpunct_tokenize) and show that they differ on the apostrophe and the question mark.
• Cells 14–17 lowercase and tokenize every line; cells 18–22 remove NLTK’s English stopwords and print the before/after for line 50.
• Cells 23–31 compare the WordNet lemmatizer and the Porter stemmer on the same tokens. Cell 26 tabulates irregular forms; cell 31 prints both outputs for line 50, the last two rows of the worked example (the notebook splits with wordpunct_tokenize, so ' and ? survive there). Cell 28 calls the lemmatizer without a part of speech.
• Cells 33–38 switch to the AG News classification set (120,000 training and 7,600 test texts, four classes) and plot the class distribution, which is balanced; the evaluation note returns to it.
• Cells 44–46 fit a TF-IDF vectorizer with min_df=4 on the AG News texts, without stopword removal or lemmatization; cell 46 prints the vocabulary size 29,350. The other two vocabulary sizes of the worked example, 88,057 and 65,006, are not computed in the notebook: they come from the same vectorizer without the frequency cut, the first with lowercase=False.

Next#

We now have a list of units per document, and the units are words. Two problems remain. A word that never appeared in training has no entry in the vocabulary, so a model can only map it to a single unknown symbol. And a word-level vocabulary of a real corpus has hundreds of thousands of entries, most of them seen a handful of times. The next note replaces words with learned subword units, which bound the vocabulary size and give an unseen word a representation built from known pieces; with all 256 bytes in the base vocabulary, every string has one.

References#

• The Unicode Consortium. Unicode Standard Annex #15: Unicode Normalization Forms. The definitions of NFC, NFD, NFKC and NFKD.
• M. F. Porter. An algorithm for suffix stripping. Program, 14(3), 1980. The Porter stemmer.
• S. Bird, E. Klein, E. Loper. Natural Language Processing with Python. O’Reilly, 2009. Chapter 3 (processing raw text) for tokenizers, stemmers and the WordNet lemmatizer.
• D. Jurafsky, J. H. Martin. Speech and Language Processing, 3rd ed. draft. Chapter 2 (regular expressions, tokenization, normalization).