Quantization

A trained model is a set of matrices of 32-bit floats, and serving it means holding them in memory and multiplying by them for every token. A 7 B-parameter model is 28 GB in 32-bit and 14 GB in 16-bit; at generation time the cost is dominated by reading those bytes, once per token. Quantization stores the numbers in fewer bits. This note derives what that does to a matrix product, measures it on a small model, and explains how the method scales to large language models.

Notation. A real value $x$ in a known range $[a, b]$; its integer code $x_q \in [a_q, b_q]$ (for int8, $[-128, 127]$); scale $s$ and zero point $z$. For a linear layer, $Y = XW + b$ with $X \in \mathbb R^{n \times d}$.

Baseline: 32-bit floats everywhere#

Every weight is a float32: 4 bytes, about 7 significant digits, a range of $10^{\pm 38}$. Nothing in a trained network uses that precision; weights of a layer typically lie in a range of a few units and are not distinguishable at the third decimal. Half precision (float16 or bfloat16) halves the memory with no visible loss and is the default for serving. This note goes below 16 bits with integer codes, and integers need a map. Eight-bit floating-point formats are another route: FP8 tensor cores arrived with NVIDIA’s Hopper (H100) GPUs in 2022, and which formats run fast depends on the hardware generation. The formats covered here are integer affine codes (int8, int4) and the NF4 codebook.

The affine map#

$$x_q = \operatorname{round}\!\Big(\frac{x}{s} + z\Big), \qquad x \approx s\,(x_q - z), \qquad s = \frac{b - a}{b_q - a_q}, \quad z = \operatorname{round}\!\Big(\frac{b\,a_q - a\,b_q}{b - a}\Big).$$

The scale is the real width of one integer step; the zero point is the integer that represents $0.0$, chosen so that $a \mapsto a_q$ and $b \mapsto b_q$. Codes are clipped to $[a_q, b_q]$. The round-trip error $|x - s(x_q - z)|$ is at most $s/2$ for $x$ inside the range, and unbounded outside it, so the choice of $[a, b]$ is the whole art: a range that is too wide wastes codes, a range that is too narrow clips the outliers that matter most.

Worked example#

Range $[a, b] = [-2, 3]$, int8. Then $s = 5/255 = 0.0196$ and $z = \operatorname{round}((3 \cdot (-128) - (-2) \cdot 127)/5) = \operatorname{round}(-26.0) = -26$.

$x$	$x/s + z$	$x_q$	$s(x_q - z)$	error
$-2$	$-128.0$	$-128$	$-2.000$	0
$0.37$	$-7.13$	$-7$	$0.3725$	$0.0025$
$1$	$25.0$	$25$	$1.000$	0
$3$	$127.0$	$127$	$3.000$	0

Figure

Affine int8 quantization of the range [−2, 3]

What happens to a matrix product#

Quantize the input $X$ (scale $s_X$, zero $z_X$), the weight $W$ ($s_W, z_W$) and the bias. Substituting $X = s_X(X_q - z_X)$ and $W = s_W(W_q - z_W)$ into $Y_{ij} = b_j + \sum_k X_{ik} W_{kj}$ and expanding the product of two differences:

$$Y_{ij} = s_b (b_{q,j} - z_b) + s_X s_W \Big[\sum_k X_{q,ik} W_{q,kj} - z_W \sum_k X_{q,ik} - z_X \sum_k W_{q,kj} + d\, z_X z_W\Big].$$

Everything inside the bracket is integer arithmetic: an int8 × int8 matrix product accumulated in int32, two row and column sums, and a constant. The floats appear only in the final rescaling by $s_X s_W$ and the bias. The output can be re-quantized to $(s_Y, z_Y)$ by applying the map once more. This is the identity that makes quantization a speed-up rather than only a memory saving: when the hardware has fast int8 matrix units (modern GPUs and CPUs do), the bracket runs several times faster than the float product, and the weights read from memory are a quarter the size.

Figure

A quantized linear layer

Two things are required for the speed-up to appear. The kernel must exist for the hardware, and the activation ranges must be known. Dynamic quantization measures the range of $X$ at run time, per batch, and quantizes weights ahead of time; it needs no calibration data and pays a small cost per batch. Static quantization fixes the activation ranges in advance from a calibration set of a few hundred inputs, so that the whole graph runs in integers; it is faster and more fragile. Quantization-aware training simulates the rounding during fine-tuning so that the weights adapt to it; it recovers accuracy at the lowest bit widths at the cost of a training run.

Measurements on a small classifier#

The model is DistilBERT (66.96 M parameters), fine-tuned on IMDb sentiment for two epochs; the test set is 25,000 reviews. Baseline: 0.249 GiB in float32, accuracy 0.9313. Memory counts parameters and buffers and is given in GiB ($2^{30}$ bytes).

Implementation	What is quantized	Memory, GiB	Accuracy	Time
float32	nothing	0.249	0.9313	24.5 s per batch of 64 (CPU); 2 min 4 s for the test set (GPU)
Hand-written int8 linear layers (the formula above)	linear weights and inputs	0.129	—	slower than float (no int kernel)
PyTorch dynamic quantization (torch.ao)	linear layers, int8	0.089, an undercount	—	20.4 s per batch (CPU)
bitsandbytes 8-bit (LLM.int8())	linear layers, int8 on GPU	0.085	0.9266	3 min 50 s for the test set (GPU)

Reading the table:

• Memory halves with the hand-written layers: the embedding tables, 36 % of the parameters, stay in float32 and become two thirds of the quantized model. The two library numbers need care. Dynamic quantization stores its int8 weights as packed objects that the count of parameters and buffers does not include, so its 0.089 is essentially the float embeddings alone; the true size is close to the hand-written 0.129. bitsandbytes loads everything it does not quantize in float16, which is why it goes below 0.129. Quantizing the embeddings as well would go further.
• Accuracy drops by half a point at 8 bits with no retraining; this is typical for post-training int8 on encoders.
• Speed is where implementations differ. The hand-written int8 layer is 24× slower than float32 (1.23 ms against 51 μs for a $256 \times 128$ input on a $128 \times 128$ layer), because PyTorch has no optimized int8 matmul on that CPU path and the sums are done in generic int32 ops; the arithmetic is right (mean absolute error 0.0011 against the float output) and the kernel is missing. The built-in dynamic quantization is 17 % faster than float on CPU. On the GPU, comparing full passes over the test set on the same device, the 8-bit model is slower: 3 min 50 s against 2 min 4 s in float32. For a 67 M-parameter model that is not limited by memory traffic, the int8 kernels and the outlier handling of LLM.int8() cost more than they save. Quantization pays off only with a kernel that wins on the target hardware at the model’s size; otherwise it is a memory tool.

The torch.ao path also does not run on GPU at all (the dynamic int8 linear op is CPU-only), which is one reason practitioners use the bitsandbytes, GPTQ and AWQ ecosystems for Transformers rather than the framework’s own quantization.

Weight-only quantization for large language models#

Generation with a large decoder is memory-bound: each token needs every weight read once and does little arithmetic per byte. Reducing the bytes is the lever, so LLM quantization is usually weight-only: weights in 4 or 8 bits, activations kept in 16-bit, with kernels that dequantize weights on the fly inside the matrix product. Two schemes dominate:

LLM.int8() (Dettmers et al., 2022) quantizes weights and activations to int8 per row and column, except that a few activation columns with outliers (large magnitudes that appear in models above a few billion parameters) are kept in 16-bit and multiplied separately. Without the outlier path, 8-bit accuracy collapses above 6 B parameters; with it, it matches 16-bit.

NF4 (Dettmers et al., 2023) is a 4-bit format whose 16 levels are placed at the quantiles of a standard normal rather than evenly, because trained weights are approximately normal and evenly spaced levels waste codes in the tails. Combined with double quantization (quantizing the per-block scales themselves to 8 bits) it stores a 7 B model in about 3.5 GB and is the base of QLoRA in the PEFT note.

GPTQ: quantizing weights with calibration data#

Rounding every weight to its nearest level ignores that the weights of a layer act together: a rounding error in one weight can be compensated by adjusting the others. GPTQ (Frantar et al., 2023) does this compensation layer by layer.

This section follows the paper’s orientation. The layer computes $W X_{\text{cal}}$ with $W \in \mathbb R^{d_{\text{out}} \times d_{\text{in}}}$, the transpose of the $W$ above, and $X_{\text{cal}} = X^\top \in \mathbb R^{d_{\text{in}} \times N}$ holds the $N$ calibration token vectors as columns. A row $w \in \mathbb R^{1 \times d_{\text{in}}}$ of $W$ holds one output’s weights, and $\operatorname{quant}(w_q)$ is the weight $w_q$ rounded to its nearest level.

1. Run a small calibration set (128 sequences is the usual recommendation) through the model and collect $X_{\text{cal}}$. Changing a row by $\delta$ changes that output by $\delta X_{\text{cal}}$, so the squared output error is $\|\delta X_{\text{cal}}\|^2 = \tfrac12\, \delta H \delta^\top$ with Hessian $H = 2 X_{\text{cal}} X_{\text{cal}}^\top \in \mathbb R^{d_{\text{in}} \times d_{\text{in}}}$, the same for every row.
2. Quantize the input coordinates (the columns of $W$) one at a time, all rows at once because they share $H$. Let $F$ be the set of coordinates not yet quantized and $H_F$ the Hessian restricted to them. Rounding coordinate $q$ moves it from $w_q$ to $\operatorname{quant}(w_q)$, and the update of the coordinates in $F$ that minimizes the output error with that coordinate fixed is the Optimal Brain Surgeon step (Hassibi and Stork, 1993; Frantar et al., 2023, Section 3, Eq. 2): $$\delta_F = -\frac{w_q - \operatorname{quant}(w_q)}{[H_F^{-1}]_{qq}}\, (H_F^{-1})_{:,q}.$$ Its $q$-th entry is $-(w_q - \operatorname{quant}(w_q))$, which puts coordinate $q$ exactly on its level; then $q$ leaves $F$.
3. Proceed through the coordinates in a fixed order, with a lazy batched update and a small dampening of $H$ for numerical stability.

Worked example (toy numbers). Two strongly correlated inputs with $X_{\text{cal}} X_{\text{cal}}^\top = \begin{pmatrix} 1 & 0.9 \\ 0.9 & 1 \end{pmatrix}$, a row $w = (0.3, 0.4)$ and integer levels. Rounding each weight on its own gives $(0, 0)$, an error $\delta = (-0.3, -0.4)$ and $\|\delta X_{\text{cal}}\|^2 = 0.09 + 0.16 + 2 \cdot 0.9 \cdot 0.12 = 0.466$. GPTQ rounds $w_1$ to $0$. The inverse Hessian’s first column divided by its first entry is $(1, -0.9)$, so $\delta = -0.3 \cdot (1, -0.9) = (-0.3, 0.27)$ and $w_2$ moves to $0.67$, which rounds to $1$. The result $(0, 1)$ has error $(-0.3, 0.6)$ and $\|\delta X_{\text{cal}}\|^2 = 0.09 + 0.36 - 2 \cdot 0.9 \cdot 0.18 = 0.126$, under a third of the independent rounding. With the sign of the update flipped, $w_2$ would move to $0.13$ and round back to $0$.

The result is a 4-bit (or 3-bit) model whose layer outputs are as close as possible, in the least-squares sense on the calibration data, to the original’s. Scales are stored per group of 128 weights along a row (group_size=128), a compromise between per-tensor (coarse) and per-weight (no saving). The quantized weights are packed eight 4-bit codes per int32 (qweight), with the zero points packed likewise (qzeros), and specialized kernels (ExLlama) read them.

Calibration data matters. A 125 M-parameter model (OPT-125M) quantized to 4 bits with a few hundred WikiText-2 sequences as the calibration set generates fluent text (“Hello my name is Alex and I’m a 25 year old male…”); the same model calibrated on a single sentence generates My name is "M"""""""". One sentence of $n$ tokens gives $n$ token vectors, so its $H$ has rank at most $n$, far below the layers’ input sizes (768 or 3,072 in OPT-125M); it is singular, and only the dampening makes it invertible. A plausible explanation is that the compensation then rests on a few poorly estimated directions and moves the remaining weights where the real inputs do not go. The run does not measure the spectrum of $H$, so this is a hypothesis, as is the general advice that calibration data should be sufficient and diverse. A pre-quantized 7 B chat model from the Hub loads into 4-bit layers and generates normally.

Distillation and pruning#

Quantization keeps the architecture and shrinks the numbers. Two other compression routes change the architecture:

Knowledge distillation (Hinton, Vinyals and Dean, 2015) trains a smaller student to match a larger teacher’s output distribution, not only the hard labels:

$$\mathcal L = \alpha\, \mathcal L_{\text{CE}}(y, p_s) + (1 - \alpha)\, T^2\, D_{\mathrm{KL}}\big(\operatorname{softmax}(z_t / T)\,\|\,\operatorname{softmax}(z_s / T)\big),$$

where $z_t, z_s$ are teacher and student logits and the temperature $T > 1$ softens both distributions so that the student also learns which wrong classes the teacher considers close. The $T^2$ factor keeps the gradient magnitude comparable across temperatures. DistilBERT itself, the model measured above, is BERT-base distilled to 6 layers with 40 % fewer parameters and 97 % of the GLUE score (Sanh et al., 2019). For language models, distillation also takes the form of training on a large model’s generated text (the Alpaca route of the instruction tuning note).

Pruning removes weights (unstructured) or whole heads, neurons or layers (structured). Unstructured sparsity saves memory only with sparse kernels; structured pruning gives a smaller dense model and needs retraining to recover.

Trade-offs#

Bits	Memory vs fp16	Typical quality change	Needs
16	1×	none	nothing
8 (weights + activations)	½	small (< 1 point here)	int8 kernel; outlier handling above ~6 B
4 (weights only, GPTQ / NF4 / AWQ)	¼	small to moderate; grows for small models	calibration data (GPTQ) or none (NF4); dequantizing kernel
3 and below	< ¼	noticeable; task-dependent	careful calibration, often QAT

Throughput follows memory only when the kernel exists, and accuracy results on one benchmark do not transfer to another: a model quantized with perplexity as the criterion can lose more on reasoning or long generation than the perplexity suggests. Any specific number for a current model is a snapshot; the mechanism and the trade-off shape are what to keep.

What changed#

	Float model	Quantized model
Storage per weight	32 or 16 bits	8 or 4 bits + a scale per group
Matrix product	float	integer bracket + one float rescale
Accuracy	reference	small loss at 8 bits, more at 4
Speed	reference	faster only with a hardware kernel
Training	normal	frozen (PTQ) or simulated rounding (QAT); LoRA on top (QLoRA)

Limitations#

• Range and outliers. A few large activations can force a wide range that wastes most codes; mixed precision for outlier columns is the standard patch.
• Kernels. The memory saving is universal; the speed-up is not. Without a matching kernel, quantized layers can be slower than float.
• Small models suffer more. Fewer parameters leave less redundancy to absorb rounding; 4-bit is routine at 7 B and risky at 100 M.
• Benchmark drift. Quantization is evaluated on a metric; losses elsewhere are easy to miss.
• Not reversible. The original precision is gone; keep the float checkpoint.

What you should remember#

• Affine map: $x_q = \operatorname{round}(x/s + z)$, $x \approx s(x_q - z)$, with $s = (b - a)/(b_q - a_q)$; error $\le s/2$ inside the range, clipping outside.
• A quantized linear layer is an integer matrix product plus row/column sums, rescaled once by $s_X s_W$.
• DistilBERT on IMDb: 0.249 → 0.085 GiB and 0.9313 → 0.9266 at 8 bits; on the GPU the 8-bit model was slower than float32, because speed depends entirely on the kernel and the model size.
• LLMs are quantized weight-only (int8 with outlier columns, NF4, GPTQ 4-bit with group size 128); GPTQ compensates rounding errors one input coordinate at a time using a Hessian from calibration data, and in this run a one-sentence calibration set ruined the output.
• Distillation trains a small model on a large model’s soft outputs; pruning removes parameters; both change the architecture, quantization does not.

Notebook#

Two notebooks. Cell numbers count every cell, markdown included, from 1. Quoted numbers come from the run saved with the course; cells changed since then for current library versions have no saved output.

week12_pt0_Torch_and_ONNX_Quantization_+_PEFT.ipynb:

• Cell 4 states the distillation loss; cells 6–18 load IMDb and a DistilBERT fine-tuned on it (now the public lvwerra/distilbert-imdb, so current accuracies differ slightly; the numbers here are from the course’s own checkpoint, trained for two epochs), build batches of 64, print 66,955,010 parameters, 0.249 GiB (memory is get_memory_footprint() divided by $2^{30}$), and accuracy 0.9313 (2 min 4 s over the test set on GPU).
• Cell 19 derives the integer matrix product reproduced above; cells 20–22 implement QuantizedTensor (per-tensor min–max range with a $10^{-6}$ margin) and QuantizedLinear; cells 24–29 time a $128 \to 128$ layer (51 μs float, 1.23 ms quantized) and measure the mean absolute error 0.0011; cells 31–32 replace every linear layer and print 0.129 GiB.
• Cells 36–44 apply torch.ao.quantization.quantize_dynamic (0.089 GiB as counted by get_memory_footprint(), which skips the packed int8 weights; 20.4 s against 24.5 s per batch on CPU); cell 46 shows the GPU failure of the dynamic int8 op (it runs only when a CUDA device is present).
• Cells 50–57 load the model with BitsAndBytesConfig(load_in_8bit=True): 0.085 GiB, accuracy 0.9266, 3 min 50 s over the test set on GPU; the markdown after it claims a 1.5× speed-up by comparing one GPU batch (3.6 s) with one CPU batch; the same-device comparison is the two full test-set runs on GPU (3 min 50 s against 2 min 4 s).
• Cells 59–69 run static quantization with ONNX Runtime: export to ONNX, min–max ranges from 200 padded calibration examples with LayerNorm, GELU, residual additions and softmax kept in float, the accuracy of the quantized model, and dynamic int8 through the same pipeline for comparison. The markdown in cell 68 reports about 0.80 for static int8 against 0.91 for the float model on the same subset of reviews, the cost of plain min–max calibration; the course version of these cells was marked as not working and had no outputs.
• Cells 71–76 are the QLoRA template described in the PEFT note.

week12_pt1_Distillation_and_GPTQ_Quantization.ipynb:

• Cells 19–23 quantize facebook/opt-125m to 4 bits with GPTQConfig(bits=4, group_size=128, dataset="wikitext2", desc_act=False), inspect the packed qweight and qzeros int32 buffers, and generate the fluent sentence quoted above.
• Cells 26–28 repeat with a one-sentence calibration set and generate the degenerate output.
• Cells 32–38 load a pre-quantized Llama-2-7B chat model, print the QuantLinear layers and the quantization config (4-bit, group size 128, ExLlama kernel), and generate 50 tokens.
• Cells 41–51 fine-tune it with LoRA, as described in the PEFT note.

Next#

The course has now built, pretrained, adapted and compressed models. The next section turns to the tasks those models are used for and asks, for each, what the input and output are, which metric measures it, what the classical baseline was, and what the Transformer-era approach changed. The text classification note opens it with the task that has run through the whole course, and collects every AG News result in one table.

References#

• B. Jacob et al. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. CVPR, 2018. The affine scheme and the integer matrix product.
• T. Dettmers, M. Lewis, Y. Belkada, L. Zettlemoyer. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. NeurIPS, 2022.
• T. Dettmers, A. Pagnoni, A. Holtzman, L. Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS, 2023. NF4 and double quantization.
• E. Frantar, S. Ashkboos, T. Hoefler, D. Alistarh. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR, 2023.
• B. Hassibi, D. G. Stork. Second order derivatives for network pruning: Optimal Brain Surgeon. NeurIPS, 1993.
• J. Lin et al. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. MLSys, 2024.
• G. Hinton, O. Vinyals, J. Dean. Distilling the Knowledge in a Neural Network. NeurIPS Deep Learning Workshop, 2015.
• V. Sanh, L. Debut, J. Chaumond, T. Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv, 2019.
• Hugging Face. Quantization overview. Which backend supports which device and bit width.