Parameter-efficient fine-tuning — NLP

Full fine-tuning in the previous note reached 0.890 on SST-2 and paid for it with gradients and optimizer states for all 124 million parameters and a full copy of the model per task. Linear probing paid almost nothing and reached 0.737. The question is whether there is a way to train a few parameters and still get the fine-tuning number. There is, and the main method turns on one observation about what fine-tuning actually changes.

Notation. A pretrained weight matrix $W \in \mathbb R^{d_{\text{out}} \times d_{\text{in}}}$ (frozen); its fine-tuned version $W + \Delta W$ ; a layer input $x$ and output $h$ . $r$ is a rank; $\sigma$ a sigmoid.

Baseline: all or nothing#

Two regimes exist so far. Train everything ( $\Delta W$ is free, $d_{\text{out}} d_{\text{in}}$ parameters per matrix, hundreds of matrices) or train only a head ( $\Delta W = 0$ everywhere). The first is expensive, the second underfits. The methods below sit between them by restricting the form of $\Delta W$ or by adding a small module whose parameters are the only ones trained.

Figure

Where each PEFT method adds trained parameters

One Transformer block (residual connections and normalization omitted). Grey parts are pretrained and frozen, orange parts are the only ones trained. Prompt tuning prepends k learned vectors to the input embeddings and touches nothing inside. LoRA adds a low-rank branch B A alongside the attention projection matrices, which can be merged into W after training. Adapters insert small bottleneck layers after the sub-layers, which stay in the model at inference. In all three, the per-task copy is only the orange parts.

Adapters#

Houlsby et al. (2019) insert a small bottleneck module after each sub-layer of every block: project the $d$ -dimensional activation down to $m \ll d$ , apply a non-linearity, project back up, and add the result to the activation (a residual). Only the adapter weights ( $2dm$ per adapter plus biases) and the layer norms are trained. With $d = 768$ and $m = 64$ an adapter has about 100 K parameters, and a 12-block model with two adapters per block adds 2.4 M, 2 % of BERT-base, for accuracy within a point of full fine-tuning on the GLUE tasks of the paper. The cost is at inference: the adapters are extra layers that run on every forward pass and cannot be removed.

LoRA: a low-rank update#

Hu et al. (2022) built LoRA on the observation, empirical and made earlier by Aghajanyan, Zettlemoyer and Gupta (2021), that the update a pretrained model needs for a downstream task has low intrinsic rank: most of the directions in $\Delta W$ do not matter. The layer computes

h = W x + \Delta W x = W x + B A x = W x + B\,(A x),

evaluated right to left: $Ax$ is $r$ numbers, $B(Ax)$ is back in $d_{\text{out}}$ dimensions. $A$ is initialized with small Gaussian values and $B$ with zeros, so $\Delta W = 0$ at the first step and training starts from exactly the pretrained model. A scale $\alpha / r$ multiplies the correction so that changing $r$ does not change the effective learning rate.

Figure

LoRA: a frozen weight matrix plus a trained low-rank correction

The pretrained matrix W (d × d, blue) is never updated. Two small matrices are trained instead: A, which maps the d-dimensional input down to r numbers, and B, which maps them back up to d. Their product B A is a d × d matrix of rank at most r, and the layer computes h = W x + B A x. A starts random and B starts at zero, so at the first step the correction is zero and the layer behaves exactly like the pretrained one. The two thin matrices hold 2dr parameters against d² for W; with d = 768 and r = 8 that is 12,288 against 589,824.

Worked example#

$d_{\text{in}} = d_{\text{out}} = 4$ , $r = 1$ . A full update is 16 numbers; LoRA trains $A = (1, 0, -1, 0)$ (4 numbers) and $B = (2, 0, 0, 1)^\top$ (4 numbers). Their product is

BA = \begin{pmatrix} 2 \\ 0 \\ 0 \\ 1 \end{pmatrix}\begin{pmatrix} 1 & 0 & -1 & 0 \end{pmatrix} = \begin{pmatrix} 2 & 0 & -2 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 1 & 0 & -1 & 0 \end{pmatrix},

a $4 \times 4$ matrix of rank 1: every row is a multiple of $A$ . For $x = (1, 1, 0, 1)$ , $Ax = 1$ and $BAx = (2, 0, 0, 1)$ : the correction adds the direction $B$ scaled by how much $x$ aligns with $A$ . Eight trained numbers specify a 16-entry matrix, at the price that the matrix can only point in one direction. With $r$ directions it can point in $r$ .

Parameter count#

Per matrix: $r(d_{\text{in}} + d_{\text{out}})$ against $d_{\text{in}} d_{\text{out}}$ . For the fused attention projection of GPT-2 ( $768 \to 2{,}304$ , producing $Q, K, V$ at once) with $r = 8$ : $8 \times (768 + 2{,}304) = 24{,}576$ per block, 294,912 over 12 blocks, plus the 1,536-parameter head: 296,448 trainable parameters, 0.24 % of the model. The original paper applies LoRA to the query and value projections only; here it is applied to GPT-2’s fused projection, which covers query, key and value because GPT-2 computes them in one matrix.

Result#

Regime	Trainable	Val. accuracy	Val. loss
Full fine-tuning	124 M (100 %)	0.890	0.277
Linear probing	1,536 (0.001 %)	0.737	0.541
LoRA, $r = 8$ on attention	296 K (0.24 %)	0.899	0.271
Prompt tuning, 12 tokens	10.8 K (0.009 %)	0.663	0.656
LP-FT	124 M	0.900	0.258

LoRA matches full fine-tuning (the one-point difference is noise on 872 examples) with 400 times fewer trained parameters, a learning rate of $10^{-4}$ (the new matrices start from scratch, so the small-step caution of full fine-tuning does not apply), and no optimizer state for the frozen body. The adapter for a task is a file of 296 K numbers that can be swapped in and out of one shared base model.

Prompt tuning#

Lester, Al-Rfou and Constant (2021) train nothing inside the model. A sequence of $k$ virtual tokens, free vectors of dimension $d$ , is prepended to every input’s embeddings; the model is frozen; only those $k \times d$ numbers (and here the head) are trained. The method rests on the observation that large language models can be steered by a prompt, and replaces hand-written prompts by learned ones. Initializing the virtual tokens from the embeddings of a natural sentence (Predict if sentiment of this review is positive or negative, 11 tokens, padded to 12) gives the optimizer a sensible start.

Result: 0.663 with 10,752 trainable parameters, below linear probing. The paper’s experiments use T5 encoder–decoder models on SuperGLUE, and there prompt tuning matches full fine-tuning only near the largest size tried, about 11 B parameters, and lags well behind for the smaller ones (Lester et al., 2021, Figure 1). That curve is evidence about T5, not a rule for every architecture, but it points the same way as this run: a 124 M GPT-2 is far below the size at which the gap closed. Soft prompts themselves are not tied to one architecture. The paper prepends them to the input of an encoder–decoder, and the same mechanism applies to an encoder or to a decoder-only model; how well it works depends on the model and its size, and has to be measured for each setup. Its appeal is size: a task is a few thousand numbers.

Prefix tuning (Li and Liang, 2021) is the stronger relative: learned vectors are injected as extra keys and values into every attention layer rather than only at the input.

Fine-tuning a quantized model: QLoRA#

LoRA keeps the frozen body in memory, and for a 7 B model that is 14 GB in 16-bit. Dettmers et al. (2023) store the frozen body in 4 bits (the NF4 format of the quantization note, 3.5 GB for 7 B), dequantize each matrix on the fly for the forward pass, keep the LoRA matrices in 16-bit, and apply a second quantization to the quantization constants themselves. The gradients flow through the dequantized weights into $A$ and $B$ only. The same recipe works on a model quantized with GPTQ: for a 4-bit Llama-2-7B, LoRA with $r = 8$ on the four attention projections adds 8.4 M trainable parameters, and ten training steps on a quotes dataset run on a single consumer GPU. The quality of QLoRA fine-tunes is close to 16-bit LoRA in the paper’s experiments; the mechanism that makes it possible is that the frozen weights never need gradients, so their storage format is free to be anything the forward pass can read.

Trade-offs#

Method	Trained	Inference cost	Where it applies	Quality (this task)
Full fine-tuning	100 %	none	any	0.890
Adapters	~2 %	extra layers	any	≈ full (literature)
LoRA	0.1–1 %	none after merging	linear layers	0.899
Prompt tuning	< 0.01 %	$k$ extra tokens per input	any; competitive at large scale (T5 in the paper)	0.663
QLoRA	LoRA on a 4-bit body	4-bit body	large decoders	≈ LoRA (literature)

Practical choices: $r$ between 4 and 64 (the paper found $r = 1$ to $8$ sufficient for GPT-3 on its tasks, and quality flat beyond); apply to attention projections first, and to the feed-forward matrices as well when the budget allows; learning rate around $10^{-4}$ to $3 \times 10^{-4}$ ; dropout on the LoRA path.

What changed#

	Full fine-tuning	LoRA
$\Delta W$	Free	Rank $\le r$
Parameters per task	All	$r(d_{\text{in}} + d_{\text{out}})$ per matrix
Optimizer memory	2 states per weight	2 states per LoRA weight
Artefact per task	Full model	Adapter file
Inference	Same	Same, after merging
Learning rate	Small	Larger (new weights)

Limitations#

Rank is a bet. Tasks that may need large, high-rank changes (a new language, long continued pretraining on a new domain) can be served less well by a small $r$ . This is a capacity and optimization trade-off to measure case by case, not a rule that LoRA cannot learn domain content.
Frozen body, limited capacity. Everything new is stored in $\Delta W$ . That update is trained and changes the outputs, facts included; a small $r$ limits how much it can change. What separates task adaptation from continued pretraining is mainly the objective and the data (labelled task examples against large amounts of raw text), and either can be run with low-rank updates.
Scale dependence. Prompt tuning’s usefulness depends on model size; LoRA’s hyperparameters ( $r$ , target modules, $\alpha$ ) depend on the model and task and are chosen empirically.
Adapters add latency, LoRA does not; QLoRA trades dequantization compute for memory.
Noise. The comparisons above rest on one run each on 872 examples; differences of a point are not conclusions.

What you should remember#

LoRA: $h = Wx + BAx$ with $W$ frozen, $A$ random, $B = 0$ at start; $r(d_{\text{in}} + d_{\text{out}})$ trained parameters per matrix; merge $BA$ into $W$ for inference.
On SST-2 with GPT-2 small, LoRA ( $r = 8$ , attention) gives 0.899 against 0.890 for full fine-tuning with 0.24 % of the parameters trained.
Adapters add small bottleneck layers (extra inference cost); prompt tuning learns input vectors, caught up with full fine-tuning only near 11 B parameters in Lester et al.’s T5 experiments, and gave 0.663 on GPT-2 small here.
QLoRA puts the frozen body in 4 bits and trains LoRA on top, which fits 7 B-parameter fine-tuning on one GPU.
The rank is a hypothesis about how much the task needs to change the model. A small rank usually suffices for task adaptation; larger changes, such as a new domain, may need a larger $r$ , more target modules or full training, and the choice is checked on validation data.

Notebook#

week10_PEFT.ipynb. Cell numbers count every cell, markdown included, from 1. Quoted numbers come from the run saved with the course; cells changed since then for current library versions have no saved output.

Cells 35–39 configure LoraConfig(task_type=SEQ_CLS, r=8, lora_dropout=0.1, target_modules=['c_attn']), where c_attn is GPT-2’s fused query–key–value projection, and wrap the model with get_peft_model; cell 39 counts 296,448 trainable parameters. The saved output of cell 38 lacks the print_trainable_parameters line; with the LoRA matrices included the model has 124,738,560 parameters, so the trainable share is 0.24 %.
Cells 40–43 train with AdamW at $10^{-4}$ for 1,000 steps and report validation loss 0.2708 and accuracy 0.8991.
Cells 45–49 configure PromptTuningConfig with 12 virtual tokens initialized from the sentence quoted above (num_virtual_tokens=12, token_dim=768), print 10,752 trainable parameters, and cells 52–54 train and report 0.6560 loss and 0.6628 accuracy.

week12_pt1_Distillation_and_GPTQ_Quantization.ipynb, cells 41–51: a 4-bit GPTQ Llama-2-7B is prepared with prepare_model_for_kbit_training, wrapped with LoraConfig(r=8, lora_alpha=32, target_modules=["k_proj","o_proj","q_proj","v_proj"], lora_dropout=0.05), which prints 8,388,608 trainable parameters, and trained for 10 steps with Trainer; the “all params” figure printed there (270 M) counts packed 4-bit weights as single int32 parameters and is not the model’s true size.

week12_pt0_Torch_and_ONNX_Quantization_+_PEFT.ipynb, cells 71–76: the QLoRA configuration (load_in_4bit, nf4, double quantization, bfloat16 compute) and LoraConfig(r=16, target_modules="all-linear") for a Mistral-7B base, which puts LoRA on the feed-forward matrices as well as the attention projections; the cells are a template and are not run to completion.

Next#

Adaptation so far has meant a labelled task: sentiment, topic. The models people actually interact with were adapted to something broader, the behaviour of answering an instruction, and the next note describes that process: supervised fine-tuning on instruction–response pairs, then optimization against human preferences, with LoRA and QLoRA as the usual way to afford it.

References#

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen. LoRA: Low-Rank Adaptation of Large Language Models. ICLR, 2022.
A. Aghajanyan, L. Zettlemoyer, S. Gupta. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. ACL, 2021.
N. Houlsby et al. Parameter-Efficient Transfer Learning for NLP. ICML, 2019. Adapters.
B. Lester, R. Al-Rfou, N. Constant. The Power of Scale for Parameter-Efficient Prompt Tuning. EMNLP, 2021.
X. L. Li, P. Liang. Prefix-Tuning: Optimizing Continuous Prompts for Generation. ACL, 2021.
T. Dettmers, A. Pagnoni, A. Holtzman, L. Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS, 2023.
Hugging Face. PEFT documentation. The library used in the notebooks.