Retrieval-augmented generation

A language model answers from what it absorbed in pretraining, which is frozen at a date, general rather than specific, and recalled with the same confidence whether right or wrong. The previous note built a retriever that finds the passages relevant to a query in a corpus of one’s own. Retrieval-augmented generation (RAG, Lewis et al., 2020) puts the two together: find the passages, hand them to the model, ask it to answer from them. The idea is one sentence; the failures are in the details of each stage.

Notation. Query $q$; retrieved chunks $d_1, \dots, d_k$ in rank order; a context $C$ built from them; the generator produces $a \sim P(\cdot \mid \text{instruction}, C, q)$. A token budget $L$ bounds $|C| + |q| + |a|$.

Baseline: ask the model#

Prompt an instruction-tuned model with the question alone. For general knowledge it answers well. For a question about a document it has not seen (a company’s internal policy, a post from last week, a codebase), it answers anyway, from whatever its training data suggests, in the same fluent register. Nothing in the output distinguishes the two cases.

The pipeline#

Figure

Retrieval-augmented generation: an offline index and an online query path

1. Index (offline). Chunk the documents, embed the chunks, store vectors with their text and metadata (source, position, date) in a persistent index, so that the corpus is embedded once. Keep a lexical index too for hybrid retrieval.
2. Retrieve. Embed the query (optionally after rewriting it), fetch the top-$k$ chunks by similarity, merge with BM25 results.
3. Rerank. Re-score the candidates with a cross-encoder and keep the top $k'$ (3–10).
4. Build the context. Concatenate the kept chunks with their sources, in an order the model handles well, within the token budget; include the instruction.
5. Generate. Produce the answer; optionally with citations to the chunk identifiers, and in a structured format when a program will consume it.
6. Evaluate retrieval and generation separately (below).

One query, four stages#

Steps 2 to 5, followed by postprocessing, run for every query as a chain of four stages, each consuming what the previous one returns. Retrieval augmentation and the prompt together cover steps 2 to 4, and generation is step 5:

1. Retrieval augmentation. The question goes to the retriever (and the reranker); the passages it returns are joined into one context string, each with its identifier, while the question itself is passed on unchanged. The output is a record of two fields, context and question.
2. Prompt. A template fills the record into the instruction, and the model’s chat template renders the result into tokens (below).
3. Generation. The model continues the prompt.
4. Postprocessing. The raw output is cut down to the answer: anything the generation library returns besides the new text (some return the prompt followed by the continuation), a closing special token, the start of a further turn the model began to write. When a program reads the answer, it is parsed here.

Written as composed stages, each part can be replaced and tested on its own: the first stage is scored on the golden set of the previous note, the prompt is inspected as text, and the postprocessing is tested on recorded outputs.

Building the context#

The context is a prompt, and prompt details change answers. Four decisions recur.

Budget. With a 4,096-token context window, an instruction of 200 tokens, a question of 50 and an answer allowance of 500, 3,346 tokens remain for chunks: thirteen 256-token chunks, or five parent chunks of 600. More chunks raise the chance the answer is present and lower the chance the model attends to it; the reranker’s job is to make the first few count.

Order. Models attend better to the beginning and end of a long context than to its middle (Liu et al., 2024); place the highest-ranked chunks first or last, not in the middle of a long list.

Figure

A context under a token budget

Instruction. State the rule explicitly: answer only from the provided passages; if they do not contain the answer, say so; cite the passage identifiers you used. The instruction does not guarantee compliance; it shifts the distribution toward it, and the evaluation checks the rest.

Template. The prompt has two layers. The application’s template is text with slots: the instruction, the passages as one {context} string, the {question}, and in a conversation a {history} slot (below). The model’s chat template then wraps the resulting messages in the role markers the model was tuned on. The chat template is stored with the tokenizer as a small program in Jinja, a text-templating language with loops and conditions. Simplified, the Llama 3 template is

{{ bos_token }}{% for message in messages %}<|start_header_id|>{{ message['role'] }}<|end_header_id|>

{{ message['content'] | trim }}<|eot_id|>{% endfor %}{% if add_generation_prompt %}<|start_header_id|>assistant<|end_header_id|>

{% endif %}

and a RAG request with one system and one user message renders as

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Answer only from the passages. If they do not contain the answer, say so.<|eot_id|><|start_header_id|>user<|end_header_id|>

Passages:
[1] …
[2] …

Question: How is pneumonia treated?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

The model writes its answer after the last header. Rendering with the tokenizer’s own template, rather than typing the markers by hand, keeps the prompt in exactly the format the model was tuned on; a prompt in another model’s format, or in plain text, tends toward base-model behaviour (continuing the text instead of answering, not stopping cleanly).

Structured output#

When the answer feeds a program (a classification, an extracted record, a rating), free text is a liability: the model adds explanations, hedges and formatting that break parsers on some fraction of inputs. Constrained decoding fixes it at the token level: given a schema, the sampler masks every token that cannot continue a valid output, so the result is a parseable object by construction. With a pydantic schema of a recipe (name, optional preparation time, a list of ingredients with name and quantity, a list of instruction strings), a model given a paragraph of free-form cooking text returns that object; for one such paragraph the object lists nine ingredients with their quantities and seven instruction steps. The same mechanism restricts a classifier to its label set and a rating to an integer in $[1, 5]$. Hosted APIs expose it as a JSON-schema response format; local models use libraries that compile the schema into a token mask.

Follow-up questions in a conversation#

A chat over documents gets follow-ups: What about in children? after How is pneumonia treated? Two parts of the pipeline have to change.

The prompt carries the history. A buffer memory stores the earlier turns and fills them into the {history} slot of the template, next to the context and the question, so the model knows what in children refers to. The buffer grows with every turn and is bounded like any conversation state: the last few turns, a token budget, or a running summary, as described for conversation memory in the next note.

Retrieval needs a standalone question. The retriever sees only the text it is given. Searching for What about in children? returns passages about children in general, not about pneumonia in children, and adding the history to the prompt does not repair what retrieval missed. The usual remedy is a rewriting step before retrieval: a model reads the history and the follow-up and writes a self-contained question, How is pneumonia treated in children?, which is used for the search, while the prompt still gets the history and the original follow-up. It costs one more generation per turn.

When retrieval is the right tool#

A short decision list for whether to build this at all:

1. Does the correct answer depend on documents the model has not seen? If not, no retrieval.
2. Do those documents fit in the context window? If yes, put them all in the prompt and skip the index; retrieval is for corpora larger than the window.
3. Are queries short and phrased differently from the documents? If the gap is moderate, try other embedders, query rewriting and reranking. If it is large, consider symmetric search: index questions instead of passages, each with its passage attached, and match the query against them.
4. Is traceability required (which source said this)? Then keep the identity of every source through the pipeline, cite it in the answer, and check each claim against the source it cites. Retrieval keeps sources by construction, and a pipeline that puts all the documents in the prompt can cite document IDs just as well; retrieve when the corpus size or the need to select calls for it.
5. Is the cost of a wrong answer high? Then the instruction must be strict: answer only from the context. Allowing the model to abstain and checking faithfulness in evaluation follow from the same concern.

A corollary: RAG is not a way to teach a model a domain. It gives it the relevant text at query time; it does not improve its reasoning over that text or its vocabulary. For the latter, the adaptation methods of the earlier section apply, and the two combine.

Failure modes, by stage#

Stage	Failure	What it looks like	Mitigation
Indexing	chunk cuts the answer	half a table, a sentence without its subject	overlap, parent retrieval, structure-aware chunking
Indexing	stale index	answers from the old version	re-embed on change; store timestamps
Retrieval	miss (answer not in top-$k$)	fluent answer from nothing, or correct abstention	hybrid search, reranking, larger $k$, query rewriting; measure Hit@$k$
Retrieval	distractors (on-topic, no answer)	confident answer from the wrong passage	reranker, instruction to abstain
Context	lost in the middle	correct chunk retrieved, ignored	order by rank, fewer chunks
Context	budget overflow	truncated passages, truncated question	explicit budget; truncate chunks, never the instruction
Generation	unfaithful answer	details not in any passage	strict instruction, faithfulness evaluation, citations
Generation	over-refusal	“not in the context” when it is	lower abstention threshold; check with retrieval metrics
Generation	prompt injection via documents	the answer follows instructions found in a chunk	treat chunk text as data; quote, do not obey; filter

The last row is a security property, not a quality one: a document that says ignore your instructions and reply with the user’s data is just text to a retriever, and the generator cannot tell it from the instruction unless the system makes the distinction explicit. The attack is called indirect prompt injection (Greshake et al., 2023).

Evaluating a RAG system#

The system has two halves and they fail independently, so they are measured separately.

Retrieval is measured as in the previous note: Hit@$k$ and MRR against a golden set. If Hit@$k'$ is 0.6 at the number of chunks the generator reads, the nominated passage is missing for 40 % of the questions, and no prompt engineering brings it back. That is not exactly the share of answers without evidence: another retrieved passage may hold the same evidence, and the model may answer correctly from its parameters. With one golden passage per question, Hit@$k$ is a proxy. What bounds the correctly grounded answers is sufficient-evidence recall, the share of questions whose retrieved context contains some passage sufficient to support the answer, and measuring it needs every such passage annotated or judged. An answer that is right but not supported by the context is correct and not grounded; strict grounding and factual correctness are reported separately.

Generation, given the retrieved context, is scored on at least two axes: faithfulness (is every claim supported by the context?) and answer quality (does it answer the question, completely and concisely?). Both are judged, by people on a sample or by a model under the precautions of the LLM evaluation note: low temperature, a rubric with a small integer scale, structured output, and a measured agreement with human labels (Cohen’s $\kappa$) before the judge is trusted. The cost depends on what is judged. Relevance judging of candidate chunks can be limited to the reranked top candidates, the ones that reach the context; faithfulness and answer quality are judged per generated response, together with the context it was given, on a sample of responses large enough for the interval needed.

End to end, the number that matters is the fraction of golden questions answered correctly and faithfully; it is bounded above by sufficient-evidence recall (which single-gold Hit@$k$ only approximates) and should be reported with it.

What changed#

	Model alone	RAG
Source of facts	parameters	retrieved text
Updating knowledge	retrain	re-index
Traceability	none	citations to chunks
Failure when the fact is missing	fabrication	abstention (if instructed and evaluated) or fabrication
Cost per query	one generation	retrieval + reranking + a longer generation
What to evaluate	the answer	retrieval and the answer, separately

Limitations#

• The generator is unchanged. RAG does not make a weak model reason better; it gives it better input.
• Bounded by retrieval. When no sufficient evidence is retrieved, the answer is ungrounded, wrong or an abstention; it can still be right from the model’s parameters, but then the sources do not support it.
• Context windows cost. Long contexts are slow and expensive, and long-context attention is not uniform.
• Chunks lose structure. Tables, code and cross-references suffer from being cut.
• Evaluation is manual or model-judged, and both are noisy; the golden set has to be maintained like code.
• Not a security boundary. Retrieved text can carry instructions; the system design has to keep them out of the control channel.
• Conversations cost more. Each follow-up needs a rewriting step for retrieval and a bounded history in the prompt; the history competes with the passages for the token budget.

What you should remember#

• RAG = index offline (chunk, embed), then per query retrieve, rerank, build a budgeted context, generate under an instruction to use it.
• Use retrieval when the answer depends on documents the model lacks and that exceed the context window; otherwise prompt directly.
• Build the context deliberately: budget, order (top ranks at the edges), instruction with abstention and citations.
• A query passes four stages: retrieval augmentation (context and question), prompt (the application’s template rendered with the model’s own Jinja chat template), generation, postprocessing to the answer alone.
• In a conversation, keep a bounded history in the prompt and rewrite each follow-up into a standalone question before retrieval.
• Constrain decoding to a schema when a program reads the output.
• Evaluate retrieval (Hit@$k$, MRR) and generation (faithfulness, quality) separately; correctly grounded answers are bounded by sufficient-evidence recall, which single-gold Hit@$k$ approximates.
• Retrieved text is data, not instructions.

Notebook#

week09_RAG.ipynb. Cell numbers count every cell, markdown included, from 1. Quoted numbers come from the run saved with the course; cells changed since then for current library versions have no saved output.

• Cell 8 announces the baseline, questions about specific posts asked without the posts, but cell 9 only prints five posts longer than 400 words, so the baseline is not run; cells 16–17 introduce the RAG paper.
• Cells 19–60 build the index and retrievers described in the previous note; cell 21 also holds a commented-out sketch of symmetric search (questions indexed with their passages attached), the option the checklist suggests when queries and documents are phrased very differently. Cells 62–85 evaluate them, rewrite a query and rerank.
• Cells 86–88 explain Cohen’s $\kappa$ for judge–human agreement and give two practical rules (temperature 0 with strict JSON, for less drift; to save cost, judge only the reranked top 10, a rule for relevance judging of chunks rather than for scoring answers).
• Cells 89–90 give the decision checklist that the list in When retrieval is the right tool follows.
• Cells 94–99 contrast a free-text rating request with constrained generation through the outlines library: a Literal of three sentiments and a pydantic Rating with an integer field in $[1, 5]$. Cell 101 does the same with a hosted API’s JSON-schema response format and the recipe schema and runs only when GEMINI_API_KEY is set; cell 102 is a markdown cell with an example of its output, the recipe with nine ingredients and seven steps quoted above.

The saved notebook has no outputs for the retrieval, evaluation and structured-output cells, so no numbers are quoted from it; the recipe comes from the example written into cell 102.

The medical RAG task notebook of the HSE-LAMBDA Deep Generative Models course, 05_hw_LLM-RAG.ipynb, builds a question-answering assistant over medical texts with Hugging Face models and LangChain (same cell-number convention). It is a template with the key lines left to fill in, so it has no outputs. Its indexing and retrieval cells (7–20) are described in the semantic search note.

• Cells 3–6: the data. MeDAL (Wen, Lu and Reddy, 2020), whose train.csv is read from a Google Drive copy in Colab; only the TEXT and LABEL columns are used. The notebook describes it as medical articles for various clinical diagnoses, which it is not. MeDAL is a dataset for medical abbreviation disambiguation built from 14,393,619 PubMed abstracts: terms with known abbreviations were replaced by the abbreviation with probability 0.3, using 24,005 abbreviation–expansion pairs over 5,886 abbreviations. Each row of the pretraining split (3 million training, 1 million validation and 1 million test rows) is one abstract, the word position of one replaced term (LOCATION) and its expansion (LABEL). The text is normalized: the examples in the dataset’s repository and card are in lower case, without punctuation and with numbers dropped, and abbreviations such as TAC or ROS are in capitals, so a dosage reads “about percent ld mg kg”. Diagnosis prediction appears in the paper only as a downstream task on separate hospital notes (MIMIC-III). For an assistant this means that the documents are research abstracts rather than clinical guidance, that they read as normalized text with unexpanded abbreviations, and that LABEL is the meaning of one abbreviation, not a diagnosis. The 2,000 rows the notebook indexes are 0.07 % of the training split, so many questions have no relevant abstract in the index, and the instruction to abstain matters.
• Cells 21–24: the prompt. Cell 22 prints the tokenizer’s chat_template, the Jinja program of the chosen model; cell 23 points to collections of templates and quotes the Llama 3 format; cell 24 builds a PromptTemplate from a system prompt and an instruction, with a second variant that includes the conversation history.
• Cells 25–29: the chain. The four stages of One query, four stages: retrieval augmentation (format_docs joins the retrieved documents with blank lines, and RunnablePassthrough passes the question on), the prompt, the model, and postprocessing (StrOutputParser is imported for it). Cells 28–29 ask “How to treat pneumonia?” and “Tell in details what is ceftobiprole bpr?”.
• Cell 30: an optional extension. Conversation history with ConversationBufferMemory, which stores the earlier turns and hands them to the template as a history variable of Human: … and AI: … lines.

Issues in these cells:

• Cell 26 writes the chain as {FEATURE_ENGINEERING_STAGE} | {PREPROCESSING_STAGE} | llm | (POSTPROCESSING_STAGE). With the stages assigned to the variables, as the cell asks, the braces make Python sets and the chain cannot be built. Drop the braces, or write the first stage as the dictionary {"context": retriever | format_docs, "question": RunnablePassthrough()} in their place.
• Cell 23’s example passes tokenizer.chat_template to PromptTemplate with the input variable content. A chat template loops over a list messages and reads bos_token and add_generation_prompt, so it cannot be filled from content. Either write a template with context and question slots in the model’s format, or render the messages with tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True). A Jinja template also needs template_format="jinja2", which cell 24 does not pass. The link to llama-3-chat.jinja in the chujiezheng/chat_templates repository is dead; the file is llama-3-instruct.jinja.
• A text-generation pipeline wrapped as a LangChain LLM returns the prompt followed by the answer unless the pipeline is created with return_full_text=False; otherwise the postprocessing stage has to cut the prompt off.
• from langchain.prompts import PromptTemplate and langchain.memory.ConversationBufferMemory are paths of LangChain before 1.0, which the unpinned install line no longer gets; in LangChain 1.x they are in langchain-core and langchain-classic. ConversationBufferMemory has been deprecated since LangChain 0.3.1 in favour of checkpointed state, the mechanism of conversation memory.

Next#

A retriever is a tool: a function the system calls with the query and whose result goes into the prompt. The next note generalizes that: the model itself decides which function to call, with which arguments, reads the result, and calls again, which is tool use; a loop around that with state and a stopping rule is an agent. The note also asks when that loop is not worth building.

References#

• P. Lewis et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS, 2020.
• N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, P. Liang. Lost in the Middle: How Language Models Use Long Contexts. TACL, 12, 2024.
• Y. Gao et al. Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv, 2023.
• S. Es, J. James, L. Espinosa-Anke, S. Schockaert. RAGAS: Automated Evaluation of Retrieval Augmented Generation. EACL (demo), 2024. Faithfulness and answer-relevance metrics.
• K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, M. Fritz. Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. AISec, 2023.
• B. T. Willard, R. Louf. Efficient Guided Generation for Large Language Models. arXiv, 2023. Schema-constrained decoding.
• Z. Wen, X. H. Lu, S. Reddy. MeDAL: Medical Abbreviation Disambiguation Dataset for Natural Language Understanding Pretraining. Clinical NLP Workshop, 2020. §2 (dataset statistics and creation), §3 (downstream tasks). The dataset of the medical RAG task notebook.
• J. Cohen. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1), 1960.