RAG: what it is and how it actually works

Why RAG exists

An LLM stores knowledge in its weights. Lewis et al., who coined the term in 2020, call this parametric memory. It has three problems for real applications:

1. It’s frozen. The model knows nothing after its training cutoff, and nothing about your private data.
2. It can’t show its work. There’s no pointer from an answer back to a source, so you can’t verify it.
3. It’s expensive to change. Teaching a model new facts by fine-tuning is slow, costly, and unreliable for precise facts like “API keys rotate every 90 days.”

RAG adds non-parametric memory: an external index you can update in seconds, search at query time, and cite. The model’s job shifts from knowing the answer to reading the answer from the passages you hand it.

The two pipelines

A RAG system is really two systems that share an index. One runs offline whenever documents change; the other runs on every question.

flowchart TD
subgraph IDX [Indexing: offline, when documents change]
direction TB
S[Sources: docs, wikis, tickets, code] --> P[Parse and clean]
P --> C[Split into chunks]
C --> E1[Embed each chunk]
E1 --> V[(Vector index + metadata)]
C --> K[(Keyword index: BM25)]
end
subgraph QRY [Answering: online, every question]
direction TB
Q[Question] --> E2[Embed the question]
E2 --> R[Retrieve top-k candidates]
R --> RR[Rerank, keep the best few]
RR --> A[Build prompt with numbered sources]
A --> G[LLM writes a cited answer]
end
V -.-> R
K -.-> R

Indexing, step by step

1. Parse and clean. Turn PDFs, HTML, Markdown, and code into plain text while keeping structure you’ll want later: headings, page numbers, file paths, URLs. Garbage in here (navigation menus, repeated footers, broken tables) becomes garbage retrieved later.
2. Chunk. Split each document into passages small enough to embed well and to fit several into a prompt, typically a few hundred tokens. Chunking has its own note: embeddings and semantic chunking, under the hood.
3. Embed. Run each chunk through an embedding model, which maps text to a fixed-length vector (768 numbers in the example below) such that texts with similar meaning land close together.
4. Index. Store each vector with its text and metadata (source, section, date, access rights) in a vector index built for fast nearest-neighbor search. Many systems also build a keyword index (BM25) over the same chunks, because embeddings are weak at exact matches like error codes and function names.

Answering, step by step

1. Embed the question with the same embedding model used for the chunks.
2. Retrieve the top-k chunks (often 20 to 50) whose vectors are closest to the question’s vector, optionally merged with the top keyword matches. This step favors recall: cast a wide net.
3. Rerank. Score each candidate against the question with a more expensive model, typically a cross-encoder that reads question and chunk together, and keep the best few (3 to 8). This step favors precision.
4. Assemble the prompt: instructions, the chosen chunks labeled with their sources, and the question.
5. Generate. The LLM answers from the provided sources and cites them, or says the sources don’t cover it.

A worked example, with real numbers

Five tiny “documents” from an internal handbook:

The question: “How often do I have to change my API key?” Note that it says change, not rotate, and never mentions 90 days. Embedding everything with nomic-embed-text-v1.5 and scoring by cosine similarity gives:

Three things to notice:

• Retrieval worked on meaning, not words. “Change my API key” matched “keys are rotated” with no shared verb.
• The runner-up shares words, not meaning. sts.md mentions API keys but answers a different question. With k = 2, it would go into the prompt too, and the model has to ignore it. That’s what reranking and good prompts are for.
• Scores aren’t probabilities. Even the expense policy scores 0.49. Similarity scores are only meaningful relative to each other for one model; a fixed cutoff like “ignore anything under 0.5” doesn’t transfer between models or domains.

With k = 2, the prompt the model receives looks like this:

Answer using only the sources below. Cite them like [keys.md].
If they don't contain the answer, say so.

[keys.md]
API keys are rotated every 90 days from Settings > Security > Keys. The old key keeps working for 24 hours
after you rotate it.

[sts.md]
Service accounts don't use API keys. They authenticate with short-lived tokens issued by the internal STS.

Question: How often do I have to change my API key?

A good answer: “Every 90 days, from Settings > Security > Keys; the old key keeps working for 24 hours after you rotate it [keys.md].”

sequenceDiagram
participant U as User
participant App
participant E as Embedding model
participant I as Vector index
participant L as LLM
U->>App: How often do I have to change my API key?
App->>E: embed the question
E-->>App: 768-dim vector
App->>I: top-2 nearest chunks
I-->>App: keys.md 0.854, sts.md 0.682
App->>L: instructions + 2 sources + question
L-->>App: Every 90 days ... [keys.md]
App-->>U: answer with citation

The whole thing in about 30 lines

No framework beyond the model clients: a matrix of normalized vectors, a dot product, and a prompt.

import numpy as np
from langchain.chat_models import init_chat_model
from langchain.embeddings import init_embeddings

embedder = init_embeddings("<provider>:<embedding-model>")
llm = init_chat_model("<provider>:<chat-model>", temperature=0)

docs = {
    "keys.md": "API keys are rotated every 90 days from Settings > Security > Keys. ...",
    "sts.md": "Service accounts don't use API keys. ...",
    # ...
}

# Indexing: one vector per chunk, normalized so a dot product is the cosine similarity.
names = list(docs)
matrix = np.array(embedder.embed_documents([docs[n] for n in names]))
matrix /= np.linalg.norm(matrix, axis=1, keepdims=True)


def retrieve(question: str, k: int = 2) -> list[str]:
    q = np.array(embedder.embed_query(question))
    q /= np.linalg.norm(q)
    scores = matrix @ q  # cosine similarity with every chunk at once
    return [names[i] for i in np.argsort(-scores)[:k]]


def answer(question: str) -> str:
    context = "\n\n".join(f"[{name}]\n{docs[name]}" for name in retrieve(question))
    prompt = (
        "Answer using only the sources below. Cite them like [keys.md].\n"
        "If they don't contain the answer, say so.\n\n"
        f"{context}\n\nQuestion: {question}"
    )
    return llm.invoke(prompt).content

Production systems replace each line with something sturdier (a real chunker, an ANN index instead of matrix @ q, hybrid search, a reranker, metadata filters, evaluation), but the shape stays exactly this.

RAG vs. long context vs. fine-tuning

They combine well: fine-tune for tone and output format, use RAG for facts, and use a long context window to fit more retrieved passages.

Where RAG breaks

How to tell if it’s working

Evaluate the two halves separately, or you won’t know which one to fix:

• Retrieval: for a set of real questions with known answer passages, measure recall@k (is the right chunk in the top k?) and MRR (how high is it ranked?). Most bad RAG answers are retrieval failures.
• Generation: given the retrieved passages, is the answer faithful (supported by them) and relevant (does it answer the question)? Frameworks like RAGAS automate these with LLM judges; spot-check them by hand.

Variations you’ll hear about

• Hybrid search: combine BM25 and vector results, usually with reciprocal rank fusion.
• Query rewriting / HyDE: have an LLM rewrite the question, or write a hypothetical answer and search with its embedding, which often sits closer to real answer passages than the question does.
• Agentic RAG: the model decides whether, what, and how often to retrieve, as a tool call or a graph node. See State vs. Memory for a LangGraph version.
• GraphRAG: build an entity graph over the corpus for multi-hop and “summarize everything” questions.
• Contextual retrieval: prepend a short, document-aware summary to each chunk before embedding it.

References