RAG, Explained Without the Diagram Everyone Copies

Retrieval-augmented generation sounds like an architecture. It is closer to a workaround, and understanding what it is working around tells you most of what you need to know about when it helps.

The problem it solves

A language model knows what was in its training data. It does not know your company's internal documentation, yesterday's support tickets, or the contents of the PDF you just uploaded. You have two options: change the model, or change the question.

Changing the model — fine-tuning — is expensive, slow to iterate, and surprisingly bad at teaching facts. Fine-tuning is good at teaching style and format, and mediocre at teaching content.

Changing the question is cheap. Instead of asking "what is our refund policy?", you find the relevant paragraph of your policy document first and ask "given this text, what is our refund policy?" That is the entire idea. Everything else is engineering around the word find.

The pipeline, briefly

1. Chunk your documents into pieces of a few hundred words.
2. Embed each chunk — convert it to a vector of numbers that captures roughly what it means.
3. Store those vectors in a database that can find nearest neighbours quickly.
4. At query time, embed the question the same way and retrieve the closest chunks.
5. Stuff those chunks into the prompt and ask the model to answer using them.

Every tutorial draws this. It works on a demo in an afternoon. Then it meets real documents.

Where it actually breaks

Chunking destroys context. Split a document at 500 tokens and you will cut tables in half, orphan headings from their content, and produce chunks that begin with "This means that…" with no antecedent. The retrieved text is technically relevant and practically useless.

The fix is structural chunking, not size-based: split on headings, keep the heading path as a prefix on every chunk, never split a table or code block. This single change tends to improve real-world quality more than swapping in a better model.

Semantic search misses exact terms. Embeddings capture meaning, which means they are weak precisely where you need literal matching — error codes, product SKUs, function names, version numbers. A user searching for ERR_TOO_MANY_REDIRECTS wants that exact string, and the embedding of that string is uncomfortably close to the embedding of every other error code.

The fix is hybrid search: run both a keyword search (BM25) and a vector search, then merge the results. Reciprocal rank fusion is about fifteen lines of code and is the standard answer.

def reciprocal_rank_fusion(rankings, k=60):
    """Merge several ranked lists. Documents ranked highly by more than
    one retriever rise to the top."""
    scores = {}
    for ranking in rankings:
        for position, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + position + 1)
    return sorted(scores, key=scores.get, reverse=True)

The top 5 results are not the best 5. Vector similarity is a rough filter, not a ranking. Retrieve 50 candidates cheaply, then use a cross-encoder reranker to score each one against the query properly and keep the best 5. Rerankers are small, fast, and consistently the highest-impact addition after hybrid search.

Questions do not look like answers. "How do I reset my password?" and the document text "Navigate to Settings → Security and select Reset" have quite different embeddings. Several fixes exist — generating hypothetical answers to search with, or embedding an LLM-written summary alongside each chunk — and all of them help more than people expect.

No answer is an answer. If the retrieval finds nothing relevant, the model will still produce something confident. Your prompt must explicitly permit "I don't know," and your retrieval must have a relevance floor below which you return nothing at all. This is the difference between a system people trust and one they stop using after it invents a policy.

How to tell if it is working

Do not evaluate the whole system at first. Evaluate retrieval alone, because retrieval is where nearly all the failures live.

Build a set of 50–100 real questions with the document that should be retrieved for each. Then measure recall@k: how often is the correct document in the top k results? If recall@10 is below 90%, no amount of prompt engineering downstream will save you — the model simply never sees the answer.

This evaluation set is tedious to build and it is the highest-value artefact in the entire project. Teams that skip it end up tuning prompts for weeks against a retrieval bug.

Do you even need it?

Three honest alternatives, all cheaper:

Just put everything in the prompt. Context windows are large now. If your entire corpus is under a few hundred thousand tokens — a product manual, a policy handbook, a codebase's documentation — skip the pipeline entirely. Paste it all in. This is dramatically simpler and often more accurate, and it is the right answer far more often than the ecosystem admits.

Use ordinary search. If your users know the vocabulary of your domain, a well-tuned keyword search with a good UI beats a mediocre RAG system and never invents anything.

Fetch by ID. A surprising number of "RAG" problems are really "look up this specific record" problems. If the user is asking about order #4471, do not do a semantic search — query the database. Give the model a tool that fetches, rather than a vector store that guesses.

The summary

RAG is a way of getting relevant text in front of a model at the moment it needs it. The generation half is mostly solved. The retrieval half is a search engineering problem that has been studied for thirty years, and the teams who treat it that way — hybrid retrieval, reranking, structural chunking, a real evaluation set — end up with systems that work. The teams who treat it as a vector database purchase do not.