Retrieval-augmented generation — RAG — is the standard pattern for letting an AI answer questions over your own documents: fetch the relevant material, put it in the prompt, let the model answer from it with citations. It's conceptually simple, which is exactly why so many RAG systems disappoint. The naive version is easy to build and easy to get wrong. Here's where it breaks and how to fix it.

The first realisation: it's a retrieval problem

When a RAG system gives bad answers, the instinct is to blame the model or swap in a bigger one. That's almost always misdirected. The model can only answer from what it's given. If retrieval fetched the wrong chunks — or missed the chunk that contained the answer — no model, however capable, can recover. Garbage in, confident garbage out.

So the first principle of fixing RAG: diagnose retrieval before you touch generation. For a sample of failing questions, look at what was actually retrieved. Was the answer in there at all? Usually, for the failures, it wasn't. That tells you where to spend your effort.

Chunking: the underrated lever

How you split documents into retrievable chunks quietly determines everything downstream.

• Too large, and each chunk contains a lot of irrelevant text that dilutes the embedding and wastes context.
• Too small, and chunks lose the context needed to be meaningful — a sentence pulled out of its section may be useless or misleading.
• Split badly (mid-sentence, mid-table, across a logical boundary), and you fracture the very information you're trying to retrieve.

Good chunking respects document structure: split on sections, headings, and paragraphs rather than blindly every N characters. Keep related content together. Overlap chunks slightly so an answer that straddles a boundary isn't lost. This unglamorous work pays off more than almost anything else.

Embeddings and the limits of pure vector search

RAG typically retrieves by embedding — turning text into vectors and finding chunks whose vectors are nearest to the query's. This captures semantic similarity, which is powerful: it finds relevant text even when the wording differs.

But pure vector search has a known weakness: it can miss exact matches. Product codes, names, error strings, specific identifiers — the things keyword search nails — can slip through semantic search because they don't carry much semantic signal. The fix is hybrid search: combine vector search with traditional keyword search and merge the results. This single change rescues a large fraction of real-world retrieval failures, because real queries mix conceptual questions with specific terms.

The missing step: reranking

Here's the highest-leverage upgrade most RAG systems lack. Initial retrieval — vector or hybrid — is fast but coarse; it casts a wide net and gets the roughly relevant chunks. A reranker is a second, more precise model that takes those candidates and reorders them by true relevance to the query, so the best chunks land at the top of what you feed the LLM.

The pattern is: retrieve maybe 20–50 candidates quickly, rerank them, keep the top handful. It adds a step and a little latency, and it dramatically improves the quality of what reaches the model. Many teams skip it and wonder why answers are mediocre.

Evaluation: you can't improve what you don't measure

Most RAG systems have no evaluation, which means tuning is guesswork. Build a test set of real questions with known good answers, and measure two things separately:

• Retrieval quality — did the right chunks get fetched? (Measure this independently; it isolates the most common failure.)
• Answer quality — given good chunks, did the model answer well?

Splitting these tells you exactly where to invest. Usually it's retrieval.

A checklist for better RAG

• Chunk on document structure, with slight overlap, not blind fixed sizes.
• Use hybrid (keyword + vector) search, not vector alone.
• Add a reranking step.
• Build a retrieval-quality eval and a separate answer-quality eval.
• Include citations so users — and you — can verify where answers came from.
• Diagnose retrieval first when quality is poor; the model is rarely the bottleneck.

RAG isn't magic and it isn't a model problem. It's an information-retrieval engineering problem wearing an LLM hat. The teams whose RAG actually works are the ones who treat it that way.