“Add RAG” has become the default answer to every LLM accuracy problem, usually explained with a diagram of three boxes and an arrow. That diagram hides all the engineering. Here is what a retrieval-augmented generation system actually does, stage by stage — and where each stage quietly fails.
The problem RAG solves
An LLM only knows two things: what it learned in training, and what’s in the current context window. Your company’s documents are in neither. RAG bridges that gap with a simple contract: before the model answers, go find the passages most relevant to the question and paste them into the prompt. The model doesn’t “learn” your data — it reads a fresh, tiny slice of it on every request.
Everything that makes RAG work — or fail — lives in how well that slice is chosen.
Stage 1: Ingestion and chunking
Documents can’t be retrieved whole; a 60-page policy PDF doesn’t fit usefully into a prompt. So ingestion splits documents into chunks — and chunking is the most underrated decision in the whole pipeline.
- Fixed-size chunks (say, 500 tokens with 50-token overlap) are the default everywhere and the cause of half of all bad RAG answers. They cheerfully cut tables in half and separate a heading from the paragraph that explains it.
- Structure-aware chunking splits on real boundaries — sections, headings, table rows — so each chunk is a self-contained thought. It requires actual parsing work (PDFs resist this hard), which is exactly why cheap builds skip it.
- Chunk size is a trade-off: small chunks retrieve precisely but lose surrounding context; large chunks carry context but dilute the embedding. A common production pattern is retrieve small, expand big — match on a focused chunk, then hand the model the whole parent section.
If your RAG system gives confident wrong answers, look here first. Retrieval can only ever be as good as the chunks it has to choose from.
Stage 2: Embeddings and the vector index
Each chunk is run through an embedding model, which outputs a vector — a list of a few hundred to a few thousand numbers positioned so that texts with similar meaning land near each other. “How do I reset my password?” and “Steps to recover account access” end up close together despite sharing almost no words. That’s the entire magic: semantic similarity becomes geometry.
The vectors go into a vector index that can answer “which stored vectors are nearest to this query vector?” in milliseconds, even across millions of chunks (typically via approximate nearest-neighbor algorithms like HNSW — trading a sliver of accuracy for enormous speed).
Two practical notes that outrank any database choice:
- The embedding model matters more than the vector database. A better embedding model improves every single retrieval; the database mostly changes cost and latency.
- Pure semantic search has blind spots — exact part numbers, error codes, names. Production systems almost always run hybrid retrieval: vector similarity plus old-fashioned keyword (BM25) search, merged. Semantic search finds meaning; keyword search finds
ERR_4012.
Stage 3: Retrieval, reranking, generation
At query time: embed the question, pull the top-k candidate chunks (say 20), then — in better systems — pass them through a reranker, a model that reads the actual question-chunk pairs and reorders them by true relevance. Rerankers are slower than vector math but much smarter, which is why the pattern is retrieve wide with the fast method, rerank narrow with the smart one.
The winning chunks are formatted into the prompt with instructions to answer from the provided context and to say so when the context doesn’t contain the answer. That last instruction is your main defense against hallucination — and it only works if retrieval actually delivered the right passages. Most “hallucinations” in RAG systems are really retrieval failures: the model was never shown the truth, so it improvised. Debug retrieval before you blame generation.
Where RAG pipelines break in production
- Stale indexes. The docs changed; the vectors didn’t. Without an ingestion pipeline that re-embeds on change, your system confidently serves last quarter’s policy.
- Contradictory sources. Retrieval happily returns the 2023 version and the 2026 version of the same rule. The model averages them. Version-aware ingestion and metadata filters (date, department, product line) are unglamorous and essential.
- Nobody measures retrieval. Teams eyeball final answers instead of asking the measurable question: for these 100 real queries, did the right chunk appear in the top results? That number — retrieval recall — is the single most diagnostic metric in the system.
Do you even need the pipeline?
A genuinely useful recent development: context windows have grown enough that for small corpora — a few hundred pages — you can skip retrieval entirely and put everything in the prompt, or lean on provider-side file search that handles chunking and retrieval for you. Think of it as the folder versus pipeline decision: a folder you hand to the model wholesale, versus a pipeline you engineer and control.
The folder approach wins when the corpus is small, changes rarely, and per-request cost isn’t critical. The pipeline wins when you have real volume (paying to re-read 500 pages on every request adds up fast), need access control over who can retrieve what, need freshness guarantees, or need to cite exactly which passage backed each claim. Most serious internal-knowledge systems still end up with the pipeline — but starting with the folder is a legitimate way to prove value in a week.
For the adjacent decision — when retrieval beats fine-tuning and when it doesn’t — we’ve written a separate deep dive: RAG vs fine-tuning.
The honest summary
RAG is not a checkbox; it’s an information-retrieval system with an LLM at the end, and it inherits fifty years of search-engineering lessons: garbage chunks in, garbage answers out; measure recall, not vibes; keep the index fresh. Get those right and RAG is the most reliable way we know to make an LLM an expert in your material.
Building a knowledge assistant over your own documents? That’s a large part of what we build — and the first thing we’ll look at is your chunking.