Module 2 of 6 · Expert · 10 min read
Retrieval-augmented generation
Grounding models in your data: chunking, embeddings, hybrid search, reranking and evaluation.
- 1RAG retrieves relevant passages and puts them in the prompt.
- 2Most RAG failures are retrieval failures, not generation failures.
- 3Hybrid search plus reranking is a strong default.
01The pipeline
A typical RAG system has two halves: indexing and querying.
- Index: split documents into chunks, embed each chunk, store vectors and text.
- Query: embed the question, retrieve similar chunks (plus keyword matches), rerank, and pass the top results to the model.
- Generate: the model answers using the retrieved passages, ideally citing them.
02Where it breaks
If the right passage is not retrieved, no model can answer correctly. Common causes: chunks that split key information, embeddings that miss exact terms (product codes, names), and queries phrased differently from documents.
Fixes include contextual chunk headers, hybrid BM25 plus vector search, query rewriting and a reranker model.
03RAG vs long context
With million-token windows, you can sometimes skip retrieval and paste whole documents. That is simpler and often more accurate for a few documents, but slower and costlier at scale.
Agentic search, where the model issues its own search queries iteratively, increasingly replaces fixed retrieval pipelines for complex questions.
Finished reading?
Mark it complete to track your progress, then test yourself with the LLM Internals quiz.