A Retrieval-Augmented Generation Tutorial: 6 Decisions

A retrieval-augmented generation tutorial worth following treats RAG as six decisions, not one library import. You chunk, you embed with context, you index with both dense and lexical search, you rerank before you trust the top results, you assemble the prompt with a citation format, and you evaluate before you ship. Skip contextual embeddings and reranking and you leave the single biggest recall gains on the table: documented cuts of up to 67% in retrieval failures for the teams that do both.

Most RAG tutorials stop at "embed the chunks, query the vector store, stuff the results into the prompt." That version works in a demo against fifty documents. It falls over the day your corpus hits ten thousand documents and users start asking questions that need more than semantic similarity. This is the version I would ship.

Key takeaways

• RAG is a pipeline of six decisions: chunking, contextual embeddings, hybrid indexing, reranking, prompt assembly, and evaluation. Weak links anywhere sink the whole system.
• Contextual embeddings alone cut top-20 retrieval failures 35% (5.7% to 3.7%); adding contextual BM25 gets you to 49%; adding a reranker on top gets you to 67%, per Anthropic's published benchmark.
• Vector-only search misses exact-match queries (SKUs, error codes, names). Hybrid search that combines dense and lexical retrieval is now the production default, not an optional add-on.
• A reranker is a second, more expensive scoring pass. It is usually worth the latency; you should measure that, not assume it.
• Evaluate with the four-metric RAGAS pattern (faithfulness, answer relevance, context precision, context recall) before you ship, not after a customer complains.

What retrieval-augmented generation actually is

Retrieval-augmented generation is a technique that retrieves relevant text from an external knowledge base and inserts it into an LLM's prompt before generation, so the model answers from your data instead of only what it memorized during training. It exists because a model's training data goes stale, and fine-tuning a model on every document update does not scale. RAG keeps the model current by changing what it reads, not what it knows.

That definition covers the concept, not why most pipelines underperform in production. The gap between "RAG works" and "RAG is reliable" sits entirely in the six decisions below. For the deeper case on when static, single-pass retrieval stops being enough, see my pillar piece on active retrieval-augmented generation.

If you are deciding whether to build this yourself or bring in help that has done it before, that decision point is exactly where ViitorCloud's RAG and knowledge-integration build starts: audit which of the six decisions your pipeline makes on purpose, and which ones default.

Step 1: Chunk your documents

Bad chunking guarantees a bad pipeline no matter how good your embedding model is. If a chunk splits a definition from the sentence that explains it, no embedding on earth retrieves the full answer, because the full answer was never in one place to begin with.

Two chunking strategies cover most cases. Recursive chunking splits on natural boundaries, paragraph, then sentence, then word, until each chunk fits your target size, typically 300 to 800 tokens. Semantic chunking splits where the topic shifts, measured by embedding similarity between adjacent sentences, producing more coherent chunks at a higher preprocessing cost.

Overlap matters more than most people budget for it. A 10 to 20% overlap between adjacent chunks keeps a fact near a boundary from getting orphaned on one side of it. Also consider a small2big pattern: embed and search over small, precise chunks, but retrieve the larger parent section for the LLM to read. You search precisely and generate with full context instead of forcing one chunk size to do both jobs badly.

I worked through this on a retail knowledge base at ViitorCloud. Fixed 512-token windows retrieved cleanly on product-spec questions and fell apart on multi-step return-policy questions, because the policy's conditions and its exceptions landed in different chunks. Switching to small2big fixed it without touching the embedding model at all. I go deeper on sizing, overlap, and the small2big pattern in my full guide to chunking strategies for RAG.

Step 2: Generate contextual embeddings, not raw chunk embeddings

Here is the mistake nearly every RAG tutorial makes: it embeds the chunk exactly as it was split, with no context about where that chunk came from. A chunk that reads "the fee is waived after the first year" is nearly meaningless in isolation. Waived for what? Which plan? Which year?

Anthropic's contextual retrieval technique fixes this by having an LLM generate a short context summary, 50 to 100 tokens, describing what the chunk is about and where it sits in the document, then prepending that summary to the chunk before embedding it and before adding it to a lexical (BM25) index. The chunk goes from "the fee is waived after the first year" to something like "this chunk is from the Premium Plan pricing section and describes the annual maintenance fee: the fee is waived after the first year."

The published numbers are specific and worth internalizing. Contextual embeddings alone cut the top-20-chunk retrieval failure rate 35%, from 5.7% to 3.7%. Contextual embeddings plus contextual BM25 cut it 49%, to 2.9%. Adding a reranking pass on top of both cuts it 67%, to 1.9%, according to Anthropic's contextual retrieval writeup. That is the single largest, best-documented recall lever in RAG right now, and it is a preprocessing step, not an architecture change.

The cost is real, so name it now, not after you have built it. Contextual embeddings mean an LLM call per chunk at indexing time, which means every source-document change re-embeds the corpus with an LLM in the loop, not a cheap embedding model. On a large, frequently updated corpus, that is a recurring cost line, not a one-time setup step.

Step 3: Build a hybrid index, dense and lexical

Vector-only search is the second-most common RAG mistake, right behind raw-chunk embedding. Dense (vector) search finds semantic matches: it knows "cancel my plan" and "terminate my subscription" mean the same thing. It is genuinely bad at exact matches, a SKU, an error code, a name, because those tokens carry little semantic signal. A vector search happily returns something "close" instead of the one document with the exact string the user typed.

Lexical search (BM25) is the mirror image: it nails exact-token matches and struggles with paraphrase. Hybrid search runs both in parallel and merges the results, typically with a weighted score or reciprocal rank fusion, so you get semantic recall and exact-match precision in the same query. By 2026, hybrid search is a default requirement for production RAG, not an optional add-on. Four vector databases now account for the overwhelming share of that workload: Pinecone, Qdrant, Weaviate, and Postgres with pgvector, per Firecrawl's 2026 vector database comparison.

I cover the full decision tree, including when vector-only search is genuinely fine, in my comparison of hybrid search versus vector search. The short version: if any real query in your traffic includes an ID, a code, or a name, you need the lexical half of the index. That is most production systems.

Step 4: Add a reranker before you trust top-k

Retrieval and ranking are not the same job, and treating them as one is a top failure mode. Your vector or hybrid search returns the top-k candidates, ranked by a cheap similarity score computed independently per chunk. That score has never compared the candidates to each other or reasoned about the query in context. It is a fast, approximate first pass, and trusting its ranking as final is where a lot of "good retrieval, bad answer" cases come from.

A reranker fixes this with a second pass. A cross-encoder reranker, Cohere Rerank is the common managed option, a local cross-encoder model the self-hosted fallback, scores the query against each candidate chunk jointly, producing a materially more accurate ranking. Retrieve a wider net, top-50 say, then rerank down to the top-5 to 8 that go in the prompt.

The honest trade-off: reranking is a second scoring pass on every query, on top of the retrieval call you already paid for. That is added latency and cost per request, and it does not show up as a problem until query volume is real. An extra 100 to 300 milliseconds is invisible in a low-traffic demo. At production volume, it is a line item you have to defend.

Step 5: Assemble the prompt and call the model

By the time you reach prompt assembly, you have reranked chunks and a context window budget to spend them against. Three things matter here: budget the tokens, format citations so claims are traceable, and handle multi-hop queries as a retrieval problem, not a prompting problem.

Do not stuff every retrieved chunk into the window because the model technically supports a large context. Long context is not a substitute for good retrieval: more tokens dilute the signal the model attends over and, on some benchmarks, degrade accuracy. Budget the context window like the scarce resource it is.

Multi-hop queries, questions needing facts from two documents combined, are where single-pass RAG genuinely struggles. If a meaningful share of your traffic is multi-hop, that is a retrieval architecture decision, query decomposition, iterative retrieval, not something a cleverer system prompt fixes.

Step 6: Evaluate before you ship

Every step above is a lever you can get wrong without knowing it, because a pipeline that retrieves badly can still generate a plausible-sounding answer. The fix is to stop trusting the demo and start measuring retrieval and generation separately, on a frozen set you did not build to make yourself feel good.

RAGAS pioneered the four-metric pattern that remains the reference point most 2026 RAG eval tools build on, per the RAGAS evaluation writeup: faithfulness (is the answer grounded in the retrieved context, not invented), answer relevance (does it address the actual question), context precision (how much of what was retrieved was relevant), and context recall (did retrieval surface everything needed to answer). Each is scored by an LLM-as-judge against your rubric, not a hardcoded string match.

That trace is illustrative, not a real client run, but the shape is the one I see repeatedly: faithfulness and relevance look fine because the model writes fluent answers from whatever it gets, while context recall quietly fails because the reranker cutoff dropped a chunk the answer needed. Gate the deploy on recall, not on how confident the answer sounds. I lay out the full harness, targets, and cadence in my book on retrieval that survives contact with production.

Where retrieval-augmented generation breaks in production

Contextual embeddings and reranking are the two moves with the best documented recall payoff, and the two that cost the most once the corpus is real-sized. Reranking means every query pays for a second scoring pass. Contextual embeddings mean re-embedding the corpus with an LLM call per chunk every time source documents change. Neither shows up in a demo built on fifty static documents.

Both show up once your corpus and query volume are production-sized, and that is exactly when a stale, un-refreshed index starts quietly returning outdated chunks that still outrank the correct, newer ones. Nobody notices until a customer gets an answer built on a policy you changed three months ago. Schedule re-embedding on document change, not a calendar, and treat index freshness as a monitored metric, not an assumption.

Frequently asked questions

What is retrieval-augmented generation in simple terms?

RAG is a way of giving an LLM access to your own documents at answer time instead of only what it learned during training. The system retrieves the most relevant chunks of your data for a given question and inserts them into the prompt, so the model answers from your current information rather than a stale or generic memory.

How do I pick the right chunk size for a RAG pipeline?

Start with 300 to 800 tokens per chunk with 10 to 20% overlap, then test against real queries, not assumptions. If answers cut off mid-fact, your chunks are too small or your overlap is too thin. If retrieval returns irrelevant surrounding text, they are too large. A small2big pattern, searching over small chunks but returning the full parent section to the model, sidesteps most of this tuning entirely.

Do I need a reranker, or is vector search alone good enough?

If your queries are simple, single-fact lookups against a small corpus, vector search alone can be fine. The moment your top-k results are noisy, or a wrong-but-similar chunk keeps outranking the right one, a reranker is the fix, and Anthropic's benchmark shows it stacking with contextual embeddings for the largest documented recall gain. Measure the lift on your own eval set before you decide it is worth the added latency.

Is RAG better than fine-tuning for keeping an LLM up to date?

For most knowledge-freshness problems, yes. RAG updates what the model reads, which is as fast as updating a document. Fine-tuning updates what the model knows, which requires a retraining run every time the underlying facts change. I compare the two directly, including where fine-tuning still wins, in RAG versus fine-tuning.

If you are past the tutorial stage and need this built against a real corpus, evaluated properly, and priced honestly, that is the work behind ViitorCloud's RAG and knowledge-integration builds: chunking, contextual embeddings, hybrid indexing, and reranking, with the eval harness wired in from day one instead of bolted on after the first bad answer reaches a customer.