Chunking Strategies for RAG: Fixed Beats Semantic

Chunking strategies for RAG set the ceiling on retrieval recall before your embedding model ever gets a vote. Fixed-size chunking, recursive and token-based, with overlap, is still the more reliable production default. Semantic chunking is the cleaner idea on a whiteboard, and the one that underperforms it in the most rigorous benchmark run to date.

I have watched teams switch to semantic chunking expecting a recall bump and get a worse one instead. The demo looks fine, because retrieval still finds something close to the right passage. Generation is where it falls apart: the chunk is too small to answer from, and nobody notices until a real user asks a real question.

This is what the data shows, why semantic chunking fails quietly instead of loudly, and where contextual retrieval and late chunking pick up the slack that neither fixed nor semantic chunking closes on its own.

Key takeaways

If you read nothing else, read these.

• Chunking sets the recall ceiling before the embedding model runs. A wrong chunk boundary is a lost answer no downstream reranker can recover.
• Fixed-size chunking beat semantic chunking in the most rigorous head-to-head so far. Recursive 512-token splitting scored 69% document accuracy against 54% for semantic chunking on a 50-paper benchmark.
• Semantic chunking failed for a specific, measurable reason. Its similarity-based boundaries produced fragments averaging about 43 tokens, too small to give the model enough context to answer correctly.
• Contextual retrieval cut failed retrievals by up to 67% by prepending a short context summary to each chunk before embedding, not by changing how the text gets split.
• Late chunking skips the split-then-embed order entirely. It embeds the full document first and pools chunk embeddings afterward, which helps passages that lean on pronouns or earlier references.

What chunking strategies for RAG decide

Chunking decides what unit of text becomes retrievable in the first place. Split a document the wrong way and you get chunks that are semantically incomplete, or so large they bury the answer in noise the model has to sort through. Every downstream choice, which embedding model, which vector index, which reranker, operates on whatever a chunk turned out to be. None of them can retrieve a fact that landed in the wrong chunk, or got separated from the sentence that gave it meaning.

This is why I tell teams to fix chunking before they argue about embedding models. A better embedding model on badly chunked text still embeds badly chunked text, at greater expense. The chunk boundary sits upstream of everything else in the pipeline, which makes it the highest-leverage place to get right and the easiest place to get wrong quietly.

Chunking is decision one of six in my RAG production tutorial; this piece is the deep dive on that first decision, because every other decision in the pipeline inherits from it.

Fixed-size chunking: how recursive, token-based splitting works

Fixed-size chunking, more precisely recursive or token-based splitting, cuts a document into windows of a set token count, typically 256 to 1,024 tokens, and slides an overlap of 10 to 20% between consecutive chunks so a sentence that falls on a boundary still shows up whole in a neighboring chunk. Recursive splitters try to break on paragraph or sentence boundaries first and only fall back to a hard token cut when a section runs too long.

Picture a 40-page product manual. A token-based splitter with a 512-token window and a 50-token overlap turns it into roughly 60 overlapping chunks. A table heading and the value beneath it might still land in separate chunks if a hard cut falls between them; that is a real weakness. But the chunks that do stay whole carry enough surrounding text for the model to answer from, which is the property semantic chunking gives up to win a cleaner-sounding split.

Fixed-size chunking costs nothing extra to compute, since it never touches the embedding model until after the split. It behaves predictably at any corpus size, and 512-token windows with modest overlap are the configuration most retrieval frameworks ship as a default for a reason: they have been tested for longer than semantic splitting has existed as an option.

Getting the chunk right is necessary but not sufficient. The next lever many teams reach for is deciding when to retrieve at all, a generation-time question rather than an input-side one, which is what I cover in active retrieval augmented generation. That fix happens after chunking. This one has to happen first.

Semantic chunking: embedding-similarity boundaries, and where it's supposed to win

Semantic chunking replaces the fixed token count with a similarity check. It embeds each sentence, or a small sliding window of sentences, compares consecutive embeddings, and cuts the chunk boundary wherever similarity drops, on the theory that a topic shift is a better place to split than an arbitrary token count.

The pitch is real. A chunk boundary chosen by meaning should keep one idea together and let a new idea start its own chunk, instead of slicing a paragraph in half because a token counter hit its limit. On a whiteboard, that is a more defensible unit of retrieval than cutting every 512 tokens regardless of what the sentence is doing.

It also costs more to compute. Every candidate boundary needs an embedding-similarity comparison at index time, which is extra latency and extra spend before you have retrieved a single query.

The benchmark data on chunking strategies for RAG

The clearest test of these two approaches comes from Vectara's NAACL 2025 Findings paper, which measured document retrieval, evidence retrieval, and answer generation across a real 50-academic-paper benchmark, not a synthetic one. Recursive 512-token splitting scored 69% document accuracy. Semantic chunking scored 54%.

The gap traces to one measurable cause: the semantic splitter's fragments averaged about 43 tokens, far smaller than the fixed-size chunks. A 43-token fragment can retrieve cleanly and land exactly on-topic, and still starve the model of the surrounding sentences it needs to construct a correct answer. Semantic chunking optimized for topical coherence and lost the raw amount of context that generation depends on.

Contextual retrieval and late chunking fix lost context without smarter splitting

If semantic chunking is not the fix for lost context, what is? Two approaches attack the same boundary-loss problem without asking the splitter to be smarter.

Anthropic's contextual retrieval keeps ordinary fixed-size chunks and fixes what is missing from them instead of how they were cut. Before embedding each chunk, an LLM call prepends one to two sentences of context: what document this came from, what section, what it relates to, so the chunk carries meaning it would otherwise lose in isolation. Paired with contextual BM25, that cut failed top-20-chunk retrievals by 49%, from a 5.7% failure rate to 3.7%. Add a reranker on top and the reduction reaches 67%, down to a 1.9% failure rate.

That combination, dense embeddings plus a lexical signal like BM25, is the same reason I default to hybrid retrieval generally; I go deeper on when each lane wins in hybrid search vs. vector search.

Late chunking, from Jina AI's research, skips the split-then-embed order entirely. Instead of splitting text and embedding each piece alone, it runs the whole document through a long-context embedding model first, at the token level, and only pools tokens into chunk-level embeddings afterward. Every chunk embedding inherits context from the entire document it came from, with no extra model training required, which specifically helps passages that lean on a pronoun or a reference to something explained several paragraphs earlier.

Neither is free. Contextual retrieval adds an LLM call per chunk at index time, which is real spend on a large corpus. Late chunking depends on an embedding model that can hold long context, which rules out plenty of smaller, cheaper embedding models. You are trading one cost for a different one, index-time compute or a model-choice constraint, in exchange for fixing a failure mode that smarter splitting alone does not reach. Budget that index-time LLM call before you commit to the architecture, not after the first invoice; once a corpus runs into the tens of thousands of documents, it is a line item, not a rounding error.

Chunk size and overlap by query type: factoid vs. multi-hop

Chunk size and overlap should follow the shape of the questions your corpus gets asked, not a single default copied from a tutorial.

• Factoid queries ("what is X's price," "when did Y ship") retrieve well from smaller chunks, 200 to 400 tokens, because the answer usually lives in one or two sentences and a smaller chunk keeps the noise down.
• Multi-hop and analytical queries ("compare X and Y," "why did Z happen") need larger chunks, 512 to 1,000 tokens, or contextual retrieval layered on top of smaller ones, because the answer depends on connecting facts that a tiny chunk splits apart.
• Overlap of 10 to 20% recovers a sentence that would otherwise get cut at a chunk boundary. It does not fix a chunk that is fundamentally too small; it only softens where the cut lands.

A decision framework: which chunking strategy fits your corpus

None of these are permanent. Chunking is a design decision you should be able to re-run, not a preprocessing step you set once and forget. Version it and evaluate it the way you would version a model, because it changes exactly like one.

Is semantic chunking better than fixed-size chunking for RAG?

No, not on the best evidence available. In Vectara's NAACL 2025 benchmark, recursive fixed-size chunking scored 69% document accuracy against 54% for semantic chunking, because the semantic splitter's fragments averaged only about 43 tokens, too small to give the model enough context to answer from. Semantic chunking is not wrong in theory; it just underperforms in the most rigorous test run so far.

What chunk size should I use for a RAG pipeline?

Start at 512 tokens with 10 to 20% overlap for prose-heavy corpora; that is the configuration the strongest published benchmark used and the one most retrieval frameworks default to. Drop to 200 to 400 tokens for corpora dominated by short factoid answers, and lean on contextual retrieval rather than a bigger chunk when your queries are multi-hop or analytical.

What is late chunking and how is it different from semantic chunking?

Late chunking embeds the entire document first, at the token level, using a long-context embedding model, and only pools those token embeddings into chunks afterward, so every chunk carries the whole document's context. Semantic chunking does the opposite: it decides where to cut before anything gets embedded, based on similarity between consecutive sentences. Late chunking changes when you embed; semantic chunking changes where you cut.

Does chunk overlap actually improve retrieval accuracy?

Yes, for the specific failure it targets: a sentence or fact that falls right on a chunk boundary. A 10 to 20% overlap usually recovers that case. It does not fix a chunk that is too small in the first place, and it does not substitute for contextual retrieval or late chunking when the real problem is a document-level reference the chunk never had access to.

If you are building or auditing a RAG pipeline and want the chunking discipline that holds past the demo, metadata, versioning, and evals built into the split itself, I go deeper on all of it in my book Embeddings, Honestly. And if you would rather have a team wire chunking evals and contextual retrieval in from day one instead of discovering the 43-token problem in production, that is exactly what ViitorCloud's custom AI and RAG systems team builds.