AI Engineering: Building Production LLM Applications · Retrieval-Augmented Generation

Chunking and the Retrieval Pipeline

~13 min

By the end of this lesson

Split a document corpus into chunks that keep enough context to be independently retrievable and independently useful.
Explain how Anthropic's Contextual Retrieval technique addresses context loss at chunk boundaries, and what it costs to run.

A full document is usually too big to embed as one meaningful vector — a whole contract or a whole wiki page blurs into an average that isn't close to any specific question about it — and too big to paste wholesale into every prompt. Chunking splits a corpus into smaller, independently retrievable pieces before anything gets embedded, so a query can be matched against the specific passage that actually answers it rather than the document as a whole.

The tension chunking has to balance runs in both directions. Chunks that are too large dilute an embedding's specificity — the same problem as embedding a whole document, just at a smaller scale — and waste context budget on parts of the chunk the query didn't need. Chunks that are too small lose the surrounding context a reader would need to make sense of them: a sentence pulled out of a financial filing with no company name or date attached might be technically about the right topic and still useless on its own.

Ingestion happens once; retrieval happens on every query.

That second failure — a chunk that's on-topic but missing the context to be useful — is common enough that Anthropic built and published a specific fix for it, called Contextual Retrieval. Its own example: split a financial filing into chunks, and one chunk might read only "the company's revenue grew by 3% over the previous quarter." That sentence is topically relevant to a question about quarterly revenue growth, but on its own it doesn't say which company or which quarter — exactly the information a retrieval system needs attached to rank and return it correctly.

Contextual Retrieval's fix is to generate a short piece of situating context for each chunk — which document it's from, what period or entity it concerns — and prepend that context to the chunk before two separate steps: before the chunk is embedded (Anthropic calls this Contextual Embeddings) and before the chunk is indexed for keyword search (Contextual BM25). The chunk stored and retrieved is the original; only what gets embedded and indexed carries the extra situating text.

python

# One chunk, with and without situating context
chunk = "The company's revenue grew by 3% over the previous quarter."

contextualized_chunk = (
    "This chunk is from ACME Corp's Q2 2023 SEC filing. " + chunk
)

# Embed and index contextualized_chunk; store and eventually
# return the original chunk to the model as retrieved context.

Only the version sent to the embedder and the keyword index changes; the stored chunk stays the same.

Anthropic's own measurements, across a mix of codebases, papers, and fiction, report that Contextual Embeddings alone reduced the top-20-chunk retrieval failure rate by 35% relative to a baseline chunking pipeline; combining Contextual Embeddings with Contextual BM25 reduced it by 49%; and adding a reranking step on top reduced it by 67%. These are relative reductions against Anthropic's own baseline and dataset, not a guarantee of the same improvement on every corpus — but the direction and rough size of the effect is a useful prior when deciding whether the extra step is worth it.

The extra step does cost something: generating situating context means an additional model call per chunk at ingestion time. Anthropic's guidance is that prompt caching is what makes this practical at scale, since the whole source document can be cached once and reused across the many chunk-context-generation calls drawn from it, rather than re-processing the full document from scratch for every chunk.

A chunk from a product manual reads only "Press and hold for five seconds to reset." Retrieved on its own for the query "how do I reset my thermostat," what's missing that Contextual Retrieval's approach would try to fix?Answer it yourself first, then open this.

The chunk doesn't say which product or which button — the situating context (the document or section it came from) is exactly what Contextual Retrieval prepends before embedding, so the retrieval system has that information even though the stored chunk itself stays unchanged.