Retrieval-Augmented Generation

An embedding is a list of numbers — a vector — produced by a model trained to place text with similar meaning near each other in that numeric space. "A rich cup of coffee" and "a strong espresso" end up close together; "a rich cup of coffee" and "a tortoise moves slowly" end up far apart. Nothing about the vector is human-readable on its own; the only property that matters is its distance to other vectors.

That one property is the whole idea behind retrieval-augmented generation. Instead of searching for exact keyword matches, you turn "find the passages relevant to this question" into "find the vectors nearest to this question's vector" — a problem with fast, well-understood solutions at scale, in a way full-text semantic matching isn't. Embeddings also power search, clustering, recommendations, and classification more broadly, but retrieval for RAG is the use this module focuses on: embed a corpus once, embed each incoming query, and compare.

from openai import OpenAI

client = OpenAI()

response = client.embeddings.create(
    model="text-embedding-3-small",
    input="A rich cup of coffee.",
)
vector = response.data[0].embedding
print(len(vector))

OpenAI's third-generation embedding models are the most common starting point: text-embedding-3-small returns a 1536-dimension vector by default, text-embedding-3-large returns 3072, and both accept up to 8192 tokens of input per call. Both were also trained with a technique that lets you shrink the vector — the dimensions parameter — without destroying its concept-representing properties: OpenAI's own benchmark shows a text-embedding-3-large embedding shortened to 256 dimensions still outperforming an unshortened, older text-embedding-ada-002 embedding at 1536 dimensions. Smaller vectors mean less storage and faster comparisons, so this trade-off is worth using deliberately rather than defaulting to the largest size available.

If you shrink a vector yourself after the fact — slicing it down — rather than passing dimensions at request time, you have to re-normalize it to unit length afterward, or your distance comparisons will be off.

from openai import OpenAI
import math

client = OpenAI()

docs = ["A rich cup of coffee.", "A bright herbal tea.", "A tortoise moves slowly."]
query = "delicious espresso"

data = client.embeddings.create(model="text-embedding-3-small", input=docs + [query]).data
doc_vecs, query_vec = [d.embedding for d in data[:-1]], data[-1].embedding

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.hypot(*a) * math.hypot(*b))

ranked = sorted(zip(docs, doc_vecs), key=lambda pair: -cosine(pair[1], query_vec))
print([doc for doc, _ in ranked])

Cosine similarity — the angle between two vectors, ignoring their length — is the standard recommendation for comparing embeddings, and OpenAI's own guidance says the choice of distance function typically doesn't matter much beyond that. OpenAI's embeddings happen to already be normalized to unit length, which has two side effects worth knowing: cosine similarity can be computed as a slightly cheaper plain dot product, and cosine similarity and Euclidean distance produce identical rankings on these vectors. That equivalence is specific to normalized embeddings, though — it's not a universal property of vector search, so don't assume it holds for every embedding model you might use.

A full document is usually too big to embed as one meaningful vector — a whole contract or a whole wiki page blurs into an average that isn't close to any specific question about it — and too big to paste wholesale into every prompt. Chunking splits a corpus into smaller, independently retrievable pieces before anything gets embedded, so a query can be matched against the specific passage that actually answers it rather than the document as a whole.

The tension chunking has to balance runs in both directions. Chunks that are too large dilute an embedding's specificity — the same problem as embedding a whole document, just at a smaller scale — and waste context budget on parts of the chunk the query didn't need. Chunks that are too small lose the surrounding context a reader would need to make sense of them: a sentence pulled out of a financial filing with no company name or date attached might be technically about the right topic and still useless on its own.

That second failure — a chunk that's on-topic but missing the context to be useful — is common enough that Anthropic built and published a specific fix for it, called Contextual Retrieval. Its own example: split a financial filing into chunks, and one chunk might read only "the company's revenue grew by 3% over the previous quarter." That sentence is topically relevant to a question about quarterly revenue growth, but on its own it doesn't say which company or which quarter — exactly the information a retrieval system needs attached to rank and return it correctly.

Contextual Retrieval's fix is to generate a short piece of situating context for each chunk — which document it's from, what period or entity it concerns — and prepend that context to the chunk before two separate steps: before the chunk is embedded (Anthropic calls this Contextual Embeddings) and before the chunk is indexed for keyword search (Contextual BM25). The chunk stored and retrieved is the original; only what gets embedded and indexed carries the extra situating text.

# One chunk, with and without situating context
chunk = "The company's revenue grew by 3% over the previous quarter."

contextualized_chunk = (
    "This chunk is from ACME Corp's Q2 2023 SEC filing. " + chunk
)

# Embed and index contextualized_chunk; store and eventually
# return the original chunk to the model as retrieved context.

Anthropic's own measurements, across a mix of codebases, papers, and fiction, report that Contextual Embeddings alone reduced the top-20-chunk retrieval failure rate by 35% relative to a baseline chunking pipeline; combining Contextual Embeddings with Contextual BM25 reduced it by 49%; and adding a reranking step on top reduced it by 67%. These are relative reductions against Anthropic's own baseline and dataset, not a guarantee of the same improvement on every corpus — but the direction and rough size of the effect is a useful prior when deciding whether the extra step is worth it.

The extra step does cost something: generating situating context means an additional model call per chunk at ingestion time. Anthropic's guidance is that prompt caching is what makes this practical at scale, since the whole source document can be cached once and reused across the many chunk-context-generation calls drawn from it, rather than re-processing the full document from scratch for every chunk.

A RAG system that gives a wrong answer has at least two distinct places the failure could live, and conflating them wastes debugging time. A retrieval failure means the chunks that actually answer the question were never returned — wrong query embedding, a chunking decision that split the answer across boundaries, a corpus that never contained the answer in the first place. A generation failure means the right chunks were retrieved, and the model still got it wrong — ignoring the context, contradicting it, or adding something not actually present in what was retrieved.

The discipline this earns you: before touching a prompt, look at what was actually retrieved for the failing query. If the answer genuinely wasn't in the retrieved chunks, no amount of prompt engineering fixes it — that's a retrieval problem, and the fix lives in chunking, embeddings, or the corpus itself. If the answer was sitting right there in the retrieved context and the model still missed it, that's a generation problem, and it's the prompt or the model choice that needs attention.

Two metrics, both documented by the open-source Ragas evaluation library, separate retrieval quality into its two natural halves. Context recall asks: of everything relevant that exists, how much did retrieval actually find? It's calculated by breaking a reference answer into individual claims and checking what fraction of those claims can be traced back to the retrieved context — a reference answer like "the Eiffel Tower is located in Paris" scores full context recall only if the retrieved chunks actually support that claim, not just mention the Eiffel Tower.

Context precision asks the complementary question: of what was retrieved, how much was actually useful, and was it ranked near the top? Ragas computes it as an average of precision at each rank, so a relevant chunk buried at position five scores worse than the same chunk at position one — placement matters, not just presence. A system can have perfect recall (nothing relevant was missed) and poor precision (it's buried in noise), or the reverse, and the two numbers point to different fixes.

from ragas.metrics.collections import ContextRecall, ContextPrecision

recall_score = await ContextRecall(llm=judge_llm).ascore(
    user_input="Where is the Eiffel Tower located?",
    retrieved_contexts=retrieved_chunks,
    reference="The Eiffel Tower is located in Paris.",
)

precision_score = await ContextPrecision(llm=judge_llm).ascore(
    user_input="Where is the Eiffel Tower located?",
    retrieved_contexts=retrieved_chunks,
    reference="The Eiffel Tower is located in Paris.",
)

Once you're confident retrieval found the right material, faithfulness measures the generation side: whether every claim in the model's response can actually be supported by the retrieved context, not just whether the response sounds plausible. Ragas calculates it by breaking the generated answer into individual statements and checking each one against the retrieved context — the score is the fraction of statements that check out. A response claiming "Einstein was born in Germany on 20th March 1879" against context that says 14th March splits into two statements, one supported ("born in Germany") and one not ("20th March"), for a faithfulness score of 0.5.

Faithfulness catches a specific and common failure: a model that has the right context in front of it and still confidently states a detail that isn't actually in there — a small, plausible-sounding error that's easy to miss reading the response alone, and exactly the kind of thing a claim-by-claim check is built to catch.