AI Engineering: Building Production LLM Applications · Retrieval-Augmented Generation

RAG Failure Modes and Evaluation

~12 min

By the end of this lesson

Diagnose whether a RAG system's wrong answer traces back to a retrieval failure or a generation failure.
Apply context precision, context recall, and faithfulness to separately measure a RAG pipeline's retrieval and generation quality.

A RAG system that gives a wrong answer has at least two distinct places the failure could live, and conflating them wastes debugging time. A retrieval failure means the chunks that actually answer the question were never returned — wrong query embedding, a chunking decision that split the answer across boundaries, a corpus that never contained the answer in the first place. A generation failure means the right chunks were retrieved, and the model still got it wrong — ignoring the context, contradicting it, or adding something not actually present in what was retrieved.

The discipline this earns you: before touching a prompt, look at what was actually retrieved for the failing query. If the answer genuinely wasn't in the retrieved chunks, no amount of prompt engineering fixes it — that's a retrieval problem, and the fix lives in chunking, embeddings, or the corpus itself. If the answer was sitting right there in the retrieved context and the model still missed it, that's a generation problem, and it's the prompt or the model choice that needs attention.

Two metrics, both documented by the open-source Ragas evaluation library, separate retrieval quality into its two natural halves. Context recall asks: of everything relevant that exists, how much did retrieval actually find? It's calculated by breaking a reference answer into individual claims and checking what fraction of those claims can be traced back to the retrieved context — a reference answer like "the Eiffel Tower is located in Paris" scores full context recall only if the retrieved chunks actually support that claim, not just mention the Eiffel Tower.

Context precision asks the complementary question: of what was retrieved, how much was actually useful, and was it ranked near the top? Ragas computes it as an average of precision at each rank, so a relevant chunk buried at position five scores worse than the same chunk at position one — placement matters, not just presence. A system can have perfect recall (nothing relevant was missed) and poor precision (it's buried in noise), or the reverse, and the two numbers point to different fixes.

python

from ragas.metrics.collections import ContextRecall, ContextPrecision

recall_score = await ContextRecall(llm=judge_llm).ascore(
    user_input="Where is the Eiffel Tower located?",
    retrieved_contexts=retrieved_chunks,
    reference="The Eiffel Tower is located in Paris.",
)

precision_score = await ContextPrecision(llm=judge_llm).ascore(
    user_input="Where is the Eiffel Tower located?",
    retrieved_contexts=retrieved_chunks,
    reference="The Eiffel Tower is located in Paris.",
)

Recall and precision are scored separately, against the same reference answer.

Once you're confident retrieval found the right material, faithfulness measures the generation side: whether every claim in the model's response can actually be supported by the retrieved context, not just whether the response sounds plausible. Ragas calculates it by breaking the generated answer into individual statements and checking each one against the retrieved context — the score is the fraction of statements that check out. A response claiming "Einstein was born in Germany on 20th March 1879" against context that says 14th March splits into two statements, one supported ("born in Germany") and one not ("20th March"), for a faithfulness score of 0.5.

Faithfulness catches a specific and common failure: a model that has the right context in front of it and still confidently states a detail that isn't actually in there — a small, plausible-sounding error that's easy to miss reading the response alone, and exactly the kind of thing a claim-by-claim check is built to catch.

A RAG system answers a question confidently and fluently, but the answer is factually wrong. You check the logs: the retrieved chunks did contain the correct information. Where does the problem most likely live — retrieval or generation?Answer it yourself first, then open this.

Generation. Retrieval did its job — the right information was returned — so the failure is in how the model used (or ignored) that context. Check faithfulness next, and look at whether the prompt makes clear the model should rely on the retrieved context over its own prior knowledge.