Evaluating RAG Agents with DeepEval: Contextual Precision, Recall, and Relevancy

When a RAG agent gives a bad answer, is retrieval or generation to blame? DeepEval's RAG metrics help you find out, metric by metric.

A customer asks your knowledge assistant about warranty coverage. The answer is wrong. Was the right document never found? Found but ranked too low? Found, and then ignored or misread?

Each of those has a different fix, and guessing wastes weeks. RAG evaluation metrics let you separate them. This post explains DeepEval's five core RAG metrics in plain English, shows how to run them, and how to turn the results into a diagnosis.

Good model, bad retrieval

Retrieval-augmented generation (RAG) agents answer from your documents: they retrieve relevant passages, then generate an answer from them. Most teams evaluate only the final answer. When it's wrong, they tweak the prompt or swap the model, often when the real problem was retrieval all along.

The fix is to evaluate the two halves separately. Research on automated RAG evaluation, such as the RAGAS framework, popularized this split between retrieval quality and answer faithfulness. DeepEval provides metrics for both.

The five metrics in plain English

Retriever metrics

- Contextual precision: are the most relevant chunks ranked first? Agents and models pay more attention to what comes first; useful material buried at position eight may as well be missing.
- Contextual recall: did retrieval find everything the ideal answer needs? Low recall means the agent couldn't have answered fully even if it tried.
- Contextual relevancy: how much of what was retrieved is actually relevant? Low relevancy means noise that can distract the generator and raise costs.

Generator metrics

- Faithfulness: is every claim in the answer supported by the retrieved context? Low faithfulness means the agent is adding things, a form of hallucination.
- Answer relevancy: does the answer actually address the question?

Note the inputs. Precision and recall compare retrieval with an expected output, so you need reference answers. Relevancy only needs the question and the retrieved context.

Walkthrough

from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    ContextualPrecisionMetric,
    ContextualRecallMetric,
    ContextualRelevancyMetric,
    FaithfulnessMetric,
    AnswerRelevancyMetric,
)

question = "Does the warranty cover accidental damage?"
answer, retrieved_chunks = run_rag_agent(question)  # return both from your agent

case = LLMTestCase(
    input=question,
    actual_output=answer,
    expected_output="No. The standard warranty covers manufacturing defects for 12 months. "
                    "Accidental damage requires the optional protection plan.",
    retrieval_context=retrieved_chunks,
)

metrics = [
    ContextualPrecisionMetric(threshold=0.7),
    ContextualRecallMetric(threshold=0.7),
    ContextualRelevancyMetric(threshold=0.5),
    FaithfulnessMetric(threshold=0.9),
    AnswerRelevancyMetric(threshold=0.7),
]

evaluate(test_cases=[case], metrics=metrics)

Two practical points:

1. Make your agent return what it retrieved. You can't evaluate retrieval if you only capture the final answer.
2. Write reference answers with a domain expert. They're the yardstick for precision and recall.