Evaluations that run like tests
Test cases and metric thresholds are written in Python and run like unit tests, so evaluations fit naturally into your existing test workflow and CI.
SERVICES / OPEN SOURCE
Agent evaluations as tests with open-source DeepEval.
Independent design and implementation of AI agent evaluations with the open-source DeepEval framework: task completion, tool correctness, grounding and multi-turn metrics in a test-style workflow.
Where open-source DeepEval fits
A broad agent metric library
Ready-made metrics cover task completion, tool and argument correctness, plan adherence, grounding and multi-turn conversations.
Self-run and open source
The framework is open source under the Apache-2.0 license. Confident AI offers a commercial platform on top; the work on this page uses only the open-source framework.
Testing agents with DeepEval
Whole-run and step-level checks
Trace your agent's functions to evaluate the full run for task completion and individual steps for tool choice and arguments.
Grounding and retrieval
Faithfulness, answer relevancy and contextual precision and recall metrics show whether answers stay true to retrieved content.
Multi-turn conversations
Conversation metrics check completeness, knowledge retention and role adherence across turns, with simulated conversations where useful.
How we help
Choose metrics that match the workflow
We start from your acceptance criteria and pick or define metrics for them, rather than adopting every metric available.
Custom criteria where needed
Business-specific rules become custom rubric metrics with clear definitions, examples and thresholds.
Calibrate the judges
Most metrics use an LLM judge. We pin the judge model, compare scores with expert labels and set thresholds that reflect real quality.
Make it routine
Reviewed test cases
Test sets built from approved examples and your documents, with generated cases reviewed before they are trusted.
CI regression suites
Agreed suites run on every change, with critical metrics failing the build and judge cost tracked.
Team enablement
We train engineers to add cases, maintain metrics and interpret scores and their explanations.
Related services
Using Confident AI's platform?
If your team uses Confident AI's commercial platform for shared results, see our Confident AI evaluation platform services.
Explore serviceEvaluation Assessment
A baseline for one DeepEval agent or workflow, with prioritized findings and a release recommendation.
Explore serviceHallucination Testing
Measure how often your DeepEval agent invents facts, policies or actions, and which fixes reduce it.
Explore serviceFrom scope to handover
Start with your inputs
Representative tasks, expected outcomes, code and agent access, approved documents and a domain reviewer.
Keep reusable assets
Versioned test cases, metric definitions, calibration results, a baseline report, prioritized findings and a documented rerun process.
Support after handover
We can retest fixes, extend coverage to new DeepEval workflows or move to recurring Managed Evaluations as your agent changes. Platform usage and licenses stay in your account.
Make your next decision with evidence.
Start with one workflow, clear success criteria, and an evaluation plan your team can use.