Tests that live with your code
Test cases, providers and expectations are declared in configuration files in your repository and run from the command line, so evaluation changes are reviewed like code.
SERVICES / OPEN SOURCE
Agent test suites with open-source Promptfoo.
Independent design and implementation of AI agent evaluations with the open-source Promptfoo CLI: tool-call and trajectory assertions, simulated multi-turn conversations and CI release gates.
Where open-source Promptfoo fits
Built for comparisons
Every run is a matrix of prompts, models and test cases, viewed side by side. Model migrations and prompt changes are compared on the same cases.
Self-run and free to adopt
The evaluation engine is open source under the MIT license. Commercial tiers exist, but the work on this page uses only the open-source tool.
Testing agents with Promptfoo
Tool calls and trajectories
Assertions check which tools the agent used, the arguments it passed, the order of calls and the number of steps, not only the final message.
Simulated multi-turn conversations
A simulated user drives realistic conversations over several turns, so clarifying questions, memory and handoffs are tested end to end.
Deterministic and model-graded checks
Exact checks such as valid JSON, required content, latency and cost sit alongside rubric-based, model-graded assertions for judgment calls.
How we help
Turn business rules into assertions
We translate acceptance criteria into specific assertions, so a passing test means the agent actually did the right thing.
Build reviewed test suites
Representative, difficult and failure-prone scenarios, with expected tool calls and outcomes reviewed by your domain experts.
Calibrate the graders
Model-graded assertions are pinned to a grader model and checked against expert labels before they are trusted as release gates.
Make it routine
CI release gates
Agreed suites run on pull requests or before release, with critical assertions failing the build.
Repeats, not one-offs
Key scenarios run several times with pass thresholds, so variable agent behavior is measured rather than missed.
Team enablement
We train your engineers to add cases, maintain assertions and read results as the agent evolves.
Related services
Using Promptfoo's commercial offering?
If your team uses Promptfoo's commercial platform, see our Promptfoo evaluation platform services.
Explore serviceEvaluation Assessment
A baseline for one Promptfoo agent or workflow, with prioritized findings and a release recommendation.
Explore serviceHallucination Testing
Measure how often your Promptfoo agent invents facts, policies or actions, and which fixes reduce it.
Explore serviceFrom scope to handover
Start with your inputs
Representative tasks, expected outcomes, repository access, model and agent access, and a domain reviewer.
Keep reusable assets
Versioned configurations, assertions, a baseline comparison, prioritized findings and a documented rerun process.
Support after handover
We can retest fixes, extend coverage to new Promptfoo workflows or move to recurring Managed Evaluations as your agent changes. Platform usage and licenses stay in your account.
Make your next decision with evidence.
Start with one workflow, clear success criteria, and an evaluation plan your team can use.