Agents built in Joule Studio
Joule agents configured with instructions, context and execution steps in SAP Build, then tested in private environments before release.
SERVICES / ENTERPRISE SOFTWARE
Evaluate Joule agents against your business processes.
Independent testing and evaluation for AI agents built with SAP Joule and Joule Studio: skill selection, business-rule accuracy and grounded answers in your SAP processes.
Common Joule agent patterns
Skills as tools
Agents act through Joule skills, actions and connectors that read and change data in your SAP systems. Each skill call needs the right inputs.
Process-heavy decisions
Procurement, finance, supply chain and service tasks governed by detailed business rules. An answer that sounds right can still break a rule.
Quality issues we look for
Wrong agent or skill
Joule routes the request to the wrong agent or skill, often because agent descriptions overlap or are too vague.
Business-rule errors
Recommendations or actions that ignore stock, approval, tolerance or reservation rules your process depends on.
Mismatched parameters
Skill calls with missing or wrong values, such as the wrong plant, supplier or document number, that pass validation but produce the wrong result.
What we assess
Process outcomes
Whether each scenario reaches the intended business result, verified in a test environment against the resulting documents and records.
Skill selection and inputs
Whether the agent chooses the right skills in the right order and fills them with values grounded in the conversation and your data.
Grounding and role scope
Whether answers stay faithful to approved documents and data, and whether the agent respects the roles and authorizations it runs under.
Turning manual checks into evidence
Beyond interactive testing
Joule Studio's test chat and step timeline are useful for debugging. We turn those checks into a reviewed, repeatable test set with expected outcomes.
Generative AI hub evaluation
Where it fits, we use SAP's generative AI hub evaluation features to score prompts and configurations against your test datasets and rubrics.
Calibrated and recorded
Automated scores are compared with your process experts' judgment, and agent versions, models and settings are recorded with every run.
Related services
Evaluation Assessment
A baseline for one Joule agent or workflow, with prioritized findings and a release recommendation.
Explore serviceHallucination Testing
Measure how often your Joule agent invents facts, policies or actions, and which fixes reduce it.
Explore serviceManaged Evaluations
Recurring tests as your Joule agent, prompts, models and data change, with a monthly quality report.
Explore servicePlan the engagement
Your inputs
A test environment with representative data, agreed roles and access, version details, process rules, example requests and a process expert who can review results.
Your deliverables
An agreed benchmark, prioritized findings with reproducible cases, a reusable test set and recommendations for the next iteration.
Support after handover
We can retest fixes, extend coverage to new Joule workflows or move to recurring Managed Evaluations as your agent changes. Platform usage and licenses stay in your account.
Make your next decision with evidence.
Start with one workflow, clear success criteria, and an evaluation plan your team can use.