Your agent passed the test. What does that prove?
A release decision needs a traceable connection between expected behaviour, test conditions and observed results. Here is a practical way to build that connection.
Architecture & engineering, Evaluation & assurance Decisions behind dependable agents
What leaders need to know
Evidence that an agent works is a set of measurements, drawn from real and deliberately constructed cases, that cover what the agent achieved, how it used its tools, whether it stayed within policy, how it recovered from problems, what it cost, and what it handed to the person at the end. It is expressed as release criteria that state thresholds, account for variance between runs, and record what was not measured. A single accuracy figure on a public benchmark is none of these things, though it may be a useful starting point.
Why public benchmarks are a starting point, not evidence
Public agent benchmarks have done real work in showing what is hard. SWE-bench demonstrated that resolving real software issues requires navigating a codebase, editing several files and running tests, and that headline model capability translated poorly into that setting when the benchmark was introduced. τ-bench went further for tool-using agents by simulating a user and checking the final database state against a policy, and by reporting a pass^k metric that measures whether an agent succeeds consistently across repeated trials of the same task rather than once. Its authors reported that agents built on then-current models were far less consistent than their single-run success rates implied.
Two lessons transfer directly to enterprise evaluation. First, the state of the world after the agent acts is the thing to check, not the text it produced. Second, consistency across runs is a property worth measuring in its own right, because a workflow owner cares about the eighth run as much as the first. What does not transfer is the content: a benchmark exercises its own domain and policies, not yours.
The behaviours worth measuring
A useful taxonomy for an agent that acts in a workflow covers six areas. Each maps to a type of evidence.
| Behaviour | What to look at | Typical evidence |
|---|---|---|
| Task outcome | Was the workflow result correct, judged against a definition the workflow owner agrees with? | Real cases with expected end states; graders that check state, not prose |
| Tool behaviour | Did the agent choose the right tools with correct arguments, avoid unauthorised ones, and handle partial execution? | Traces annotated with expected tool calls; probes with misleading tool descriptions |
| Policy compliance | Did it refuse, escalate or confirm where policy requires, without over-refusing? | Scenario sets for each policy rule, including near-misses |
| Recovery | What did it do when a tool errored, context was stale, or instructions conflicted? | Fault-injection cases; recorded incidents replayed |
| Cost and latency | Steps, tokens, wall time and money per task, and how they change | Harness telemetry across releases |
| Hand-off quality | Could the person receiving the output act on it without rework? | Human review on a sample, with a rubric |
The table is deliberately ordinary. The point is coverage: most teams measure the first row and treat the rest as anecdote.
Where the evidence comes from
Three sources, in roughly this order of value:
- Real cases. Historical instances of the workflow, including the exceptions and the ones that went wrong. These are the most credible evidence and the hardest to assemble, because they need an owner willing to define what “correct” meant and because they may contain personal data that has to be handled properly.
- Constructed cases. Scenarios written to exercise a specific behaviour: a policy edge, a tool that returns an unexpected shape, a document containing instructions aimed at the agent. These are where adversarial evaluation lives.
- Recorded incidents. When the agent fails in production, the trace becomes a regression case. A harness that cannot ingest incidents will drift away from reality.
Provenance matters for all three. Each case should record where it came from, what it exercises and who judged the expected outcome, so that a reviewer can question the evidence rather than accept a score.
Graders
Evidence is only as good as the judgement applied to it. Three kinds of grader are used, in order of preference:
- Deterministic checks against end state, tool logs or structured output. Use these wherever the behaviour allows; they are cheap, repeatable and hard to argue with.
- Model-assisted grading for behaviours that need reading, such as whether an explanation is adequate. These should be calibrated against human judgement on a sample, and their agreement rate recorded and re-checked when the grading model changes.
- Human review for hand-off quality and for anything where the cost of a wrong judgement is high. Sampled, rubric-based, and recorded.
Treating a model-graded score as if it were deterministic is one of the most common ways an evaluation programme quietly loses credibility.
Turning evidence into release criteria
Release criteria translate the taxonomy into decisions. A workable criterion states the behaviour, the evidence set, the threshold, how variance is handled, and what happens on failure. For example: “On the 140-case policy set, the agent must confirm before any refund above the threshold in at least 98% of runs across five repetitions, with no run below 95%; a failure blocks release and is triaged within one working day.” The numbers are illustrative; the structure is the point.
Two properties make criteria credible. They report distributions, not a single lucky run, which is the lesson of pass^k. And they say what was not measured. The NIST AI Risk Management Framework recommends ongoing measurement and documented limitations. Both apply with particular force to systems that change every time a model or tool description is updated.
Keeping it alive
An evaluation foundation decays unless it runs on every change and unless incidents feed it. The practical arrangement is that prompt, tool and model changes trigger the harness, results are compared with the last release against the criteria, and any production incident produces a new case within a defined time. Teams that manage this find that the harness becomes the place where arguments about the agent are settled, which is what evidence is for.
What this does not settle
Evidence of this kind reduces uncertainty about behaviour within the evaluated scope. It does not establish that the agent is safe in situations nobody thought to construct, and it does not measure business value, which needs a baseline and a value model of its own. It does give a governance function something concrete to accept or reject, which is a considerable improvement on a demonstration.