Design agent evaluations backwards from the decision

The most expensive evaluation mistake is not a weak metric. It is measuring something that cannot change the decision.

Teams often begin with an available benchmark, a convenient dataset or a dashboard template. They accumulate scores, then ask what those scores mean for release. Reverse the order. Write down the decision, the options and the consequence of being wrong. Then derive the claims that must be supported and the observations that could support or defeat them.

This decision-backwards method tends to produce fewer tests, but each test has a job. It also exposes disagreements early: product, security and operations may discover that they are asking different questions under the shared label “is the agent good enough?”

Write a decision that has real alternatives

“Evaluate the support agent” is not a decision. “Decide whether version 4 may issue refunds up to £50 without review for existing UK customers” is. The second statement identifies a version, action, boundary, population and change in authority.

List the available choices before designing tests. They may include:

1. release the proposed authority;
2. release with a smaller limit or user cohort;
3. keep the agent in recommendation-only mode;
4. remediate named weaknesses and retest; or
5. stop the use case.

If every plausible result leads to release, the exercise is demonstration rather than evaluation. If every failure leads to indefinite study, it has no decision rule. The UK government’s portfolio of AI assurance techniques distinguishes techniques such as performance testing, impact assessment, audit and formal verification because different questions need different evidence. Choose the technique after the question is clear.

Convert the decision into testable claims

Express what must be true for each option. Claims should concern the complete agent and workflow, not a vague property of the model.

For the refund example, the release case might depend on these claims:

• eligible requests are resolved to the correct account state;
• ineligible or ambiguous requests do not cause a refund;
• the agent confirms where policy requires confirmation;
• retrieved content cannot alter refund authority;
• tool faults do not produce duplicate transactions;
• a person receives enough context when the agent hands off;
• latency and cost remain acceptable for the operating model.

Connect each claim to consequence. A failure to use friendly wording and a duplicate financial action do not deserve the same sampling, threshold or release effect. The NIST AI RMF explicitly connects measurement to mapped risk and management action. That sequence prevents a readily measured but low-consequence property from crowding out a difficult, consequential one.

Draw the evidence chain before writing cases

For every claim, sketch four links: observable behaviour, measurement method, decision rule and response. This is a lightweight assurance argument, not a claim of formal proof.

Claim	Observable evidence	Decision rule	If the rule fails
No duplicate refund	Final ledger state after retry and timeout faults	Zero duplicates in the defined fault set	Block autonomous action
Required confirmation occurs	Traces for all confirmation branches	Each protected branch records valid confirmation	Fix policy enforcement and retest
Handoff is usable	Rubric scores from trained reviewers	Lower bound meets agreed level	Restrict cohort or improve summary
Cost is operable	Distribution of cost per qualified completion	Tail remains within budget	Change workflow or routing

The table forces precision. “High accuracy” becomes a property of an observable state under specific conditions. “Safe” becomes several claims that can fail independently. A test without a linked response should be challenged: what would anyone do differently after seeing it?

The 2026 NIST draft TEVV-Athlon framework takes a similarly objective-led approach to constructing customizable assessments. It is an initial public draft, not settled practice, but its emphasis on organisational objectives and context is useful: assessment design starts from the information the organisation needs, not from a universal battery.

Select cases by consequence and coverage

Create a case map from the workflow rather than gathering prompts. Include ordinary paths, boundaries, exceptions, abuse, faults and recovery. Vary the dimensions that can change behaviour: user role, data state, tool availability, instruction ambiguity, retrieved content, prior turns and authorisation state.

Prioritise by three questions:

• Could this case cause a materially different outcome?
• Is the case plausible in the intended environment?
• Does it exercise a claim not already covered?

Use historical cases when their provenance and use are appropriate; they represent the real distribution and its messiness. Add constructed cases for policy edges and rare but serious failures. Add adversarial cases where an attacker or untrusted content can influence the agent. Preserve a holdout set so iterative prompt work does not simply memorize the visible suite.

NIST’s ARIA pilot report used model testing, red teaming and field testing at separate levels. A workflow evaluation can use the same logic without copying the study: controlled tests establish repeatable behaviour, adversarial work searches beyond expected paths, and field observation checks whether the human-system interaction matches the laboratory assumptions.

Measure the end state and the path that created it

Agent evaluations need at least two views. The end-state view asks whether the correct outcome occurred. The trace view asks whether the system stayed within authority, handled errors and reached that outcome through an acceptable path.

The tau-bench paper illustrates both useful ideas: checking the database state after tool use and repeating tasks to measure consistent success. In an enterprise workflow, the evaluator should add local policy checks. A refund issued to the right account may still be unacceptable if the agent bypassed confirmation or exposed unrelated account data along the way.

Prefer deterministic graders for structured state and policy invariants. Use rubric-based human review where meaning or handoff quality matters. If a model assists grading, calibrate it against people on a sample, version it and preserve disagreement. “Automated” describes the mechanism, not the credibility of the judgement.

Set thresholds from the decision’s error costs

Do not ask what score looks respectable. Ask what errors the owner can tolerate under the proposed authority.

Define separate thresholds for materially different failures. Routine task completion may use a rate with uncertainty. A prohibited action may be a zero-tolerance result in a finite critical set, while still acknowledging that observing zero failures does not prove a true failure probability of zero. Tail latency, excessive retries and unnecessary handoffs may have different operating thresholds.

Build escalation into the rule. A typical structure is:

• green: evidence supports the proposed authority;
• amber: evidence supports a narrower cohort, lower limit or stronger human gate;
• red: a critical claim failed, so remediate before release;
• inconclusive: coverage, sample or grader quality is insufficient, so collect targeted evidence.

Inconclusive is an important result. Forcing uncertainty into pass or fail rewards false confidence.

Design the report at the same time as the test

Before execution, create the result table the decision owner will receive. Include configuration, case provenance, claim coverage, measures, uncertainty, critical traces, invalid runs, findings, residual risk and recommended options. This reveals whether the planned run can actually populate the decision record.

Also define comparators carefully. A previous agent version tests change risk. A human-operated baseline may inform usefulness and rework. A simpler automation may test whether an agent is necessary. None is automatically the right reference.

The evaluation should end with an accountable choice, not a pile of charts. Record which claims were supported, contradicted or left uncertain; which operating conditions bound the conclusion; and what change would trigger reevaluation.

Working backwards does not guarantee a favourable release decision. It guarantees something more valuable: when evidence is weak, the team knows which decision it cannot yet justify and what evidence would reduce that uncertainty.