What product leaders should demand from an agent evaluation

An agent evaluation should help a product leader make a choice: release, constrain, improve, compare, expand or stop. If the evaluation cannot identify its decision, it is likely measuring what the team can score rather than what the product must accomplish.

Demand evidence about the complete user workflow. A public model benchmark may inform architecture selection, but it does not establish that an agent can act correctly with your tools, permissions, data, users and failure costs.

Start with the product decision

Write the decision and alternatives before choosing metrics. “Should we release address changes to all verified customers?” is evaluable. “Is the agent good?” is not. Define the intended users, workflow boundary, actions, exclusions and consequence of error.

The initial NIST automated benchmark practices draft begins with evaluation objectives and the intended use of results. That principle is broader than model benchmarks: the construct and conditions must fit the decision.

Agree what evidence would change the choice. If no result could block or constrain launch, the exercise is demonstration, not evaluation.

Measure the state the product creates

Agents often produce both language and action. Check final system state, not only response quality. Did the correct record change? Was required confirmation captured? Were prohibited fields untouched? Could the user understand what happened?

Use a balanced outcome model:

Dimension	Product evidence
Qualified completion	Correct outcome with all required policy conditions met
Action integrity	Right tool, arguments, sequence and resulting state
Recovery	Safe response to timeout, conflict, interruption and partial work
Handoff	Context and authority preserved for a person to continue
Experience	Comprehension, effort, trust calibration and recourse
Economics	Cost and latency per qualified completion, including review

Separating these prevents a fast but unauthorised action from counting as success, or a polished explanation from hiding a wrong transaction.

Demand representative and challenging cases

Ask where the cases came from. A credible set combines routine work, real edge conditions, policy boundaries, known incidents, fault injection and adversarial scenarios. Document who defined the expected outcome and which user groups or environments are missing.

The NIST ARIA pilot used model testing, red teaming and field testing because they reveal different behaviour. A product evaluation can adopt that layered logic without copying its exact method.

Protect a holdout set and rotate cases when teams learn them too well. Add production incidents as regressions, but do not let the suite become a museum of past failures while the workflow changes around it.

Inspect variance and grader quality

One successful run is weak evidence for a non-deterministic system. Repeat important cases and report distributions, worst cases and failure clusters. Break results down by scenario and consequence; a high average can conceal a complete failure on a small but important path.

Prefer deterministic grading of state and policy where possible. Calibrate model-assisted graders against human judgements and record the grading model/version. Use rubric-based human review for experience, handoff and high-consequence ambiguity.

Ask how disagreements and invalid runs were handled. Excluding inconvenient results without a declared rule creates confidence the data does not support.

Convert findings into release criteria

Product acceptance criteria should state behaviour, evidence set, threshold, variability treatment and decision effect. For example: every sampled high-value action requires explicit confirmation; no cross-account state change is permitted; recovery from a transient tool failure succeeds without duplicate action in an agreed proportion of repeated runs.

Numbers must come from workflow consequence and risk appetite, not borrowed convention. The NIST AI RMF Core connects measurement to mapped risk and management action. Criteria are useful when a miss triggers a defined response: block, limit scope, add a control or obtain accountable acceptance.

Keep limitations beside the decision. State what was not tested, where data is thin and which external dependencies may change.

Include the operating product

Evaluation is not complete at pre-release. Define what production evidence will show that assumptions still hold: qualified completion, intervention, blocked unauthorised action, retries, user correction, incident rate, cost and latency. Each metric needs an owner and a threshold with an action.

The GAO accountability framework brings governance, data, performance and monitoring together. For product teams, that means ownership of the post-launch learning loop, not forwarding a dashboard to risk once a quarter.

Changes to models, prompts, tools, policy, retrieval or user population should trigger proportionate reevaluation. Use a change log that connects each release to evidence.

Expect a decision record, not a victory deck

The final artefact should fit a review: scope and version, case coverage, methods, results, material findings, uncertainty, control evidence, operating obligations and decision. Link detailed traces rather than burying the choice beneath charts.

Product leaders should demand an evaluation that can support “not yet” as credibly as “go.” Its value is not proving the team was right to build the agent. Its value is making the next product decision more informed, bounded and reviewable.

Work through a release decision, not a scorecard

Imagine a support agent that can explain invoices and apply a fee waiver within policy. The initial report says it achieves 88% task success. A product leader should ask what counted as a task, which state was checked and how policy failures were scored.

A better review separates outcomes. On routine explanations, users received correct and comprehensible answers. On waiver cases, the agent selected the right tool most of the time, but repeated runs showed that one ambiguous eligibility rule produced inconsistent confirmation. Two tool timeouts also led to a retry after the waiver had already committed. The average conceals both a policy defect and a duplicate-action risk.

The release decision can now be specific. Explanations may proceed to a controlled cohort. Fee waivers remain in propose-only mode until the policy path and idempotency control pass regression criteria. Production sampling will track user correction, policy escalation and qualified completion; any cross-account access or duplicate waiver suspends the capability.

This decision is more useful than rejecting or accepting the whole product. It allocates authority according to evidence, gives engineering concrete work and creates a measurable next gate.

Product leaders should also ask what the evaluation taught them about the product proposition. If users need to re-enter context at every handoff, the issue may be workflow design rather than model intelligence. If manual review costs exceed the time saved, automation may not yet create value. Evaluation is a product-learning system when it measures the complete experience, including work displaced onto customers and operators.

Bring users and operators into interpretation, not just case creation. A statistically improved score can still accompany worse trust calibration or more hidden rework. Review a sample of sessions where users accepted, corrected and abandoned the agent’s work. Compare their explanation with traces so the team learns whether the product communicates uncertainty and action state accurately.

Finally, protect the evaluation from roadmap pressure. Freeze release criteria before the final run, preserve failed cases and require an accountable exception when criteria change. A threshold can legitimately evolve as the workflow becomes better understood, but the record should show whether the product improved or the definition of success moved. That distinction is essential for credible portfolio decisions.

A product evaluation has done its job when it sharpens the release choice, the product proposition and the next learning step at the same time.