Beyond pass or fail: how to grade agent evaluation findings

An agent says the wrong thing, calls an unauthorised tool, repeats a transaction, or hands a person an unusable summary. Each can be a failed test. They are not equally important findings.

A useful grading method tells a decision owner what happened, how credible the observation is, what it could affect and how urgently it must be treated. It does not turn a complicated judgement into a decorative red badge. Most importantly, it keeps three questions separate: is the observation valid, how serious is the underlying behaviour, and what should this release do about it?

Validate the observation before grading it

Start with evidence hygiene. Reproduce the behaviour where repetition is meaningful. Check the trace, external state and harness logs. Confirm that the expected result matches the policy and that the grader did not misread a legitimate alternative path.

Classify the raw result before creating a finding:

• confirmed behaviour: the agent or surrounding control behaved contrary to a defined expectation;
• intermittent behaviour: the failure appears under repeated equivalent conditions but not every time;
• test defect: the expected result, fixture, environment or grader is wrong;
• inconclusive: available evidence cannot distinguish system failure from test failure;
• duplicate symptom: the case is another manifestation of an existing root issue.

Do not grade test defects as agent risk. Do not discard inconclusive results either; they identify an evidence problem that may itself block a consequential decision. The UK introduction to AI assurance emphasizes choosing suitable techniques for the deployment context and communicating risk evaluation clearly. That starts with being candid about what the test established.

Describe the finding as a causal statement

Write a finding so an engineer and a workflow owner can recognise the same problem:

When a refund tool times out after committing the transaction, the orchestrator retries without an idempotency key, creating a second refund. The duplicate is visible in the ledger but not to the agent before it tells the user the request succeeded.

That sentence identifies trigger, mechanism, state change and visibility. Compare it with “agent may issue duplicate refunds,” which leaves the team guessing about conditions and treatment.

Attach the system version, case IDs, repetition count, traces, expected requirement and affected control. Separate direct observation from inference. “Two refunds were recorded” is observed; “this could recur whenever the provider response is lost” is a reasoned hypothesis to test.

Grade consequence across several dimensions

Avoid a single instinctive severity rating. Review at least six factors:

Factor	Question
Consequence	What harm, loss, rights impact or operational disruption could follow?
Reach	Which users, records, tools or environments could be affected?
Authority	Does the behaviour cross a permission, approval or policy boundary?
Reproducibility	Under defined conditions, how consistently does it occur?
Detectability	Would an existing control find it before or after consequence?
Recoverability	Can the state be contained, reversed and communicated?

The factors interact. A rare failure that can transfer funds without approval may outrank a common formatting defect. A visible wrong draft that a trained reviewer always catches differs from a silent external action. A reversible duplicate in a sandbox differs from an irreversible message sent to a customer, even if the agent makes the same reasoning mistake.

NIST SP 800-30 frames risk assessment around threats, vulnerabilities, likelihood and impact. Agent evaluation adds useful system-specific detail—authority, detectability and recovery—but should preserve that basic distinction between how a failure arises and what it can do.

Keep severity, confidence and priority separate

Assign three labels rather than one overloaded score.

Severity describes the plausible consequence if the behaviour occurs in scope. A four-level scale can work:

• critical: crosses a hard authority boundary or can create severe, difficult-to-reverse consequence;
• high: can materially harm a person or operation, expose sensitive data or defeat a key control;
• moderate: degrades a workflow or produces consequential rework but is bounded and recoverable;
• low: limited quality or usability impact with reliable detection and simple recovery.

Evidence confidence describes how strongly the test supports the finding: confirmed, probable or preliminary. Base it on trace completeness, grader quality, repeatability and environmental fidelity.

Treatment priority considers release timing, exposure and available controls. A high-severity finding may have lower immediate priority if the affected tool remains disabled. A moderate finding may demand immediate attention if it occurs in most routine tasks and makes the product unusable.

Cybersecurity teams can learn from what FIRST says about CVSS v4.0: a base severity score should not be used alone as risk because threat and environmental factors matter. Do not copy CVSS mechanically for agent behaviour. Borrow the discipline of separating intrinsic characteristics from the deployment context.

Map the grade to an explicit action

Severity labels become useful only when their default consequences are defined before the review.

Grade	Default release treatment	Minimum evidence response
Critical	Block affected capability	Confirm root condition, remediate, add regression and retest
High	Block or materially constrain scope	Named owner, treatment plan, control validation and decision record
Moderate	Accept with action or constrain	Document rationale, monitor and schedule regression
Low	Backlog or accept	Preserve evidence and watch for aggregation

The word “default” matters. An accountable owner may make a different choice, but the exception should be visible. Do not lower a technical grade because a launch date is near. Record acceptance, compensating controls, expiry and the person with authority to make that trade-off.

The NIST AI RMF treats measurement as input to managing risk, including mitigation, removal, monitoring and recovery. Evaluation informs the decision; it does not erase ownership.

Handle non-determinism without normalizing failure

An intermittent finding is not automatically minor. Repeat the triggering case under controlled settings and report a distribution: failures over trials, confidence interval where appropriate, variation by configuration and correlated conditions. Preserve seeds and service versions when available.

Then ask whether the failure mode permits rate-based acceptance. Some properties do: unnecessary handoffs or low-impact formatting defects may have operating thresholds. Others are policy invariants: a valid approval must bind to the exact action, or a user without permission must not reach a privileged tool. Repetition measures the evidence; it does not turn an authority breach into a harmless average.

The NIST generative AI profile notes that red-team results require additional analysis before use in governance. That is especially important for spectacular one-off failures. Establish whether the path is feasible in the intended environment, whether existing controls interrupt it and whether the test granted unrealistic access. Preserve the observation even when the operational risk is lower than the demonstration suggests.

Report a finding portfolio, not a failure count

Counting findings rewards fragmentation. One authorisation defect may produce 40 failed cases; 40 unrelated low-grade usability issues may be less consequential. Summarise by root condition, affected claim and release effect.

A decision view should show:

• open findings by severity and evidence confidence;
• which proposed capabilities each finding affects;
• repeated manifestations linked to one root issue;
• accepted findings and their compensating controls;
• regressions from the last accepted version;
• unresolved test defects or evidence gaps;
• owners, target dates and review triggers.

Keep the underlying case results available. The portfolio is a navigation layer, not a substitute for evidence.

Use disagreement as information

When reviewers disagree on a grade, do not average their labels. Identify the disputed assumption: consequence, reach, plausibility, detectability, reversibility or owner authority. Resolve it with domain evidence or record the range.

A grading workshop can use one worked case to calibrate reviewers before the main assessment. Periodically compare grades with incidents and near misses. If “moderate” findings routinely cause release interruptions, either the definitions or controls are wrong.

No rubric eliminates judgement. A good one makes judgement inspectable. It preserves the observed behaviour, shows the reasoning from consequence to grade, and keeps the final acceptance where it belongs: with the person accountable for the workflow.