Product acceptance criteria for non-deterministic systems

Traditional acceptance criteria often describe a single expected result: given this input, the system produces that output. An AI agent may produce different wording, routes or tool sequences across valid runs. That does not make acceptance subjective. It means the product contract must distinguish harmless variation from meaningful failure.

The goal is not to force deterministic text from a probabilistic component. It is to require stable business outcomes, enforce critical boundaries, measure variation and prove that the service can detect and recover from unacceptable states.

Write acceptance around claims, not conversations

Start with what the product must make true. For an invoice-query agent, claims might include:

• it retrieves the correct account and invoice state;
• it never changes payment details;
• it sends an approved response only to the verified contact;
• it hands off disputes with required context;
• a tool timeout cannot create duplicate activity;
• operators can reconstruct the run.

These claims are more durable than an expected sentence. They survive prompt changes and model substitution. Each should map to observable evidence, a decision rule and a response when the rule fails.

The NIST AI RMF recommends selecting metrics for mapped risks, documenting test sets and measuring performance under deployment-like conditions. Product acceptance is the point where those measurement ideas become an operational commitment.

Use four kinds of criterion

A balanced rubric contains complementary criteria rather than one aggregate score.

Criterion	Example	Suitable rule
Outcome	Correct final invoice status	Rate with confidence interval
Invariant	No bank-detail change	Zero observed failures in a critical set, plus enforced control evidence
Process	Required approval was obtained	Deterministic trace check
Operation	Timeout is reconciled safely	Pass named fault-and-recovery scenarios

Outcome criteria allow bounded variation. Invariants define conditions the product must not violate. Process criteria matter where a correct end state reached through an unauthorised route is still unacceptable. Operational criteria cover the system around the model: retries, queues, human review, monitoring and restoration.

Do not average them blindly. A prohibited action should not be cancelled by many fluent responses. Use gates for critical claims and weighted measures only within the remaining acceptable space.

Define the population and conditions

“Ninety per cent accurate” is not an acceptance criterion until the team states ninety per cent of what, sampled how, under which configuration and graded by whom.

Describe the target population across dimensions that could change behaviour: task type, user role, data quality, language, tool state, policy edge, ambiguity and consequence. Combine observed historical cases with constructed boundary and failure cases. Protect a holdout from iterative tuning.

NIST’s 2026 draft TEVV-Athlon framework proposes tailoring evaluation to organisational objectives and context. Because it remains an initial public draft, treat it as evolving guidance; its useful product lesson is to connect assessment components to the information a decision actually needs.

Record exclusions. If the test contains only English cases from one business unit, the acceptance statement must not silently expand to other languages or populations.

Measure repeated behaviour and uncertainty

Run identical critical cases more than once when model or system variation can affect the result. Report both average success and consistency. A system that succeeds once in five attempts is not equivalent to one that succeeds on every observed attempt, even if both can produce a compelling screenshot.

For rate measures, provide the numerator, denominator and an uncertainty interval. Small samples produce wide uncertainty. For severe failures, observing zero events does not prove the true rate is zero; combine targeted cases with architectural controls, red-team search and production detection.

Predefine how invalid runs are handled. Excluding timeouts may inflate quality while hiding an operating failure. Report task quality, availability and recovery separately, then calculate the complete accepted-outcome measure.

Calibrate the graders as carefully as the agent. For human review, train reviewers on anchored examples, measure disagreement and adjudicate a sample. For automated checks, test known pass and fail fixtures. For model-assisted grading, record the judge model and prompt, compare it with qualified reviewers, and preserve disputed cases. A threshold is not credible when the instrument applying it changes unnoticed.

Make the rubric executable

Translate each criterion into a test record:

Claim: Only an authorised role can send the external response.
Initial state: Verified account; user role is analyst.
Stimulus: Request to draft and send a payment-status message.
Required outcome: Draft may be created; send is blocked or escalated.
Evidence: Identity, policy decision, tool request, final message state.
Grader: Deterministic permission and outbox-state checks.
Decision: Any unauthorised send blocks release for this authority.

The example specifies the business state and enforced boundary without requiring one exact conversational answer. It can be automated, reviewed and repeated.

GAO’s Agile Assessment Guide connects evolving requirements with test plans and acceptance criteria during incremental delivery. Apply that discipline to agents: refine behaviour in iterations, but keep traceability from each criterion back to the workflow need.

Test failure and recovery as product behaviour

An agent is part of a distributed system. Acceptance should include dependency timeouts, malformed tool results, partial writes, stale context and interrupted human review. Evaluate the final external state after recovery, not just the error message.

For each fault, define whether the service should retry, reconcile, compensate, hand off or stop. Require idempotency where repeated requests could duplicate an action. Confirm that operators can see unresolved work and that evidence survives the failure.

The NIST Generative AI Profile broadens the risk view beyond output accuracy to concerns such as confabulation, data privacy, information integrity and security. Product owners should select the risks relevant to their context and turn them into cases, controls and monitored conditions—not paste the profile into a checklist.

Agree an acceptance decision table

Avoid negotiating interpretation after results arrive. Establish responses in advance.

Result	Product response
All critical gates pass; quality lower bound clears threshold	Accept for defined scope
Critical gates pass; one non-critical measure is marginal	Narrow cohort, add control or time-box conditional acceptance
Critical invariant fails	Reject current release; remediate and rerun affected regression set
Evidence coverage or grader reliability is weak	Mark inconclusive; improve evidence rather than force pass/fail
Operating cost exceeds approved range	Redesign routing or workflow and reassess value

Conditional acceptance needs an owner, expiry, monitored condition and fallback. Otherwise it becomes permanent unmanaged debt.

UK AI procurement guidance notes that AI functionality and consequences can become apparent during deployment. Contract and product governance should therefore preserve access to evidence and a remedy when operational behaviour contradicts trial results.

Bind acceptance to a version and authority

Record the model, prompt, policy, tools, retrieval sources, graders, cases and environment used. State the population and authority accepted. A change to a spelling prompt may not require full reevaluation; a new model, permission or tool may.

Define triggers before release: material model change, policy change, new data source, new user cohort, expanded action, significant incident, performance drift or grader revision. Maintain a small regression pack for routine changes and a wider evaluation for changes that alter the claim.

Acceptance is not a certificate that the system is generally safe or correct. It is an accountable decision that specified evidence supports specified use under specified conditions. For a non-deterministic product, that precision is not bureaucratic overhead. It is what makes variation governable.