What confidence is enough for an agent release decision?

There is no universal score at which an AI agent becomes safe enough to release. “Ninety-five percent confidence” sounds scientific, but confidence in what: task completion, policy compliance, a sampled failure rate, grader agreement, or the belief that the test represents production?

The release owner needs a structured sufficiency judgement. How consequential is the proposed authority? How representative and credible is the evidence? What uncertainty remains? How quickly can exposure be contained or reversed? The same evaluation result can justify a limited internal trial and fail to justify autonomous customer action.

Separate statistical confidence from decision confidence

A confidence interval describes uncertainty in an estimate under assumptions about sampling and measurement. It does not certify the test set, policy, grader or system design.

Suppose an agent succeeds on 96 of 100 independent cases. The observed rate is 96%, but the plausible population rate is a range, not a point. The NIST engineering statistics handbook explains interval methods for proportions and warns that simple approximations can be unreliable with small samples or few failures.

Even a well-calculated interval answers a narrow question. It does not tell you whether:

• the sample represents the intended workflow;
• cases are independent;
• the grader is correct;
• rare, severe failures were covered;
• policy compliance should be averaged with routine success;
• production conditions match the test.

Decision confidence combines statistical evidence with coverage, validity, consequence and controls. Keep those components visible rather than hiding them in one percentage.

Start with the cost of each error

There are at least two decision errors. A false release accepts a system whose behaviour is not good enough. A false rejection delays a system that was adequate. Their costs differ by workflow.

For an internal drafting assistant, false rejection may waste time while false release creates reviewable rework. For an agent with permission to change financial or identity state, false release can be far more consequential. That asymmetry should change the evidence burden, control design and release scope.

The NIST AI RMF frames risk through likelihood and impact and calls for measures of uncertainty connected to management decisions. Translate that into an explicit statement:

We require stronger evidence for unauthorised action than for unnecessary handoff because the former has greater consequence and weaker recovery.

That is more defensible than applying 90% to every metric.

Build a confidence case across five dimensions

Review evidence sufficiency using five independent dimensions:

Dimension	Stronger evidence looks like
Relevance	Cases and conditions match the proposed workflow and authority
Coverage	Routine, boundary, fault, adversarial and human-interaction paths are represented
Measurement	State checks are objective; semantic graders are calibrated; invalid runs are visible
Repeatability	Results persist across repetitions, configurations and reviewers where appropriate
Operability	Monitoring, containment, rollback and accountable ownership are tested

Rate each dimension as adequate, constrained or weak and explain why. A large sample cannot repair weak relevance. A perfect laboratory result cannot substitute for an untested rollback. A small but carefully targeted critical set may provide stronger evidence about an authority boundary than thousands of routine prompts.

NIST’s ARIA pilot combined model testing, red teaming and field testing. Its three-level design illustrates why different evidence types should complement rather than collapse into one score.

Choose sample size from detectable difference

Sample size should follow the precision or change the decision needs to detect. The NIST handbook’s guidance on sample sizes for proportions links sample planning to significance, power and the difference of interest.

In practical terms:

1. define the minimum unacceptable change or maximum tolerable failure rate;
2. decide how much risk of missing that change the owner accepts;
3. estimate enough cases for the relevant slice, not only the total;
4. adjust for repeated tasks, clustering and invalid runs;
5. plan before seeing results.

If failures are rare and severe, rate estimation may require an impractically large random sample. Use targeted critical cases, architecture controls and adversarial analysis as additional evidence. Observing zero failures in 50 tests does not establish a zero failure probability.

Treat critical properties as invariants where possible

Some release requirements should not be expressed as average rates. Effective authorisation must be checked on every protected action. An approval token must bind to the specific parameters. A successful response must not be emitted before an irreversible action is confirmed.

Evaluation still cannot prove these properties for every unseen situation, but deterministic enforcement changes the evidence question. Instead of trusting the model to comply at some rate, test whether an external policy mechanism denies unauthorised calls across the relevant paths and fails closed when it is unavailable.

Use statistical measurement for variable qualities such as completion, unnecessary handoff, latency and cost. Use enforced invariants for hard boundaries. Then test the enforcement and its bypasses.

Increase confidence by narrowing the release

When evidence is insufficient for full authority, change the decision rather than pretending uncertainty disappeared. Constrain:

• capability: draft but do not execute;
• cohort: trained internal users before external users;
• authority: lower transaction or data scope;
• environment: sandbox or reversible records;
• duration: time-boxed controlled release;
• volume: capped concurrent work;
• oversight: mandatory review for selected paths.

This makes exposure proportional to evidence and creates a path to learn. Define the observations, success criteria, stop conditions and next decision in advance. A pilot without those elements merely postpones judgement.

The FDA’s draft guidance on AI used in regulatory decision-making proposes that credibility assessment be tailored to a defined context of use and commensurate with model risk. Its domain is specific and its guidance is non-binding, but the context-of-use principle transfers well: evidence is credible for a purpose, not in the abstract.

Define confidence loss after release

Release confidence is perishable. A model alias changes, a new tool is added, policy is revised, production cases shift or an incident reveals an uncovered path. Record what invalidates or reduces reliance on the decision.

Also distinguish a trigger that invalidates evidence from one that merely asks for closer observation. A new tool with write authority should reopen relevant claims before use. A modest traffic-mix change may call for additional sampling while the existing scope remains. Predefining that distinction prevents both complacency and reflexive shutdown.

Typical triggers include:

• any authority, prompt, model, retrieval or tool change;
• performance outside the evaluated population;
• a critical incident or repeated near miss;
• monitoring that crosses a predefined bound;
• material grader drift or new label disagreement;
• expiry of a time-boxed acceptance.

Tie each trigger to a response: regression run, targeted evaluation, scope reduction, rollback or new decision review.

Record why the evidence was enough

The final decision should state proposed authority, evidence reviewed, key uncertainty, unresolved findings, controls, release constraints, stop conditions and named owner. Avoid “the model achieved 95%, therefore approved.”

A defensible conclusion sounds more like this:

The evidence supports a four-week release to trained staff in recommendation-only mode. Workflow coverage is adequate for the current queue; recovery evidence remains weak for external writes, so those tools stay disabled. The release stops if policy failures occur or qualified completion falls below the agreed bound.

That record does not eliminate uncertainty. It shows that uncertainty has shaped the exposure. Confidence is enough when the evidence and controls justify the specific authority being granted—and no more.