How to estimate the cost of an agent failure

“A failure could be expensive” is too vague to choose a control. “Each false payment instruction creates two hours of reconciliation, an average four-hour payment delay and a plausible external investigation” begins to support a decision.

Estimating failure cost does not require predicting one perfect number. It requires tracing a specific failure through detection, containment, correction and downstream consequence, then showing how controls change its likelihood or impact.

Define the failed state precisely

Start from an observable deviation, not a broad label such as hallucination. Examples include:

• an ineligible refund is issued;
• a correct refund is issued twice after a timeout;
• confidential text is included in an external message;
• a maintenance exception is omitted from a handoff;
• a valid case is rejected and enters manual appeal;
• a tool call changes the wrong customer record.

Specify the initial conditions, population, authority, affected asset and final external state. Separate causes from consequences. The same bad payment could arise from a fabricated account, stale retrieval, permission failure or retry defect; those causes need different controls even if some response costs overlap.

NIST’s AI Risk Management Framework starts by mapping context and risk before selecting measures and responses. A failure-cost estimate should inherit that context rather than apply an industry-average loss to every agent.

Trace the consequence path

Map what happens after the failed state:

failure occurs
  -> detection delay
  -> continued exposure or spread
  -> containment
  -> investigation and evidence collection
  -> correction and reconciliation
  -> service restoration
  -> notification or escalation where required
  -> remediation and reevaluation

At each stage, name the team, elapsed time, direct spend, diverted capacity and affected outcome. Include the human burden on customers or employees where relevant; do not force every consequence into money.

NIST’s incident-response guidance integrates preparation, detection, response and recovery with wider cybersecurity risk management. An agent error is not automatically a cybersecurity incident, but its lifecycle is a useful operational template for failures that require coordinated response.

Build a scenario cost equation

For scenario (s), estimate:

Scenario loss = detection + containment + investigation + recovery + business disruption + other material external consequences.

where:

• (C_d): detection and triage;
• (C_c): containment;
• (C_i): investigation and evidence;
• (C_r): correction, recovery and retesting;
• (C_b): business disruption and lost outcome;
• (C_x): other material external consequences assessed by the appropriate owner.

Keep cash, staff time, delayed value and non-monetised harm separate in the underlying table. Aggregation should not hide who bears the cost.

A worked example for a duplicate supplier payment might use ranges:

Cost element	Low case	Central case	High case	Evidence owner
Triage and containment	1 staff-hour	3 hours	8 hours	Operations
Reconciliation	2 hours	6 hours	20 hours	Finance
Recovery and retest	4 hours	12 hours	40 hours	Engineering
Payment disruption	Minor delay	Supplier escalation	Material service delay	Procurement
External consequence	None identified	Case-specific	Specialist assessment	Risk/legal owner

The units can be converted using approved internal cost methods. Do not publish illustrative figures as evidence from a real incident.

Estimate frequency without false confidence

Expected annual loss is often represented as:

Expected loss = the sum, across scenarios, of frequency × loss.

The formula is simple; the inputs are not. Historical data may be sparse, the workflow may be new, and observed failures may depend on changing controls. Use ranges or distributions. Distinguish frequency per attempt, per accepted case and per year.

Sources for estimation can include production history from an analogous process, evaluation failure rates, red-team findings, fault-injection exercises, near misses and structured expert judgement. Correct for detection gaps where possible. A system with poor logging may appear to have fewer incidents.

The 2026 Green Book discusses expected values while warning that complex uncertainty can make probability estimates spuriously precise. Use expected loss to compare options, not to claim certainty.

Model correlated and tail events

Multiplying a per-run error rate by volume assumes failures are independent. Agent failures may cluster after a model change, policy error, corrupted retrieval source or unavailable dependency. One defect can affect many runs before detection.

Create event scenarios:

• isolated case detected immediately;
• repeated cases during a detection window;
• systemic release affecting a cohort;
• adversarial exploitation of one weakness;
• failure combined with unavailable rollback or evidence.

Stress volume, time-to-detect, recoverability and spread. Report the plausible loss range for each. A low average expected loss may still coexist with a tail scenario outside risk appetite.

HM Treasury’s Orange Book frames risk management as integral to objectives and calls for clear roles, monitoring and improvement. Use the scenario model to decide who accepts residual risk and what information they need.

Compare controls by changed consequence

Estimate how a proposed control affects frequency, detection delay or impact. Examples:

Control	Primary effect	Cost question
Approval above a payment threshold	Reduces execution frequency for severe cases	Reviewer capacity and delay
Idempotency key and state check	Prevents duplicate action	Engineering and storage cost
Canary release	Limits cohort exposed to a defective version	Slower rollout and parallel support
Outcome reconciliation	Shortens detection time	Monitoring and investigation effort
Kill switch with tested rollback	Limits spread and recovery time	Exercise and availability cost

Calculate residual scenario cost after control, then compare control cost. Do not assume an architectural control is perfect. Use evaluation or exercise evidence for its effectiveness and include control failure in tail scenarios.

GAO’s Cost Estimating Guide recommends sensitivity and risk analysis plus updates with actual costs. Apply that discipline to both failure and control estimates.

Keep professional boundaries visible

Technical teams can estimate engineering recovery, operational effort and observed service effects. Legal exposure, regulatory obligations, contractual liability, financial reporting treatment and harm valuation require the relevant qualified owners. Record their inputs rather than guessing.

Avoid assigning a convenient monetary value to privacy, safety, discrimination or trust simply to complete a spreadsheet. A decision can combine quantified cost with explicit non-financial constraints and unacceptable states.

Also distinguish gross impact from recoveries such as insurance, supplier credits or reversed transactions. Recoveries may be uncertain, delayed or subject to terms; they do not erase the operational failure.

Update the model through exercises and events

Run a tabletop or controlled fault scenario. Measure the actual time to notice, locate evidence, identify affected cases, contain action and restore service. Replace assumptions with observations. Capture near misses because they reveal exposure without waiting for a loss.

Version the estimate with the system and workflow. New authority, volume, population, dependency or detection control can change both frequency and impact. After an incident, preserve observed cost and update the range without rewriting the earlier forecast.

A failure-cost model is successful when it changes a decision: add an idempotency control, narrow authority, improve detection, rehearse response, price a contract remedy, or decide the expected benefit does not justify the residual exposure. Precision is secondary to a transparent chain from failed state to consequence to control.