Failure-cost modelling
Understand the cost of agent failure before relying on average performance
Failure-cost modelling becomes relevant when an AI proposal includes expected benefits but treats failures as a single undefined contingency. A focused review would examine a failure event with a causal path, operational consequence and cost boundary. This is most relevant to risk owners and business-case analysts, especially where teams must distinguish observed loss and modelled exposure.
A measure needs a unit, comparator and decision
Illustrative working format
- Measurement unit
- A failure event with a causal path, operational consequence and cost boundary.
- Failure and incident examples
- Establish the starting evidence and its measurement boundary.
- Modelled exposure
- Keep this distinct when interpreting the proposed measure.
- A risk and investment sensitivity brief
- Connect uncertainty to the management choice it can change.
Decision at stake
Understand the cost of agent failure before relying on average performance
Make the decision explicit
- Risk owners and business-case analysts
- An AI proposal includes expected benefits but treats failures as a single undefined contingency.
- Decision
- Which failure modes materially change the business case and what evidence should inform their treatment.
- Observed loss
- Use failure and incident examples to establish the relevant starting condition.
For risk owners and business-case analysts, the immediate question is which failure modes materially change the business case and what evidence should inform their treatment. This is most useful when there is a real choice between proceeding, changing the design, collecting more evidence or narrowing the intended use. The starting conversation should identify that choice and its consequence before deciding how much work the topic warrants.
An average success rate could obscure a low-frequency failure whose recovery effort or external consequence dominates the decision. A useful review follows a failure event with a causal path, operational consequence and cost boundary, rather than stopping at a convenient component metric or document boundary. The evidence should separate observation, interpretation and recommendation so the accountable owner can see what it supports.
Prerequisites
Start with failure and incident examples
Give each input a specific job
- Failure and incident examples
- Defines the operating boundary.
- Recovery and rework activities
- Examine frequent rework.
- Exposure, frequency and severity assumptions
- Shows conditions affecting incident response cost.
- Business and finance owner
- Confirm access and a reviewer who can resolve gaps.
For Failure-cost modelling, the sponsor should identify the decision, authoritative data sources and the owner of each measure. A baseline or credible comparator is especially important. Where the relevant information is missing, the work should establish a prospective measurement plan or narrow the claim rather than manufacture historical precision.
Useful starting material would include failure and incident examples; recovery and rework activities; exposure, frequency and severity assumptions. These inputs have different jobs: the first establishes context, the second supplies behaviour or comparison material, and the third exposes operating conditions that may qualify a conclusion. Existing material would be reviewed before requesting new collection, and a missing input would be recorded as an evidence gap rather than treated as a favourable result.
Coverage
Observed loss and modelled exposure need different evidence
Keep the review distinctions visible
- Observed loss
- Report separately from modelled exposure.
- Frequent rework
- Report separately from rare severe event.
- Incident response cost
- Report separately from wider business consequence.
Observed loss versus modelled exposure
Failure modelling would start with a concrete event and trace the activities it causes. Rework, investigation, communication and downstream repair would be included only within a clear and defensible cost boundary.
Frequent rework versus rare severe event
Observed incident costs would remain separate from hypothetical severity and frequency. Sparse historical evidence may justify scenario analysis but should not be converted into an invented probability distribution.
Incident response cost versus wider business consequence
The decision view would connect costly failure paths to candidate controls and tests. A low-frequency event may deserve attention because of consequence, while a frequent minor defect may dominate ordinary operating effort.
Illustrative scenario
Duplicate supplier request: examine what changes the decision
Examine the constructed decision boundary
Illustrative scenario
- Duplicate supplier request
Routine cleanup and a possible conflicting commitment have different consequences.
- Cost pathways
Separate observed recovery tasks from modelled severe-event exposure.
- Control priority
Identify missing frequency and consequence evidence without inventing loss estimates.
Consider a fictional agent that occasionally duplicates a supplier request. Most duplicates require clerical cleanup, while a particular pathway could trigger conflicting commitments. The proposed model would separate those events, document recovery tasks and distinguish observed costs from scenarios. It would support prioritising controls and evidence collection without inventing incident probabilities or loss estimates.
The team might decide that ordinary duplicate cleanup can be measured from existing records, while the commitment pathway needs a constructed adverse scenario and specialist input. The model would keep those evidence levels separate. That distinction lets the sponsor prioritise investigation without inventing a precise likelihood for the severe event. These are hypothetical decision paths, not observed client outcomes.
How the work proceeds
Define the comparison before interpreting the value
Follow the evidence through the work
- Failure and incident examples
Define the unit, period and comparator before interpreting a change.
- Recovery and rework activities
Inspect this evidence for missing cases and incompatible definitions.
- Frequent rework
Keep this distinction visible when assessing the apparent value.
- A risk and investment sensitivity brief
Show assumptions and the observation that could reverse the decision.
Define the measurement
Define material failure modes and affected business activities. The opening step turns the operating concern into a question that can be examined. It would identify a failure event with a causal path, operational consequence and cost boundary, the outcome the owner cares about and the conditions that would make an observation decision-relevant. Scope changes would remain explicit if the evidence reveals a different problem.
Check the evidence
Map direct recovery, investigation and downstream consequences. The comparison material would include recovery and rework activities, with exposure, frequency and severity assumptions available to explain the operating conditions. Differences between observed loss and modelled exposure would remain visible beside the observation rather than disappear into a pooled result.
Examine alternatives
Separate observed cost evidence from frequency and severity assumptions. The review question would explicitly distinguish frequent rework from rare severe event. A material discrepancy would be traced back to failure and incident examples and the relevant execution or decision record, so a proposed change addresses the observed cause rather than the most convenient explanation.
Support the decision
Explore sensitivity and identify controls or data that change the decision. The resulting risk and investment sensitivity brief would explain the distinction between incident response cost and wider business consequence and identify what further observation would change the recommendation. The business and finance sponsor would retain the decision, including whether a narrower scope or a different approach is preferable.
Indicative artefacts
Use a risk and investment sensitivity brief to make the next action explicit
Every artefact needs an operating use
Illustrative working format
- An agent failure-cost taxonomy
- Boundary — failure and incident examples.
- A scenario and cost-driver model
- Findings about frequent rework.
- A risk and investment sensitivity brief
- Action on incident response cost.
The final artefact set depends on the agreed decision. Useful outputs can include:
- An agent failure-cost taxonomy. The working foundation for the question, including relevant definitions, boundaries and assumptions so another reviewer can understand what was examined.
- A scenario and cost-driver model. The substantive observations and interpretation, preserving adverse results, disagreement and limitations that affect how the evidence may be used.
- A risk and investment sensitivity brief. The connection to action: what the owner could do next, what evidence is still missing and what change would justify revisiting the conclusion.
Scope and authority
Measured evidence and assumptions have different roles in a funding decision
Keep analytical work separate from decision authority
- Defined measures and management analysis
- Can include an agent failure-cost taxonomy.
- Business and finance owner
- Finance and business owners validate assumptions and retain funding decisions.
- Audited returns or investment approval
- Cannot be inferred from a scenario and cost-driver model.
The proposed work supports management analysis and does not approve an investment or establish audited financial results. Prices, returns and realized benefits depend on actual evidence. Accounting, regulated advice and legal interpretations remain with qualified owners; authoritative financial and CRM records remain in their approved systems. To discuss failure-cost modelling, the first practical step is to identify the relevant owner and bring a sanitised description of the decision, the current alternative and the material uncertainty. That gives scoping a concrete starting point without implying a preapproved package or favourable conclusion.
Ways to engage
Related engagement scopes
AI Impact Measurement Framework
Define what better should mean before scaling an agent. Connect quality, cost, adoption and business outcomes to a baseline your team can revisit.
Continue exploring
Related insights and resources
Measure the work that improves
Connect agent activity to accepted business outcomes, establish a credible comparison, and distinguish useful capacity from realized financial value before deciding whether to scale.
Before the pilot: five questions for agent readiness
A compelling demo leaves important questions open. Use a five-dimension rubric to examine the data, authority, evidence, ownership and people behind your next agent initiative.
Where smaller models can strengthen agent architecture
Smaller models are not merely budget substitutes. Used behind explicit routing and escalation rules, they can improve latency, locality and architectural control.