Measure cost per quality-adjusted completion, not cost per token

A cheaper model call does not necessarily produce a cheaper service. One agent may use fewer tokens but trigger more retries, reviews and corrections. Another may cost more per run but resolve a larger share of work correctly on the first attempt. Token price describes a purchased input; product and finance leaders need the cost of an accepted outcome.

Cost per quality-adjusted completion joins the full workflow cost to a defined standard of completion. It is not a universal accounting metric. It is a decision model for comparing versions, routing choices and operating designs on equal terms.

Define a completion before counting cost

Choose the unit the workflow owner values. Examples include a customer case correctly resolved, a code change merged after review, an invoice exception reconciled, or a maintenance plan accepted by an authorised engineer.

Specify the acceptance conditions. A support case might count only when the answer is correct, the promised action occurred, no prohibited disclosure happened, and the customer did not return within a defined period for the same unresolved issue. If one of those conditions is absent from the data, label the measure provisional.

Do not use “agent task completed” merely because the orchestration framework emitted a success event. The external business state decides completion.

Build the complete numerator

For a measurement period, calculate total relevant cost:

Total cost = model + platform + tools + data + review + correction + operations + allocated costs.

The last term can include allocated evaluation, observability, security, support and implementation costs when the decision requires a fully loaded view. Keep direct and allocated costs separately visible so the result can serve both product optimisation and investment appraisal.

The GAO Cost Estimating and Assessment Guide recommends defining purpose and scope, building a technical baseline and work breakdown, documenting assumptions, performing sensitivity analysis and updating estimates with actuals. That is more useful for agent economics than multiplying tokens by a public list price.

Track costs at run level where practical. Include failed and abandoned attempts. Excluding them makes an unstable design look efficient.

Reconcile telemetry to invoices and labour evidence. A trace may estimate tokens while the supplier bills cached input, minimum capacity or a bundled tool differently. Reviewer time may be inferred from queue timestamps that include waiting. Sample the measurement path, document allocation rules and report missing cost coverage. Apparent precision in a cost formula is not a substitute for reliable source data.

Build a defensible denominator

The simplest metric is:

Cost per quality-adjusted completion = total relevant cost / accepted completions.

Accepted completions are completed cases meeting the predefined quality and control standard.

Suppose 1,000 cases enter the workflow. The system records 920 technical completions, but review finds 820 accepted outcomes. Total relevant cost is £12,300.

• cost per technical completion: £12,300 / 920 = £13.37;
• cost per accepted completion: £12,300 / 820 = £15.00.

The second measure exposes the economic effect of rework and quality loss. If 40 additional cases are later corrected at extra cost, decide whether they count as delayed completions and include all associated work. Apply the rule consistently across comparators.

For graded work, teams may use quality bands rather than a binary result. Do not invent a weighted “quality unit” simply to improve the chart. Publish the weighting and test whether the ranking changes under plausible alternatives.

Keep critical failures outside the average

A low CPQAC can coexist with an unacceptable risk. If an agent occasionally sends a payment to the wrong destination, averaging the expected loss into routine cost may hide a release blocker.

Use a guardrail structure:

1. critical authority, safety, security and privacy gates;
2. accepted-outcome definition for normal quality;
3. cost per accepted outcome;
4. separate expected-loss or stress scenario for residual failures.

The NIST AI RMF connects measurement to mapped risk and organisational tolerance. Economics informs the decision only after the proposed operating scope clears its non-negotiable controls.

Compare against a real baseline

The baseline should use the same completion definition. If the current human process measures closed tickets while the agent measures correctly resolved tickets, the comparison is biased before calculation begins.

Capture baseline variation by case type, complexity and worker experience. A field study summarised by NBER on generative AI in customer support found different effects across worker experience levels while tracking duration, quality and outcome. The general lesson is not to borrow its reported effect for another workflow; it is to examine heterogeneous outcomes instead of assuming one average applies to everyone.

Compare at least:

• current process;
• simpler rules or search improvement where plausible;
• agent with proposed controls;
• agent under lower-volume and higher-volume scenarios.

HM Treasury’s Magenta Book separates process, impact and value-for-money questions. Use all three: whether the system was implemented as intended, whether outcomes changed, and whether the benefits justify the costs.

Diagnose the metric rather than only reporting it

Decompose CPQAC into levers:

Lever	Diagnostic measure	Possible response
Model and context	Cost per attempt	Route simpler cases differently
Reliability	Attempts per accepted completion	Fix retries, tools or state handling
Review	Reviewer minutes per case	Narrow review using validated risk signals
Correction	Rework rate and minutes	Improve cases with highest defect contribution
Operations	Incident and support cost	Improve observability and recovery
Mix	Cost by case class	Re-scope tasks the agent should handle

The objective is not always to drive human review towards zero. Review can create value by preventing costly errors. Optimise the whole outcome, not one labour line.

GAO’s AI accountability framework places monitoring alongside governance, data and performance. A cost metric without quality and monitoring can reward a degraded system before its failures become visible.

Use distributions and sensitivities

An average hides long, expensive tails. Report median and high-percentile cost, especially where tool loops or manual escalation can expand. Separate ordinary cases from rare complex ones. Track the share of costs caused by the most expensive cases.

Run sensitivities for:

• model and platform price changes;
• increased review after an incident;
• lower success on a new cohort;
• volume above or below committed tiers;
• higher tool latency or retries;
• depreciation period for implementation cost.

If the preferred design changes under a small assumption shift, the decision is fragile. That is useful information for contract negotiation and staged rollout.

Give the metric an owner and response

Product should own the accepted-outcome definition with domain and risk input. Finance should review cost allocation and assumptions. Engineering should preserve run-level telemetry. Operations should validate review, correction and incident effort. No single function can produce a credible measure alone.

Define what happens when CPQAC rises: investigate case mix, compare quality, inspect dependency or pricing changes, and decide whether to tune, reroute, renegotiate or narrow scope. Never respond by quietly weakening the quality denominator.

Review the numerator and denominator together. A rising cost may be acceptable if a stricter quality gate prevents costly downstream failure; a falling cost may be bad news if reviewers stopped recording rework. Keep a bridge from the previous period that attributes movement to price, volume, case mix, attempts, review and acceptance rate.

Token cost remains useful for invoice reconciliation and component optimisation. It becomes strategically useful only when connected to the work the organisation intended to complete. Cost per quality-adjusted completion creates that connection—and makes clear when apparently cheap intelligence is creating expensive operations.