What should appear on an AI impact dashboard?
Give decision owners a balanced view of outcomes, quality, human work, cost, risk and change—with every measure tied to an action.
An AI dashboard can be full of motion while saying little about impact. Prompts, active users, tokens and completed runs describe activity. They do not show whether the workflow improved, whether quality held, whether people inherited hidden review work, or whether risk remained within the accepted boundary.
The right dashboard begins with a decision. Is the owner deciding whether to expand a cohort, change authority, fund another phase, remediate a quality problem or stop the service? Every prominent measure should contribute to that choice or explain a change in it.
Put the outcome at the top
Choose the unit of value the workflow exists to produce: correctly resolved cases, accepted changes, reconciled exceptions, completed inspections, or another externally verifiable state. Show count, rate and trend against a baseline or comparator.
A leadership summary for a support agent might begin:
| Decision signal | Current period | Comparator | Decision threshold | Owner response |
|---|---|---|---|---|
| Accepted resolutions | 8,420 | 7,900 baseline-adjusted | ≥ baseline | Product review if below |
| First-contact resolution | 74% | 70% | ≥72% | Inspect case mix and defects |
| Reopen rate | 6.1% | 6.4% | ≤6.5% | Quality sampling |
| Critical prohibited actions | 0 observed | 0 | 0 in defined set | Suspend authority if observed |
| Cost per accepted resolution | £X | £Y | Approved range | Finance/product review |
Illustrative labels should be replaced with the organisation’s real measures and approved thresholds. A metric without a response is decoration.
GAO’s review of procurement performance practices highlights linking measures to strategic goals, developing them with users and favouring outcome-oriented measures such as timeliness and quality. Those principles translate directly to an AI impact view.
Show quality beside volume
Do not allow throughput to outrun its quality denominator. Include accepted-outcome rate, rework, appeal or reopen rate, critical defects, and a calibrated sample review. If quality is graded, show the distribution rather than only an average.
Make sampling visible: period, population, sample size, selection method, grader and uncertainty. A green score based on twenty convenient cases should not look equivalent to a representative review.
Separate customer-facing quality from policy compliance and technical reliability. A response can be helpful but unauthorised; a correct response can arrive after the decision is no longer useful.
Expose the complete human system
Track human minutes for review, correction, escalation and recovery. Show queue length, handoff quality and time to decision. Segment by role and case type so a productivity gain in one team does not hide work transferred to another.
The ILO’s review of generative AI evidence considers work organisation, coordination, autonomy and job quality alongside productivity. Use a small set of workforce indicators relevant to the design: interruption burden, override rate, training coverage, or an appropriately governed worker-experience measure.
Do not treat lower human involvement as automatically better. The target is the right involvement for the consequence, with capacity that matches arrivals.
Put cost on the same outcome basis
Show model, platform, tool and storage cost, but add review, correction and relevant operating cost. Divide by accepted outcome. Report median and high-percentile run cost where loops or long context create a tail.
Keep investment and recurring operation distinct. A dashboard for a pilot may show implementation burn and forecast; a mature service should show actual versus forecast, cost per accepted outcome, volume-tier exposure and material variance.
Add a diagnostic view by case class, model route and release. This lets teams distinguish price change from degraded quality or increased case complexity.
Include guardrails and control health
The dashboard should surface conditions that can change authority even if average value looks positive:
- prohibited actions or data accesses;
- unresolved critical findings;
- incidents and near misses;
- monitoring coverage and evidence gaps;
- rollback or kill-switch readiness;
- policy exceptions and expired approvals;
- material changes awaiting reevaluation;
- drift in important cases or cohorts.
The NIST AI RMF connects risk measurement, production monitoring and management responses. Reflect that chain: each guardrail needs a source, threshold, owner and action. Avoid collapsing incomparable risks into one “trust score.”
Segment impact so averages do not hide distribution
Show outcomes by the populations and case types that can experience different effects. Relevant segmentation might include language, channel, complexity, location, user role or experience. Apply privacy and minimum-cell controls; not every slice belongs on a widely accessible dashboard.
HM Treasury’s AI impact guidance calls for intended and unintended outcomes, variation between groups, and a clearly defined baseline. A dashboard can monitor these dimensions, but monitoring alone does not prove causality.
Label the strength of comparison. “Versus last month” is a trend. “Versus matched control” can support a stronger claim. Preserve study design and uncertainty outside the chart rather than implying every delta is caused by AI.
Track change, not just state
Every result needs configuration context: workflow version, model, prompt or policy release, retrieval source, tool version and autonomy level. Mark significant changes on trend lines. Otherwise teams will debate whether a movement reflects behaviour, case mix or instrumentation.
Include leading indicators for planned change:
- percentage of traffic on a new release;
- regression pack status;
- unresolved evaluation gaps;
- changed dependencies;
- expiry date for conditional acceptance.
The dashboard should enable a click from aggregate signal to evidence: affected cases, traces and decision record, with access controlled appropriately.
Balance leading and lagging measures. Evaluation coverage, unresolved findings and review queues can warn that control is weakening before customer outcomes move. Reopens, incidents and realised cost confirm consequences later. A leading measure should be validated against the outcome it is meant to predict; otherwise it can become another activity target that teams optimise without improving the service.
Use a layered dashboard rather than one crowded screen
Structure information by decision altitude:
- Executive view: outcome, quality, total cost, critical risk and decision state.
- Product view: adoption, case mix, funnels, review, rework and cohorts.
- Operations view: latency, failures, queues, recovery and incidents.
- Evaluation view: test coverage, uncertainty, graders, findings and releases.
- Finance view: actual versus forecast, unit economics and sensitivities.
The definitions must reconcile across layers. A leadership number should drill down to the same accepted cases finance and product use, not a separately curated total.
The Magenta Book distinguishes process, impact and value-for-money evaluation. The layered view keeps those questions connected without pretending they are one metric.
Govern the dashboard itself
Create a metric register with definition, formula, source, owner, refresh, quality check, baseline, threshold and known limitation. Version it. Restrict access to sensitive operational or personal data. Test for missing records after outages and pipeline changes.
Hold a recurring decision review, not a slide-reading meeting. Ask:
- What changed, for whom, and after which release?
- Is it signal, case mix or measurement failure?
- Which threshold was crossed?
- What action, owner and due date follow?
- What evidence would reverse the decision?
Retire metrics that never inform action. Add a measure only when it answers a new decision or closes a known blind spot.
An impact dashboard should make it harder to confuse usage with value and averages with assurance. Its success is not the number of green tiles. It is whether accountable owners can see the outcome, understand the trade-offs, and act before a weak signal becomes an expensive operating fact.