How to write an AI success story that survives scrutiny

Many AI success stories begin with a real piece of work and end with a claim the evidence never established. A prototype becomes a deployment. A faster model response becomes productivity. A positive participant comment becomes transformation. The story gets smoother as the evidential chain disappears.

A credible case study does the opposite. It shows what changed, for whom, under which conditions, compared with what, and how strongly the intervention contributed. It also has permission to disclose the facts it uses.

Separate activity, output, outcome and impact

Start by placing each proposed statement on an evidence ladder.

Level	AI example	What it can support
Activity	workshop held, agent configured, test run	proof that work occurred
Output	evaluation pack, deployed workflow, trained users	proof that something usable was produced
Experience	user understanding, satisfaction, confidence	evidence about participant response
Operational outcome	changed cycle time, error pattern, handoff rate	evidence about workflow performance
Organisational impact	cost, service, revenue, risk or mission effect	evidence about wider consequence

The AMEC evaluation taxonomy distinguishes activities, outputs, audience responses, outcomes and impacts. This is a useful guardrail beyond communications: a completed pilot is an output, not proof of scaled business value.

Write the strongest supported level, not the most impressive level. “The team deployed a supervised assistant and reduced median handling time in the observed pilot” is more credible than “AI transformed service operations” when the evidence is local and time-bounded.

Establish the baseline before celebrating the delta

A percentage improvement is uninterpretable without the starting measure, cohort, time period and calculation. Define what the previous process included: human preparation, exception handling, quality review, rework, delays and downstream correction.

Use a comparison suitable to the claim. That may be the same workflow before the intervention, a contemporaneous comparison group, stepped rollout, matched cases or a carefully described benchmark. Account for volume, case mix, staffing, seasonality and policy changes where they could explain the result.

The Magenta Book defines evaluation as systematic assessment of design, implementation and outcomes, and emphasises understanding what works, for whom and why. A case study need not become a research paper, but it should respect the same logic.

Build a claim-evidence rubric

Review every material claim before drafting the narrative.

Test	Weak	Stronger
Specificity	“improved productivity”	named measure, workflow, group and period
Baseline	informal recollection	versioned data from a defined comparison
Measurement	selected successful examples	declared method including failures and exclusions
Contribution	change happened after AI	causal chain plus plausible alternatives examined
Reproducibility	screenshot or testimonial	traceable calculation and controlled evidence
Scope	general enterprise claim	bounded conclusion for observed conditions
Permission	presumed approval	named approval for facts, identity, quotes and assets

Rate each claim as directly observed, derived, attributed opinion, hypothesis or unsupported. Remove unsupported claims; label hypotheses; and keep the source or calculation available internally for review. If confidentiality prevents enough context for a claim to be assessed, narrow the claim rather than decorating it with false precision.

Show contribution without pretending sole causation

AI initiatives occur alongside training, process redesign, staffing changes, policy updates and management attention. A case study should show the contribution path rather than assigning every improvement to the technology.

Map the chain: intervention, changed behaviour or process, immediate output, observed outcome and wider effect. Then list alternative explanations and evidence for or against them. The Magenta Book Annex A describes contribution analysis as testing a reasoned theory of change against observed results and other influencing factors when simple experimental attribution is not available.

Use calibrated language. “Contributed to” may fit when multiple changes worked together. “Was associated with” reports a relationship without claiming causality. “Caused” requires a design capable of supporting that conclusion. These distinctions make the story more useful to another organisation deciding whether conditions are comparable.

Include the operating conditions

Success usually depends on controls and human work omitted from marketing copy. State the scope of agent authority, human review, exception handling, model and system versions, data conditions, user training and duration of observation.

Include adverse and mixed findings. Perhaps routine cases improved while complex cases did not. Perhaps quality held only with added review effort. Perhaps one user group benefited and another faced more corrections. Reporting these conditions prevents a buyer from assuming the result will transfer unchanged.

UK Research and Innovation’s narrative impact guidance defines impact as demonstrable contribution beyond academic outputs and warns against repeating outputs as impacts. Its broader lesson is important: use evidence of how work was actually used, not merely evidence that it existed.

Treat permission as part of evidence quality

Accuracy does not create permission. Agree whether the organisation, individuals, quotes, screenshots, metrics and technical details may be disclosed. Record who approved the final wording and whether the permission expires or changes after a contract or role ends.

An anonymised story still needs care. A distinctive combination of sector, geography, workflow and metric may identify the client. Generalise non-essential details, but retain enough context for the finding to be interpreted. Do not invent a composite client or imply endorsement.

If permission covers only internal learning, keep the record internal. A public case study is not owed to the supplier or project team. The absence of permission is a publication constraint, not evidence that the work did not happen.

End with the decision the evidence supports

A success story should help another leader decide, not simply admire the result. Close with the conditions under which the approach is promising, the evidence still needed, and what happened next: scale, adapt, restrict, repeat or stop.

The latest UKRI guidance on demonstrating impact advises providing evidence of scale and significance and distinguishes a produced report from evidence that findings were used. This is a useful final check. Did the story document an artefact, a changed workflow or a wider result? Is the language matched to that level?

Give every quantitative claim a compact internal calculation note: numerator, denominator, inclusion rules, exclusions, source system, time window, analyst and version. For qualitative claims, retain the question, sampling approach, coding or synthesis method and contrary responses. The public story may stay readable while the internal evidence remains available for review.

Set a review date for time-sensitive claims. A pilot result can become misleading after the workflow, model, controls or user population changes. If an older story remains public, label the observed period and avoid present-tense wording that implies the result still holds unchanged.

Finally, let the client or affected organisation review factual and confidentiality issues without giving any party licence to erase unfavourable but material findings from an ostensibly analytical case study. If an agreed public version cannot remain accurate, do not publish it.

The most persuasive AI story is not the one with the largest number. It is the one whose claims remain intact when a sceptical reader asks for the denominator, baseline, method, alternative explanation, operating condition and permission. Scrutiny is not an obstacle to the narrative; it is what makes the narrative worth trusting.