Why promising AI pilots become permanent experiments

The demo works. Users like it. The sponsor wants another presentation. Three months later, the pilot still has temporary access, hand-picked cases, volunteer reviewers and no owner prepared to run it.

This is not a mysterious “pilot-to-production gap.” It is the predictable result of designing an experiment to show that AI can do something rather than to support a decision about an operating workflow.

The OECD’s 2026 review of AI in government reports that limited impact measurement contributes to a proliferation of pilots with little potential to scale (Digital Government Outlook). The pattern extends beyond government: experimentation is easy to start when the operating obligations are deferred.

A prototype and a pilot answer different questions

A prototype explores whether an idea can work and helps reveal requirements. A pilot exercises a proposed service with representative users, data, integrations and controls so the organisation can make a bounded transition decision.

Confusing them creates false confidence. A prototype may use copied data, manual tool execution and a researcher standing beside every user. It can be useful evidence about desirability or technical possibility. It says little about identity, authorisation, live data quality, exception volume, support, recovery or unit economics.

The GOV.UK alpha guidance explicitly treats alpha as a place to test risky assumptions with prototypes that may be discarded; moving forward requires confidence in user need, cost-effectiveness, budget and people. Keep that separation. Do not turn a promising prototype into a perpetual quasi-service.

Five unresolved decisions keep pilots suspended

The workflow decision

The pilot has no precise statement of which cases will be served, what outcome matters or which current pain it should improve. Positive feedback becomes the only success signal, and scope expands whenever someone suggests another use.

The authority decision

The experiment relies on elevated test accounts or manual approval. Nobody has defined what the agent may read, recommend, change or communicate in live operation, or who can delegate and revoke that authority.

The evidence decision

The team reports output quality on curated examples but does not measure workflow outcomes, control compliance, review effort, failure recovery or impact on affected people. The sponsor cannot compare the pilot with the current process.

The ownership decision

The innovation team built the pilot, but the business function has not accepted product ownership. Security, data, operations and support are consulted late. Everyone can influence the pilot; nobody owns live performance.

The investment decision

There is funding for experimentation but no credible view of integration, evaluation, support, vendor, compute and change costs. The pilot therefore cannot compete for sustained operating investment.

Design backwards from four legitimate outcomes

Every pilot should end with one of four decisions:

1. Stop: the value is insufficient or a critical constraint cannot be resolved.
2. Revise: the hypothesis remains plausible, but the workflow, solution or control design must change.
3. Expand evidence: a defined uncertainty needs a broader or more representative test.
4. Prepare for live: the evidence justifies production engineering and operating readiness work.

“Continue the pilot” is not a fifth outcome unless it names the unresolved question, new evidence required, owner, timebox and decision date. Otherwise it is avoidance.

OECD guidance on structured generative-AI experimentation frames experimentation as a pathway for capability-building and risk management before deployment. The practical implication is that a pilot needs a learning contract, not an open-ended permission to explore.

Build a production-representativeness matrix

Record where the pilot resembles live operation and where it does not:

Dimension	Pilot condition	Intended live condition	Consequence of the gap
Users	trained volunteers	varied role proficiency	adoption and misuse unknown
Cases	curated normal cases	normal and exceptional mix	quality overstated
Data	copied snapshot	changing authoritative records	freshness and access unknown
Integration	mocked tools	external systems	side effects and recovery unknown
Authority	shared test credential	delegated least privilege	accountability untested
Review	researchers on call	operational queue	workload and delay unknown
Monitoring	debug traces	governed telemetry	incident detection unknown
Cost	trial credits	sustained volume	economics unknown

Do not require full production infrastructure to learn anything. Use the matrix to prevent inference beyond the evidence. If external effects are mocked, the pilot may support a recommendation-quality decision but not an action-safety decision.

Select the pilot population with the same care. A team of enthusiastic experts can tell you whether the interface is promising, but it will not reveal how occasional users interpret uncertainty or whether ordinary review queues can absorb the load. Include the roles and case variation needed for the decision, then state which groups remain untested. Where real users or live data would create disproportionate consequence, use staged simulations and shadow operation first. Representativeness is not an excuse to expose people to an immature service; it is a requirement to match each conclusion to the conditions actually exercised.

Assign the live owner before the pilot earns momentum

The future product or service owner should help define the hypothesis, acceptance conditions and operating model. That owner needs authority to prioritise work and accept responsibility for outcomes, not merely attend a steering meeting.

Name supporting accountabilities too: domain policy, data, technology operations, security, privacy, evaluation, incident handling, user support and commercial management. NIST calls for documented roles, executive responsibility, ongoing monitoring and safe decommissioning across the lifecycle (AI RMF Core). Those functions cannot be added as paperwork after a pilot is declared successful.

If nobody is willing to own the future workflow, that is decision evidence. Pause before technical enthusiasm creates an orphan service.

Measure benefits against a baseline and controls

Choose a small set of outcome measures tied to the workflow: completed cases, elapsed time, rework, customer effort, error consequence or capacity released for higher-value work. Add control and operating measures: inappropriate actions, overrides, unresolved exceptions, review time, incident rate, recovery outcome and cost per acceptable completion.

Establish the baseline before the pilot changes behaviour. Segment results by important case groups rather than relying on an average. Report uncertainty and missing evidence. The GOV.UK benefits guidance recommends agreeing measures early, comparing observed benefits with forecasts and using evidence to decide whether to continue, change or stop.

Finish with a transition packet, not a showcase

At closure, provide:

• the decision and accountable decision-maker;
• workflow boundary and excluded cases;
• hypotheses tested and evidence observed;
• comparison with the baseline and non-agent alternative;
• production-representativeness gaps;
• unresolved risks and dependencies;
• target operating model and ownership;
• estimated build and recurring operating effort;
• production evaluation and rollout plan; and
• disposition of pilot data, access and code.

“Prepare for live” authorises another body of work. It does not mean the prototype is production-ready. Production engineering must close the documented gaps and pass its own release decision.

The best pilot can end with a stop: it saved the enterprise from scaling the wrong workflow. Close the experiment cleanly, preserve its learning and release its temporary access. The worst pilot is one that remains impressive enough to preserve but too inconclusive to govern.