Why AI productivity claims need a baseline

An employee says an AI assistant saved two hours. A team reports that it processed twice as many cases. A vendor translates those figures into annual value. Each may be directionally useful, but none is a productivity result until it is compared with what would otherwise have happened.

A baseline is the measured starting condition. A counterfactual estimates the condition without the intervention during the same period. Both protect leaders from attributing every improvement—or every deterioration—to the AI system.

Define productivity at the workflow level

Productivity relates useful output to the inputs required to produce it. For agentic work, the output must include quality and completion, not drafts or clicks.

A practical family of measures is:

Labour productivity = accepted workflow outcomes / human hours.

Total operating productivity = accepted workflow outcomes / (human cost + technology cost + correction cost).

These answer different questions. A tool may reduce employee time while increasing platform cost. It may raise throughput while lowering quality. It may shift work from specialists to reviewers rather than remove it.

Choose one primary outcome and a small set of guardrails: quality, rework, time, cost, user or customer consequence, and critical failures. Document how each is observed.

Measure business as usual before it disappears

Capture the current process early. HM Treasury’s guidance on AI impact evaluation says the business-as-usual process should be precisely documented and baseline evidence considered during design. That urgency matters because training, anticipation and process cleanup can change behaviour before formal launch.

Sample representative cases and record:

• arrivals and case mix;
• completed and accepted outcomes;
• active work time and elapsed time;
• queueing and handoffs;
• review and rework;
• failure and escalation;
• cost and relevant downstream outcome;
• variation by team, experience and case type.

Do not reconstruct all of this from memory after deployment. Instrument what is proportionate and validate existing operational data against real cases.

Keep quality in the comparison

Speed without quality is often work moved downstream. A faster first draft may require more reviewer effort. A higher case-closure count may include reopened cases. A shorter call may leave the customer unresolved.

Create the same acceptance rubric for baseline and intervention. If the historical process lacks a recorded quality measure, independently review a sample. Use blind assessment where practical so reviewers do not know which method produced the work.

The well-known field study summarised by NBER on generative AI in customer support tracked duration, quality and outcomes and found effects varied with worker experience. Its numerical result belongs to that setting. The transferable lesson is measurement design: evaluate several outcome dimensions and look for heterogeneous effects.

Choose a credible comparator

A before-and-after chart is vulnerable to demand, seasonality, staffing, policy and learning changes. Stronger designs compare contemporaneous groups or use a phased rollout.

Options include:

Design	Useful when	Main caution
Random assignment	Access can be fairly staged	Contamination between groups
Phased rollout	Operations require sequential introduction	Time and team differences
Matched comparison	Similar untreated work exists	Unobserved group differences
Interrupted time series	Long stable history is available	Other events at launch
Theory-based evaluation	System and outcomes are complex	Causal magnitude is less direct

HM Treasury’s quality guidance for impact evaluation distinguishes observing an outcome change from attributing that change to an intervention, emphasising a robust counterfactual where causal claims are intended. Not every product decision needs a research-grade trial, but the claim language should match the design.

Account for adoption and assignment

Access to a tool is not the same as use, and use is not the same as workflow redesign. Track eligible users, activated users, actual use by task, training, manager expectations and whether cases were assigned differently.

An NBER field experiment across 66 firms and 7,137 knowledge workers found that users of an integrated tool spent less time on email in the latter half of the study, while the researchers did not detect a shift in the quantity or composition of tasks from individual-level provision (working paper). That contrast is instructive: a change in one activity does not automatically establish an organisation-level output gain.

Analyse both intention-to-treat—the effect of offering the intervention—and treatment-on-use questions, while avoiding cherry-picking only enthusiastic users. Adoption friction is part of the operating result.

Follow the released time

Time saved has several possible destinations:

• more accepted output;
• faster response or shorter queues;
• higher quality or deeper work;
• training, collaboration or resilience;
• lower overtime or improved wellbeing;
• unmeasured idle time;
• workforce reduction.

Each has different business and human consequences. Ask teams how work changed, then corroborate with outcome and workload data. The ILO’s review of empirical evidence highlights implications for work organisation, coordination, autonomy and job quality alongside productivity. A value case should not treat those as external to the measurement.

Do not automatically monetise every saved minute at salary cost. Time becomes cashable only under specific staffing and capacity decisions. It can still create valuable capacity, but name the benefit correctly.

Segment before averaging

Report effects by case complexity, worker experience, team, customer group and time since adoption where sample and privacy permit. A positive average can hide harm to one population or value concentrated among novices. An agent may accelerate standard work and slow exceptions because reviewers must untangle unusual outputs.

Predefine important segments. Avoid searching many slices after the result until one looks impressive. Report sample sizes and uncertainty. Include non-use and abandoned tasks.

Also monitor displacement: does the agent reduce work in one queue while creating escalations in another? A workflow boundary is often wider than the product dashboard.

Match the claim to the evidence

Use a claim ladder in reporting. “Users completed tasks faster in a controlled test” is narrower than “the team increased accepted output,” which is narrower than “the organisation realised financial value.” Each step needs additional evidence about quality, adoption, capacity allocation and cost.

Pre-register the primary outcome and comparison where the stakes justify it. Report null and adverse results alongside gains. If the design only supports association, say the change coincided with adoption rather than claiming the tool caused it. This discipline lets leaders combine early signals with appropriate caution instead of choosing between hype and silence.

Keep the baseline alive as the system changes

Models, prompts, tools, policies and users evolve. Preserve a versioned measurement record and maintain a comparator where feasible. Use a stepped expansion or periodic holdout for consequential changes, subject to ethical and operational constraints.

Set triggers for a refreshed evaluation: new population, expanded authority, major model change, workflow redesign, sustained drift, changed economic assumptions or material incident. A baseline from the pilot may no longer describe a mature service.

The defensible claim is rarely “AI improved productivity by X%.” It is closer to: for this workflow and population, during this period, the intervention changed these accepted outcomes and inputs relative to this comparator, with these uncertainties and distributional effects. That sentence is less convenient for a headline. It is far more useful for deciding whether to scale.