What a polished agent demo does not tell you
A demonstration shows one possible path. A buying decision needs evidence about representative work, failure, authority, recovery and operation.
Evaluation & assurance, Strategy & transformation Product and value
A good agent demonstration is designed to make a capability understandable. It is usually fast, coherent and visually satisfying. Those are virtues in a demonstration—and reasons not to confuse it with evidence for a buying decision.
The demonstrated path is one chosen configuration, with prepared data, a cooperative user and a known destination. Production contains ambiguous requests, stale records, permissions, timeouts, policy edges, hostile content and people who do not behave like a script. A demo can reveal what might be possible. It cannot establish what will happen often enough, safely enough or economically enough in your workflow.
Separate possibility from repeatability
The first successful run answers a narrow question: can this configured system complete this presented path at least once? Buyers also need to know whether it repeats across relevant variation.
Ask the demonstrator to rerun the same task several times, then vary one condition at a time: the wording, account state, document order, missing field, tool latency or user role. Observe whether the end state remains acceptable and whether the path stays within policy. Do not demand identical wording from a non-deterministic system; demand consistent satisfaction of the important requirements.
The NIST AI RMF calls for performance to be assessed under conditions similar to deployment and for limitations on generalisation to be documented. A controlled demo rarely resembles the full deployment distribution. It is a starting sample, not the population.
Find the work hidden outside the happy path
A seamless interaction can conceal preparation and cleanup. Ask what was configured before the meeting and what a person would do after the screen goes dark. Common hidden work includes curating documents, correcting records, creating tool permissions, approving actions, resolving duplicates, reviewing outputs and repairing incomplete transactions.
Turn the visible task into a work ledger:
| Stage | Questions to ask | Evidence to request |
|---|---|---|
| Prepare | Who cleans data and configures policy? | Setup steps, elapsed effort, prerequisites |
| Execute | Which models, tools and people participate? | Versioned trace and authority decisions |
| Verify | How is correctness established? | Final-state checks and reviewer rubric |
| Recover | What happens after timeout or partial action? | Retry, rollback and reconciliation trace |
| Operate | Who monitors changes and incidents? | Runbook, service boundaries and escalation |
The point is not to catch a supplier hiding labour. It is to identify the actual operating design. If human review is essential, include it in cost, capacity and quality measures rather than describing the agent as autonomous.
Ask the system to say no
Agent quality is partly visible in what the system declines, escalates or leaves unchanged. Give it a request from an unauthorised user, an action beyond its limit, conflicting instructions, an unsupported exception and a document containing instructions that should not control the workflow.
Observe enforcement, not explanation. A fluent refusal followed by a prohibited tool call is a failure. A cautious message paired with an unnecessary disclosure is also a failure. Inspect the external state—the record, transaction, message or permission—not just the conversational response.
The NIST Generative AI Profile describes risks including confabulation, information integrity, privacy and security across the lifecycle. A buyer does not need to test every abstract risk in a sales session. It does need cases linked to its own data, actions and consequences.
Introduce an ordinary failure
Ask what happens when a tool times out after receiving a request, when a dependency returns malformed data, or when the user leaves midway through an approval. These are not theatrical attacks; they are normal distributed-system conditions.
A credible agent should have a defined response: retry safely, check the final state, compensate, hand off or stop. The demo should show which state the system believes it is in and how an operator reconciles that belief with reality. If the only recovery is “restart the demo,” the buyer has learned something important.
Request one end-to-end trace with timestamps and version identifiers. Can the supplier distinguish planning from execution, show the enforced permission, identify the tool result and connect the run to its final business state? The GAO AI accountability framework organises accountability around governance, data, performance and monitoring. Those dimensions are useful prompts because a polished interface can obscure all four.
Replace vendor data with representative conditions
A supplier’s sample corpus is optimised for legibility and availability. Your environment includes access controls, duplicate records, local terminology, missing history and retention rules. Move quickly to a buyer-controlled test set whose use is authorised.
Include four classes:
- frequent ordinary cases that determine capacity and user experience;
- boundary cases around policy, permission and eligibility;
- rare high-consequence cases that influence authority;
- faults and recoveries that reveal operating resilience.
Preserve some cases from tuning. If the supplier sees and iterates on every example, the trial may show adaptation to a test pack rather than general performance. Document any exclusions so a narrow trial does not quietly become a broad product claim.
UK AI procurement guidance recognises that system functionality and consequences may only become apparent during deployment, which makes ongoing information exchange important. This means the trial should also test whether the supplier can support continued evaluation, not merely win the initial comparison.
Measure an outcome that matters to the workflow
Avoid making speed and fluency the headline measures. A fast run that requires correction or creates a bad state is not productive. Choose an outcome that the workflow owner already cares about: a correctly resolved case, an accepted code change, a valid reconciliation, or a complete handoff.
For each scenario, record at least:
- acceptable final state;
- prohibited state or action;
- human effort, including correction;
- elapsed time and tail behaviour;
- cost for the complete attempt;
- whether recovery and evidence requirements were met.
Impact claims need a baseline. HM Treasury’s guidance on evaluating AI interventions emphasises early baselines, intended and unintended outcomes, and iterative evaluation. Compare the candidate with the current process and, where plausible, a simpler automation—not just with another agent.
Make the supplier’s operating promises testable
Translate statements into conditions. “Enterprise security” becomes named identity, access, encryption, logging and incident obligations. “Human in the loop” becomes a specified decision point with context, response time and override behaviour. “Model agnostic” becomes an exercised change using a second supported model. “Easy export” becomes a sample export and reconstruction attempt.
Record which claims were directly observed, which were supported by documents, which depend on contract, and which remain assumptions. This protects both buyer and supplier from a meeting-room impression becoming an unqualified commitment.
End the demo with a decision-shaped trial
The next step after a promising demo should not be “show us more.” It should be a bounded evaluation statement:
Decide whether the candidate may support this named workflow for this cohort and authority, by testing these representative cases against these outcome, control, recovery and operating criteria.
Agree access, evidence, responsibilities, data handling, timebox and stop conditions before the trial. A result can support proceeding, narrowing scope, remediating, or declining. If the trial is structured so that only purchase counts as success, it is still a demonstration.
The strongest sales experience is not the one that eliminates uncertainty for an hour. It is the one that helps the buyer identify the remaining uncertainty, test it fairly, and understand what responsible operation would require.