How to test whether an AI agent recovers gracefully

An agent that succeeds after a tool timeout may look resilient. It may also have charged the customer twice.

Graceful recovery is not “eventually produced an answer.” It is the ability to detect disruption, determine what happened, limit further consequence, restore or reconcile state, and give a person a usable next step when automation cannot continue safely.

Test that behaviour deliberately. Waiting for production to supply failure cases is expensive, and ordinary happy-path evaluation rarely exercises the uncertain states where agent workflows become dangerous.

Define recovery outcomes before injecting faults

For each workflow step, identify its failure semantics. A read can often be retried. A write may have committed even when the response is lost. A multi-step transaction can leave one system updated and another unchanged. A retrieved document can become stale before action.

Classify the required recovery:

• retry: repeat a safe, idempotent operation within a bound;
• fallback: use an alternate tool or reduced function;
• compensate: apply a deliberate action that offsets completed work;
• rollback: restore a known prior state where the system supports it;
• reconcile: query authoritative systems and resolve uncertain state;
• degrade: continue with less authority or capability;
• hand off: transfer to a person with context and control;
• stop: prevent further action and preserve evidence.

NIST defines cyber-resilient systems as able to anticipate, withstand, recover from and adapt to adverse conditions in SP 800-160 Volume 2. Those verbs are a useful test outline. Recovery evidence should show more than endurance: it should show safe state and adaptation.

Build a fault catalogue around real boundaries

Map faults to components and transitions, not generic prompts.

Boundary	Faults to inject	State risk
Model call	timeout, refusal, malformed tool request	abandoned or unsafe plan
Tool call	error, slow response, wrong schema, partial commit	duplicate or corrupt action
Retrieval	empty, stale, conflicting, hostile content	decision on bad context
Memory	unavailable, duplicated, poisoned, expired	wrong continuity or leakage
Authorisation	expiry, revocation, scope mismatch	unauthorised execution
Orchestrator	restart, queue duplication, lost event	repeated or orphaned work
Human handoff	unavailable reviewer, delayed response	silent stall or bypass

Include combinations. A timeout plus a stale read can expose a different behaviour from either alone. Start from plausible production incidents and architecture analysis, then add rare high-consequence cases.

The ToolEmu research demonstrates how emulated tools can help search a large risk surface. Emulation is useful for exploration; release-critical recovery paths should also run against representative integrations or high-fidelity sandboxes so transaction behaviour is real.

Observe state outside the agent

The agent’s account of recovery is not authoritative. Capture external state before, during and after the fault. Record tool requests, idempotency keys, authorisation results, commits, retries, compensations, messages and handoff events.

Define invariants such as:

• no protected action without current authorisation;
• at most one committed transaction for one user intent;
• the final message never claims certainty the system cannot establish;
• partial completion is reconciled or clearly handed off;
• retry and cost limits are enforced outside the model;
• evidence survives even if the orchestrator restarts.

State-based evaluation is supported by agent benchmarks such as tau-bench, which checks the final database state and repeated reliability. Your workflow needs additional intermediate-state checks because a final state can hide a temporary disclosure, unauthorised call or compensation that arrived too late.

Grade the whole recovery journey

Score recovery across six dimensions rather than a binary completion result:

Dimension	Passing evidence
Detection	Fault or uncertainty is recognised promptly
Containment	No unnecessary action occurs after the fault
State integrity	Commits, duplicates and partial states are known
Restoration	Intended state is restored or safely reconciled
Communication	User and operator receive accurate status and next step
Learning	Trace becomes an incident or regression candidate

A retry that completes but violates containment is not graceful. A handoff that preserves state but gives the operator no case identifiers is not operationally complete. A safe stop can be a pass when continued autonomy would be reckless.

Use deterministic checks for state and limits. Use a rubric for handoff quality. Measure time to detect, time to contain and time to a stable disposition, not only total latency.

Test bounded autonomy under uncertainty

Recovery is where agents are tempted to improvise. Restrict which recovery actions are available and under what authority. The model can propose a compensation; a policy layer should determine whether it may execute it.

Test whether the agent:

• asks for clarification instead of guessing critical state;
• checks authoritative state before retrying a write;
• respects a maximum step and retry budget;
• avoids switching to a more privileged tool as a convenience;
• does not conceal errors to satisfy the user;
• stops when approval is missing or expired;
• hands off before context becomes too uncertain.

The NIST AI RMF includes response, recovery, incident management and change management in post-deployment risk treatment. A recovery policy should therefore be testable before deployment and monitored after it.

Make the human handoff an executable control

“Escalate to a human” is incomplete unless a person can receive, understand and act on the case. Test the queue, role, availability, access and response target.

The handoff package should include:

• user intent and verified identity context;
• actions attempted and authoritative state observed;
• fault and uncertainty, without fabricated diagnosis;
• approvals already obtained and their validity;
• actions that must not be repeated;
• recommended safe next step;
• trace and audit identifiers.

Run handoff exercises with the people who will receive them. Measure rework and unsafe repetition. If the reviewer cannot distinguish “not attempted” from “possibly committed,” the automation has transferred confusion rather than control.

Exercise shutdown, rollback and restoration

Test the controls operations will depend on: disable a tool, revoke credentials, halt new work, drain active tasks, restore a known configuration and replay or reconcile outstanding items. Confirm that shutdown does not leave a hidden queue continuing to act.

The NCSC secure AI development guidance calls for incident processes, logging, monitoring, update management and restoration to a known good state. For an agent, that means testing both application controls and external authorities. Removing a prompt permission is not enough if a long-lived credential can still call the tool.

Turn every confirmed failure into a stronger case

After a fault exercise, separate the system defect from the test discovery. Record configuration, trigger, state transition, consequence, detection, treatment and regression case. Add the case to a controlled suite with the expected recovery path and invariants.

Do not let the suite become a museum of identical timeouts. Group cases by recovery property and keep variants that expose distinct state risks. Periodically introduce new combinations so the system is not merely tuned to known scripts.

Graceful recovery is evidence that the system knows when normal execution has ended. The strongest result is sometimes a completed transaction; sometimes it is a clean stop with honest uncertainty. The evaluation should reward the path that preserves authority and state, not the one that produces the most confident closing sentence.