Running an AI-agent incident tabletop exercise

An AI-agent incident tabletop should not begin with “the chatbot says something offensive.” Begin with a workflow losing control: untrusted content redirects an agent, a credential permits unintended action, retries duplicate transactions, or a provider change causes a policy regression.

The exercise is a structured conversation, not a penetration test. Its purpose is to expose decisions, dependencies and evidence gaps before people must resolve them under pressure. Pair it with technical drills for controls the discussion assumes will work.

Choose one scenario with real consequences

Select a deployed or planned workflow and trace what the agent can observe and change. Build the scenario around a plausible failure mechanism and business consequence. Avoid a catastrophe so broad that every answer becomes “shut everything down.”

Example: a procurement agent reads a supplier document containing hostile instructions, uses its service identity to alter an approved order, and sends data to a new destination. Monitoring shows unusual calls, but the final user-facing message looks normal.

This scenario tests content trust, identity, authorisation, egress, evidence and supplier communication. The OECD AI incidents work emphasizes consistent incident concepts and reporting so organisations can learn across cases; a clear scenario definition supports the same goal internally.

Invite decision-makers, not an audience

Include the workflow owner, product/engineering, security operations, identity or platform engineering, privacy/data, legal or compliance as relevant, business continuity, communications and the help desk or operations team that will see the first symptom. Assign an executive incident lead.

State the exercise boundaries and professional roles. A facilitator presents information; participants make the decisions they would own. An evaluator records observed strengths, gaps and unanswered questions. Do not let the most technical attendee make every business choice.

The CISA tabletop package guidance provides reusable structures for planners, facilitators/evaluators and participants. Adapt the structure to the agent workflow rather than inventing an entirely new exercise discipline.

Build injects around decisions

Reveal the incident in stages. Each inject should force a choice or expose an assumption:

Inject	Decision to test
Anomalous tool sequence appears	Who validates and declares an incident?
Similar actions found in queued work	Can one capability be suspended without stopping everything?
Provider logs are delayed	What evidence exists locally, and who escalates?
One external transaction completed	How is business state restored or compensated?
A customer asks what happened	Who approves a factual communication?
Patch passes a small retest	What permits safe return to service?

Include ambiguity. Real teams rarely receive a perfect classification at the start. But do not hide essential architecture facts participants would reasonably know.

Test containment at the level of authority

Ask participants to identify the affected agent identity, credentials, tools, queues and active sessions. Can they revoke one token, disable one action, block one destination or isolate one tenant? What happens to in-flight work?

Stopping model calls may not stop tool execution. Rolling back an internal deployment may not reverse an email, payment or supplier order. Require a plan for partial external actions, including reconciliation and compensating action.

The NCSC secure AI guidance links secure deployment with incident management and secure operation with logging, monitoring and update management. A tabletop makes those lifecycle dependencies visible.

Preserve evidence without expanding exposure

List what responders need: system and policy version, initiating identity, retrieved-content references, model/tool trace, authorisation decision, arguments or protected digest, state changes, error/retry events and human interventions. Ask where each item lives, how long it remains available and who may access it.

Avoid a reflex to copy every prompt and private record into an unrestricted incident channel. Preserve integrity and chain of handling while minimizing secondary disclosure. Record clock differences and provider evidence gaps that may complicate a timeline.

NIST SP 800-61 Rev. 3 integrates preparation, detection, response and recovery with broader risk management. The exercise should test all of them, not stop after containment.

Rehearse communication and recovery

Prepare an internal situation report that separates known facts, working hypotheses, affected scope, actions taken and decisions required. Test escalation to executives, providers, customers or authorities according to applicable plans; legal notification duties require qualified assessment, not a generic AI rule.

Return-to-service criteria should be more demanding than “the prompt was fixed.” Identify root-control changes, credential rotation, affected-state reconciliation, regression cases, monitored rollout and the executive who accepts remaining uncertainty.

Ask what service continues while the agent is constrained. A manual fallback that no longer has trained staff is not a recovery capability.

Turn observations into owned improvements

Hold a short hotwash, then produce an improvement plan. Classify findings across people, process, architecture, telemetry, evidence and external dependency. Give each action an owner, date and acceptance evidence.

Distinguish discussion evidence from technical evidence. The tabletop may show that teams know to revoke credentials; a drill must show that revocation reaches live sessions and queued jobs. Retest material gaps and preserve the scenario as a future regression exercise.

Success is not a meeting without confusion. It is discovering confusion while consequences are simulated, then proving that the organisation improved before the real incident.

Use a compact exercise package

A practical package can be prepared in six artefacts. The scenario brief defines the workflow, architecture and assumed normal controls. The participant guide states objectives, roles and rules. The facilitator guide sequences injects and offers optional prompts without prescribing answers. An evidence sheet lists the logs, identities and system views participants may request. An evaluator rubric records decisions and observable behaviour. The improvement log converts findings into owned work.

Set two or three objectives, such as declaring and scoping the incident, containing one capability without unnecessary business shutdown, and approving return to service based on evidence. Too many objectives turn the exercise into a tour of every policy.

Use a simple evaluation scale tied to observation: demonstrated, partially demonstrated, not demonstrated or not testable in a discussion. “Not testable” is important. It identifies where a technical drill must replace conversation. Record time to key decisions, but do not treat speed as success if the decision lacks authority or evidence.

After remediation, run a focused replay. Give the team the same initial symptom but change one dependency—for example, provider logs are unavailable or the malicious instruction came through a trusted internal knowledge base. A control that works only with perfect provenance is fragile.

Store the exercise record with access appropriate to its security and personal data. Share sanitised lessons more broadly so product and platform teams can improve. The goal is a reusable organisational capability, not a confidential report that disappears after the executive readout.

Exercise design should also guard against theatrical success. Do not brief participants on every inject or allow the facilitator to supply missing ownership. Let teams request evidence through the channels they would use in reality. If access takes thirty minutes or an approval contact is unknown, record that friction.

Include at least one inject that challenges the initial diagnosis. A suspected prompt injection may turn out to be a compromised connector, an insider-approved permission or a normal retry interacting with a non-idempotent tool. The point is not to trick participants; it is to test whether they update hypotheses as evidence changes and avoid communicating an unverified cause.

The tabletop succeeds when the organisation leaves with fewer assumptions about who decides, what stops and which evidence survives—and then proves those improvements outside the meeting.