What evidence should survive an AI security incident?
Investigators need more than chat transcripts. Preserve identity, instructions, retrieved context, model and tool versions, policy decisions, credentials, network events and actual state changes.
After an agent sends a confidential document to the wrong destination, the first question is often “what did the model say?” That is only one link. Investigators must identify who initiated the work, which instructions and retrieved content influenced it, which model and configuration ran, why policy allowed the tool call, which credential was used, what crossed the network and what state changed.
If those records live in unrelated systems with incompatible clocks and retention, the organisation may know that harm occurred without being able to explain or contain it.
Build evidence around the causal chain
Use one correlation identifier from task admission through every agent, model call, retrieval query, policy decision, tool invocation, queue, network request and downstream transaction. Preserve parent-child relationships when the agent delegates or retries. A useful event identifies the actor, action, object, time, decision and result.
The evidence model should reconstruct:
initiator → instruction and context → model/configuration → proposed action → authorisation → credential and execution → external effect → monitoring and response.
NIST SP 800-61 Revision 3 integrates incident response into cybersecurity risk management rather than treating it as a late forensic activity. For agents, preparation means designing this chain before the first event.
Preserve eight evidence families
| Evidence family | Minimum useful record | Question it answers |
|---|---|---|
| Identity and delegation | Initiator, workload, parent task, approval, roles | Who authorised whom? |
| Instructions | System/policy version, user request, task plan | What was the intended job? |
| Context and provenance | Retrieval IDs, sources, trust labels, memory reads | What influenced the decision? |
| Model execution | Provider, model/version, parameters, timestamps, response ID | What component produced the proposal? |
| Policy | Rule/version, inputs, allow/deny, reason | Why was the action permitted? |
| Tool execution | Tool/version, validated arguments, response, retries | What was attempted? |
| Credentials and network | Issuance reference, audience, destination, byte counts | Through which authority and channel? |
| Business state | Before/after values, transaction ID, compensating action | What actually changed? |
Store software, prompt, policy and tool schema versions or immutable digests. Without them, replaying the text against today’s system may produce a different result and say little about the incident.
Capture provenance without creating a second breach
Full prompts and tool outputs may contain personal data, secrets, contracts or source code. Logging them into a broad observability platform can magnify exposure. Define content classes and choose a capture strategy: encrypted full content for narrowly authorised high-risk cases, field-level redaction, hashes for equality checks, or references to protected source records.
Preserve enough provenance to retrieve content lawfully and determine whether it was changed. Record source URI or object ID, version, tenant, ingestion time and trust classification. For web content, a digest and controlled snapshot may be necessary because the page can disappear or change; ensure collection respects applicable obligations and access controls.
The NCSC operational guidance for secure AI systems recommends monitoring system behaviour and inputs to support investigation and remediation, explicitly subject to privacy and data-protection requirements. That balance belongs in the evidence design, not an improvised incident decision.
Protect integrity and time
Send security events out of the agent’s write boundary. Use authenticated transport, access control, append-oriented or immutable storage, and monitoring for gaps or disabled collection. Synchronize clocks and record both event time and ingestion time; distributed agents and queues otherwise produce misleading sequences.
International best-practice guidance on event logging emphasizes centralized visibility and high-quality event logs for detecting activity across cloud and enterprise environments. For agent workflows, also preserve semantic identifiers—task, model run, tool call, policy decision—so centralization produces a narrative rather than a larger pile of records.
Hash or sign evidence bundles and record custody when material may support disciplinary, contractual, regulatory or legal processes. Technical teams should obtain appropriate specialist guidance for those requirements; a generic agent log is not automatically forensic evidence.
Access to evidence should itself be auditable. Separate routine service observability from the more sensitive incident corpus, require a case or purpose for content access, and record exports. Where engineers need to debug without seeing personal or confidential material, provide structured fields and redacted views first, with controlled escalation to protected originals.
Preserve volatile state before it disappears
Some evidence has seconds of life: in-memory plan, temporary workspace, short-lived token, network buffer, container filesystem or queue visibility timeout. Define automatic snapshots for high-severity triggers and a responder procedure that captures volatile state without unnecessarily delaying containment.
CISA’s cyber incident analysis guidance stresses separate storage, log integrity and detailed notes about observations and mitigation steps. Apply those disciplines to agent-specific state. Record why a snapshot was taken, by whom, with which tools, and which controls were altered during response.
Do not rely on the model to summarise its own prior behaviour. Its explanation may be incomplete, generated from a different context or influenced by the same hostile content. Preserve primary system events.
Test whether evidence answers incident questions
Run tabletop and technical exercises around a concrete scenario: a support agent reads an injected ticket, retrieves another customer’s document and sends it to a new address. Give responders the actual evidence store and ask them to determine:
- which user and workload initiated the task;
- where the malicious instruction entered;
- which protected records were available and accessed;
- which model, prompt, memory and tool versions ran;
- why authorisation allowed the destination and attachment;
- whether data left, exactly what, and through which channel;
- which other tasks share the affected component or credential; and
- what to revoke, preserve, notify and retest.
Measure completeness and time, not merely whether logs exist. Inject a missing service, clock skew, duplicated retries and partial provider telemetry. Confirm that the team can identify uncertainty instead of filling it with assumptions.
Run a second exercise after the normal retention threshold. This exposes evidence that exists during a fresh incident but disappears before delayed detection. If the business cannot justify retaining full content that long, ensure lower-risk metadata, integrity digests and downstream transaction records still support scoping. Record which questions will remain unanswerable and let the accountable risk owner decide whether that gap is acceptable.
Turn evidence into recovery and prevention
The NIST Generative AI Profile recommends incident response and recovery plans that account for the wider value chain and communications with downstream actors. Maintain provider and integration contacts, version records and contractual evidence access. If an external service retains essential telemetry for only a short period, automate collection or align response procedures to that window.
After containment, turn the attack input and state transition into a regression test. Correct authorisation, egress or workflow boundaries; rotate credentials; inspect related memories and tasks; and document residual uncertainty. Evidence earns its cost when it improves the next decision.
A release-ready agent should meet an evidence acceptance test: for every consequential action, an authorised reviewer can reconstruct the causal chain and actual end state without relying on the agent’s narrative; records are protected, correlated and retrievable; and the incident team has rehearsed using them. If the system cannot explain its actions after failure, it is not ready to receive authority that matters.