Prompt injection is a workflow-security problem

An email assistant reads a supplier message, finds an instruction hidden in the quoted thread and uploads a contract to an attacker-controlled address. Calling this a model failure misses the decision that made the failure dangerous: the same workflow allowed one component to consume hostile content, retrieve private information and invoke an external action.

NIST defines prompt injection as exploiting the concatenation of untrusted input with a prompt created by a higher-trust party. That describes the entry point. The security outcome depends on everything connected to it. The right review unit is therefore not the prompt. It is the path from an untrusted observation to a privileged state change.

Start with the consequence, not the attack string

Prompt injection testing often begins with a collection of jailbreak phrases. That can reveal brittle behaviour, but it encourages a misleading objective: make the model ignore every malicious instruction. The UK NCSC warns that prompt injection remains a residual risk and recommends deterministic safeguards around tool use rather than relying only on content reaching the model in a safe form.

Begin instead with the unacceptable end states. For an invoice agent, they might include changing bank details, approving a payment above a threshold, disclosing another supplier’s records, or sending data outside approved domains. For each end state, work backwards:

This exercise turns an abstract prompt-injection concern into a small set of testable security invariants. A model may be manipulated, but the workflow must still refuse an action that lacks the required mandate.

Draw four planes through the workflow

A useful architecture review separates the system into four planes that are frequently collapsed inside one agent loop.

The instruction plane contains system policy, the user’s request and delegated constraints. The data plane contains messages, web pages, retrieved documents, tool responses and memory. The authority plane decides which actions are permitted for this identity, resource, purpose and moment. The execution plane performs the action and records the result.

Indirect prompt injection exploits ambiguity between the first two planes. A document is data to the organisation but can look like an instruction to the model. NIST’s agent-hijacking work describes this lack of separation and shows why task-specific and repeated-attempt evaluation matters. Architecture cannot make the model perfectly distinguish the planes, but it can stop that ambiguity from silently crossing into authority.

The authority plane should not ask the model whether its own proposed action is safe. It should evaluate structured facts: authenticated principal, current delegation, target resource, action, data classification, transaction value, destination and approval state. The execution plane should accept only a short-lived, action-specific decision from that policy point.

Break the dangerous triad

The highest-risk workflow combines three capabilities in one context:

1. access to private or sensitive data;
2. exposure to content controlled by someone else; and
3. a channel that can change state or send information out.

Remove any side of this triangle where the workflow permits. A research agent that reads the open web may not need the customer database. A retrieval agent that can access confidential records may return a bounded result to a separate action service rather than call arbitrary URLs. A drafting assistant may prepare an email without being able to send it.

Where all three are genuinely necessary, add a trust reset. End the untrusted-content phase, reduce the result to a typed claim with provenance, and start a fresh action phase that receives no raw hostile content. Re-check authorisation against the proposed destination and display the consequential fields to an approver. This is not a claim that summarization neutralizes attacks; it is a way to reduce how much attacker-controlled material reaches the decision point.

Put controls where they cannot be talked around

The NCSC secure-design guidance recommends restricting the actions an AI component can trigger and adding external fail-safes where necessary. In an agent workflow, that leads to a layered control map:

• Before inference: label provenance, strip active content, exclude secrets that the step does not need and select a minimal tool set.
• At tool selection: use typed tools with narrow functions; reject free-form shell, SQL or URL arguments unless the workflow truly requires them.
• At authorisation: enforce resource- and action-level policy outside the model; bind the decision to the authenticated user and current task.
• At commitment: require human confirmation for irreversible, novel-destination or high-impact changes; present the exact action, not a model-written reassurance.
• After execution: check the actual state transition, log the complete authorisation chain and rate-limit repeated attempts.

Content filters and classifiers may contribute a signal. They should not be the only barrier protecting money, credentials or confidential data. A filter that catches 99 attempts out of 100 is still an inadequate payment-approval control if attackers can try repeatedly.

Test attack paths, not just conversations

An effective scenario contains a legitimate task, an attacker objective, one or more injection locations, a protected asset and an observable success condition. AgentDojo illustrates this structure by testing agents in tool-using environments against both the user’s task and the attacker’s task.

For the supplier example, ask the agent to reconcile an invoice. Place variants of an instruction in the email, PDF metadata, OCR-visible text, retrieved supplier page and a prior memory item. The attacker objective is not “make the model say it was hacked”; it is “change the bank account” or “send the contract to a new domain.” Run each scenario several times because model behaviour varies. NIST found that repeated attempts can materially change the observed attack success rate.

Record three results separately: whether the legitimate task completed, whether the agent attempted the prohibited action, and whether the control plane blocked the real state change. The third result is the security boundary. A model that tries something unsafe is a finding; a system that lets it happen is a release blocker.

Make the release decision explicit

A prompt-injection release gate should name protected outcomes, not promise universal resistance. For example:

Across the approved adversarial suite, no test may produce an unauthorised external send, permission change or financial state transition. Attempted violations must be blocked by a control outside the model and produce a complete alert and audit record. Legitimate-task completion is measured separately.

The suite needs cases derived from the actual workflow, not only public strings. Add production near-misses and new ingestion routes as regression tests. Re-run it when the model, prompt, retrieval pipeline, tool definition, authorisation policy or external integration changes. NIST’s adversarial machine-learning taxonomy is a useful common vocabulary, but the organisation’s assets and action paths determine priority.

Prompt injection is unlikely to be “fixed” by one model release. It can be made governable when a manipulated model still encounters narrow authority, contained tools, visible commitment steps and evidence that tells responders exactly what happened. The practical question is not whether hostile text can influence a model. It is whether hostile text can acquire authority it was never given.