The dangerous combination of private data, untrusted content and external actions

An agent asked to “prepare a response to this customer” may need the customer record, the incoming message and an email tool. That ordinary description hides a powerful attack path. The record is private, the message is controlled outside the organisation, and the email tool can move information beyond the boundary. If all three meet in one model context, a malicious instruction in the message can try to turn legitimate access into exfiltration.

This pattern deserves its own architecture gate. It appears in email, browsers, ticketing, procurement, code assistants, document review and any workflow where an agent reads from one trust domain and acts in another.

Recognise the three capabilities

The first capability is valuable data: personal information, credentials, source code, contracts, pricing, internal messages or operational state. The second is untrusted influence: web content, email, uploaded files, retrieved documents, tool metadata or any text an adversary can alter. The third is external effect: sending a message, posting to a URL, changing a record, executing code, initiating a transaction or writing durable memory.

The combination matters because it joins motive, steering and means. NIST’s agent-hijacking research describes attacks in which malicious instructions are placed in resources an agent encounters while completing a legitimate task. MITRE ATT&CK documents how legitimate web services can be used for exfiltration, often blending into ordinary encrypted traffic. An agent does not need a novel exploit if its approved tools already form the route.

Do not limit “external” to the public internet. A write into a shared knowledge base, a new calendar invite or an internal message can cross a meaningful security boundary and create a second-stage attack.

Inventory the dangerous intersections

Map each agent step rather than assigning one risk label to the whole product. A simple matrix exposes combinations that architecture diagrams often obscure.

The key design move is temporal separation. The agent does not need every capability at every step. Convert a monolithic loop into stages with smaller contexts, credentials and tool sets. The result may be less “autonomous,” but it is more understandable and testable.

Minimise what crosses each boundary

Separating steps is useful only if the boundaries reduce power. Pass the comparison stage a specific contract and supplier message, not a search token for the entire document repository. Return structured fields—disputed amount, cited clause, confidence and provenance—rather than the complete private context. Give the drafting step the minimum customer facts it must communicate. Give the send service an approved payload and destination, not a general email credential.

This follows the NCSC’s secure-design advice to consider the sensitivity of data, restrict actions triggered by AI components and use external fail-safes where necessary. Minimisation also makes incident analysis more credible: responders can identify which data was available to each compromised step rather than assume the entire environment was exposed.

Treat model-written summaries as untrusted derived data, not a sanitization boundary. Preserve provenance so the next control can distinguish a user-approved value from one copied out of an external document. If a field will drive a consequential action, validate its type, allowable values and relationship to the authenticated task.

Give external actions a separate mandate

An action service should ask, “Is this principal allowed to perform this exact action on this resource now?” It should not infer authority from a natural-language plan. Bind the answer to the user or workload identity, approved purpose, target, data classification and transaction limits.

For example, a support agent may draft refunds but the payment service should enforce customer, order, amount and daily limits. A changed destination, new recipient, executable attachment or access-control modification should trigger a stronger path. A person approving the action should see the exact fields that will be committed, with suspicious content visually separated from trusted interface text.

Approval is not a universal remedy. If every routine action generates a dense prompt, people will learn to accept it. Reserve approval for meaningful boundaries, reduce frequency through narrow standing policy, and reject actions that policy should never allow.

Contain communications as carefully as execution

Teams often sandbox code while leaving network access broad. That is an incomplete boundary. A contained process with unrestricted HTTPS can still disclose data to an attacker-controlled service. Use destination allowlists where the workflow permits, resolve and re-check destinations at the enforcement point, restrict protocols and methods, cap request size, and prevent the agent from supplying arbitrary callback or webhook URLs.

Proxy outbound traffic through a service that can apply identity-aware policy and record destination, method, volume and decision. Watch for new domains, unusual upload volume and destinations that differ from the user’s named task. Encryption protects traffic in transit; it does not establish that the destination is authorised.

For tools that legitimately communicate with many destinations, build a data-release rule: classify the payload, remove secrets, bind the release to an explicit purpose and retain evidence of what left. If those controls cannot be expressed, the agent may be unsuitable for unsupervised use in that workflow.

Test the complete route to harm

A useful red-team case gives the agent a legitimate task and places an instruction in a realistic hostile source. The success condition is a state change: protected content reaches a test endpoint, a record is modified, or a new recipient is added. Vary the source, encoding, placement and number of attempts. Test direct action and multi-step routes through memory, summaries and other agents.

OWASP’s agentic threat guidance frames risk across planning, memory, tools, identity and multi-agent interaction. Use that breadth, but keep each test tied to the actual architecture. Confirm not only that the attack failed, but why: the data was absent, the tool was unavailable, authorisation denied the call, egress blocked the destination, or an approval stopped commitment.

Also test the legitimate task. A control that blocks all sends may be secure but useless; a classifier that silently removes contract clauses may create a different business failure. Security and task completion need separate measures.

Use one release rule for the combination

Before release, every step that contains all three capabilities should have an approved reason and compensating controls. The owner should be able to demonstrate:

• a data-and-action map at step level;
• minimum credentials and tools for each stage;
• deterministic authorisation on consequential actions;
• constrained outbound communications;
• adversarial tests against real state changes;
• logs that reconstruct data access, policy decisions and effects; and
• a tested way to revoke credentials and halt work.

The NIST AI RMF calls for mapping risks across system components, including third parties, and for mechanisms to disengage or deactivate systems that behave outside intended use. Applied here, the decision is concrete: either break the dangerous combination, prove the boundary controls around it, or keep a person in the path. Convenience is not evidence that one context should hold the data, trust the content and command the channel.