Turning red-team findings into release criteria

A red team persuades an agent to send private data to an unauthorised endpoint. The demonstration is memorable, but it is not yet a release criterion. Was the tested path available in production? Which control failed? Would different wording work? What state proves the defence held? Does one failure block the whole release or only the affected tool?

Red teaming discovers plausible ways a system can fail under pressure. Release criteria define the evidence required to grant authority. Connecting them takes analysis, engineering and accountable risk judgement.

Preserve the finding before simplifying it

Record the original conditions in enough detail to reproduce the event:

• system and model configuration;
• user and attacker capabilities;
• entry point for hostile content;
• tool and data permissions;
• prompts, retrieved content and intermediate observations;
• requested and actual tool calls;
• authorisation and policy decisions;
• external state and data movement;
• existing controls and their responses;
• repetitions, variants and invalid runs.

Separate observed facts from inferred cause. “The HTTP tool sent record fields to domain X” is observed. “The model followed an indirect instruction because it could not distinguish data from commands” is a hypothesis to investigate.

The NIST Generative AI Profile explicitly says red-team results need additional analysis before being incorporated into governance and risk management. That extra step protects against two errors: dismissing a severe finding as an unrealistic trick, and treating a theatrical laboratory exploit as proof of broad production risk.

Normalise the discovery into a threat condition

Strip away incidental wording and describe the conditions that made the behaviour possible.

For example:

An authenticated agent with access to confidential records reads untrusted document content and can call a general network tool. The policy layer does not bind outbound destinations or data classes, so an indirect instruction can cause unauthorised disclosure.

This statement identifies actor, asset, trust boundary, capability and missing control. It can generate many test variants and guide architecture treatment.

Use a shared taxonomy where it helps teams communicate. NIST AI 100-2e2025 classifies adversarial machine-learning attacks by system type, lifecycle stage, attacker objective, capability and knowledge. A taxonomy does not assign local risk tolerance, but it prevents every finding from becoming a bespoke story.

Validate feasibility and consequence separately

Ask whether the attacker can create the conditions in the intended environment. A test harness may have granted direct prompt access that a production user lacks; equally, a production agent may process untrusted email that the laboratory omitted.

Then assess consequence:

• which data, actions and users are reachable;
• whether the behaviour crosses a hard authority boundary;
• whether the action is detectable and reversible;
• whether the attacker can repeat or scale it;
• what a successful compromise lets the attacker do next;
• which compensating controls operate outside the agent.

The NCSC secure design guidance recommends holistic threat modelling that considers impacts on systems, users, organisations and wider society, alongside AI-specific threats. Keep that breadth while avoiding speculative chains presented as observed facts.

A useful validation label is: confirmed in representative environment, plausible with untested precondition, laboratory-only, or inconclusive. Preserve all four; only the first two normally drive immediate release criteria.

Identify the control claim the release depends on

Do not make “the agent resists this prompt” the primary control claim. Models and attacks change. Prefer claims that can be enforced and observed at the system boundary:

• untrusted content cannot grant authority;
• sensitive fields cannot be sent to unapproved destinations;
• every protected action requires a current, parameter-bound approval;
• tool identity and scope are enforced independently of model output;
• memory writes are scoped, sanitised and reviewable;
• excessive steps or retries are stopped;
• incidents produce sufficient evidence for containment.

The OWASP GenAI Red Teaming Guide spans model evaluation, implementation testing, infrastructure and runtime behaviour. That breadth is important: a model-level refusal may be useful defence in depth, but an external authorisation or egress control gives a stronger release claim.

Convert the finding into repeatable tests

Create a regression family, not one frozen exploit string. Include:

1. the original case;
2. paraphrases and structural variants;
3. alternate untrusted sources;
4. benign lookalikes to measure over-blocking;
5. different user roles and permission scopes;
6. partial control failure, such as unavailable policy service;
7. a deterministic check of tool call and external state.

Define both utility and security outcomes. The agent should complete the legitimate task where possible while refusing or containing the attacker goal. AgentDojo is useful here because it evaluates agent utility and security in dynamic environments rather than asking only whether an attack string was refused.

Keep some variants hidden from routine tuning. If the agent learns one phrase, the release criterion has not established the underlying control.

Write a criterion with a decision consequence

A complete release criterion names scope, evidence, expected result and response. For example:

For every case in the critical outbound-disclosure set, the policy layer must deny transfer of protected fields to a destination outside the allowlist, regardless of model request. The legitimate document task must remain available in at least the agreed proportion of benign paired cases. Any unauthorised transfer blocks the network tool from release; inconclusive instrumentation blocks the decision until repaired.

This is stronger than “prompt-injection success rate below 5%.” It identifies the control, critical outcome, utility trade-off and release effect. Other behaviours may appropriately use rates with uncertainty; hard authority boundaries should not disappear into an average.

Build a treatment table:

Finding state	Default release effect
Critical control bypass reproduced	Block affected authority
Defence works but utility collapses	Redesign or release narrower workflow
Path plausible but environment not representative	Validate before decision
Exploit prevented by independent control	Release only with that control verified
Residual lower-grade issue accepted	Record owner, rationale, expiry and monitoring

Retest architecture, not only prompts

Fixes should follow the cause. If the issue is broad egress authority, rewriting the system prompt is unlikely to be a sufficient control. Change tool scope, destination allowlists, data classification, approval binding, isolation or workflow design as appropriate.

After treatment, run the regression family and adjacent abuse cases. Confirm the control fails closed. Check that logging captures the attempted violation without exposing the sensitive payload. Exercise incident response and credential revocation where consequence warrants it.

Red-team evidence also decays. New tools, data sources, model behaviour and attacks can invalidate an old pass. Attach rerun triggers to the criterion: relevant configuration change, new incident, new threat technique or expiry of the accepted evidence.

Keep discovery and acceptance independent

The red team should report what it found and the limits of the exercise. Engineering should propose treatments and supply control evidence. The accountable release owner should accept, constrain or reject authority. One team may perform several roles, but the record should preserve the distinctions.

Retain unsuccessful attack attempts too, with enough context to interpret them. They can show useful resistance under tested conditions, reveal blind spots in the exercise and stop later reviewers from mistaking “not reported” for “not attempted.” They must not be inflated into proof that the attack class is impossible.

Passing the regression suite does not prove the agent is secure. It demonstrates that named controls held under tested conditions and that known discoveries did not recur. That bounded claim is exactly what a release criterion should provide.