What should trigger an agent governance review?

An annual governance review is easy to schedule and easy to outgrow. An agent can gain a new tool, broader permissions, different users and a replacement model within weeks. If the review clock does not respond, yesterday’s decision silently follows a different system into production.

Governance should be event-driven. A defined change or signal opens a proportionate review, routes it to the right owners and determines whether the agent may continue, narrow, pause or require new evidence. Periodic review still matters, but mainly as a backstop for slow drift and events the organisation failed to detect.

Trigger on a change in intended purpose

A new purpose changes the standard against which the system should be assessed. An internal research assistant used to draft customer-facing advice has not merely expanded its audience; it now creates a different consequence and communication obligation.

Trigger review when:

• a workflow moves from experiment to operational use;
• an internal tool reaches customers, employees or the public;
• a recommendation becomes a default or automated decision;
• a system is reused for a population, sector or decision it was not evaluated for;
• an optional feature becomes part of a required business process.

Ask whether the original requirements, data and test cases represent the new purpose. If not, the old evidence should not be described as covering it.

The NIST AI RMF Core calls for intended purpose, context, scope, costs and benefits to be documented and for monitoring and periodic review to be planned. Purpose is therefore not a label attached once; it is a governance input that must stay aligned with operation.

Trigger on expanded authority or reduced intervention

The most important changes are often outside the model. Adding a “send,” “approve,” “purchase,” “delete” or “delegate” capability can transform a content system into an actor.

Open a review when:

• a new tool or action class is enabled;
• credential scopes, transaction limits or accessible resources expand;
• a human approval becomes sampled or post-action review;
• timeout behaviour changes from stop to proceed;
• an agent can spawn or instruct another agent;
• a sandbox or network restriction is relaxed;
• rollback, revocation or shutdown behaviour changes.

Review the enforced boundary, not only the prompt. Trace the active identity through every downstream call and test denial, partial execution and recovery. A tool may be read-only in product language while still revealing sensitive information; an apparently reversible action may propagate beyond recovery.

Trigger on data and retrieval changes

Adding a retrieval source can alter accuracy, privacy, authorisation and intellectual-property risk at once. Changing memory retention can turn ephemeral context into a persistent record.

Review when:

• a new category of personal, confidential or licensed data is introduced;
• sources cross business units, customers, tenants or jurisdictions;
• access-control metadata or retrieval filters change;
• conversation, tool output or memory retention increases;
• data is reused for evaluation, training or improvement;
• a new provider processes prompts, context, logs or telemetry;
• deletion and rights-handling mechanisms change.

The UK’s ICO AI risk toolkit organises risks and practical controls across the AI lifecycle, including security and data minimisation. Legal applicability depends on the processing and jurisdiction; operationally, any material change to data purpose, flow or persistence deserves privacy and security attention.

Trigger on component and dependency changes

A stable application version can hide a changed model endpoint, embedding service, policy library or supplier behaviour. Define which dependencies are pinned, monitored and subject to notification.

Changes that commonly require review include:

• model family, version, provider or routing policy;
• system prompts, tool descriptions or orchestration logic;
• retrieval model, chunking, reranking or index configuration;
• safety filters, policy engines and approval rules;
• material software dependencies or hosting regions;
• supplier terms affecting data use, retention or access;
• loss of a dependency needed for logging, evaluation or rollback.

Not every update needs a committee. Create pre-approved change classes with automated regression suites and clear stop conditions. A small prompt correction might run a focused test set; a provider change that affects data location and behaviour may require technical, privacy, procurement and workflow decisions.

The UK NCSC secure AI development guidelines recommend tracking models, data and prompts, documenting dependencies, managing changes and monitoring system behaviour. Treat dependency visibility as part of the trigger mechanism.

Trigger on evidence degradation

Evidence expires when its conditions no longer represent operation. Review should open even when the system configuration did not deliberately change.

Useful signals include:

• evaluation performance crosses a defined threshold;
• production cases move outside the evaluated distribution;
• test coverage falls behind new tools or workflows;
• a benchmark or dataset becomes stale, contaminated or unrepresentative;
• monitoring shows rising overrides, refusals, recovery or escalation;
• a control has not been tested within its required interval;
• source provenance or evaluation reproducibility is lost.

Attach each approval to an evidence envelope: configuration, intended use, user population, data conditions, metrics, thresholds and expiry or trigger. When operation leaves that envelope, the system should no longer inherit the decision without review.

Trigger on incidents, complaints and near misses

An incident is evidence about a scenario the organisation may have underestimated. Do not limit triggers to confirmed harm. Near misses, repeated workarounds, unexpected tool calls and credible external vulnerability reports can expose a control gap before loss occurs.

Open review when:

• unauthorised access, disclosure or action is suspected;
• the agent behaves outside intended use or ignores a control;
• a person cannot stop or reverse an action as designed;
• a complaint reveals an affected group or consequence not mapped;
• a supplier or public source reports a relevant vulnerability;
• an operator repeatedly bypasses the workflow to complete work;
• a material output cannot be reconstructed from records.

The review should decide both immediate containment and whether requirements, evaluations, monitoring and training need to change. Add confirmed failure cases to regression evidence without assuming a single test prevents recurrence.

Define materiality before the event

A useful trigger register connects event, threshold, owner, response time and interim state:

Trigger	Materiality test	Immediate state	Review owner
New external action	Can affect another party or commit resources	Disable until approved	Workflow owner
Model change	Outside accepted version or behaviour envelope	Restricted canary	Technical owner
New sensitive data	New purpose, category, recipient or retention	Block ingestion	Privacy owner
Evaluation decline	Crosses release or operating threshold	Narrow capability	Evidence owner
Control failure	Prevention, detection or recovery unavailable	Suspend affected action	Control owner
Incident or complaint	Credible material consequence or rights impact	Contain and preserve evidence	Incident owner

Materiality rules should account for cumulative change. Ten individually minor adjustments can produce a major behavioural difference. Set a change budget or periodic baseline comparison so small changes cannot evade review indefinitely.

Make the review proportionate and decisive

A trigger does not have to summon the full governance body. Define tiers:

• Recorded change: update the inventory and automated evidence; no decision change.
• Focused review: one or two control owners assess a bounded effect.
• Cross-functional review: purpose, data, authority or consequence changed materially.
• Immediate containment: suspend or narrow operation while evidence is gathered.

Every review should close with an explicit disposition: continue, continue with conditions, restrict, remediate and retest, or stop. Record the scope, evidence, open findings, owner and next trigger.

The EU AI Act discusses “substantial modification” for systems within its scope, including changes that can affect compliance or intended purpose, and in some circumstances treats the modified system as new for conformity assessment. The exact legal application of Regulation (EU) 2024/1689 depends on role and facts. Do not turn an internal materiality rubric into a legal conclusion; involve qualified counsel for applicability.

Retain a periodic backstop

Event-driven review depends on detection. Use periodic reconciliation to catch shadow systems, undocumented changes and gradual drift. The frequency should follow consequence, rate of change and evidence half-life rather than one organisation-wide anniversary.

Review the system record, deployed configuration, permissions, providers, users, data flows, incidents, exceptions, evaluation results and retirement need. Reconcile with procurement, identity, cloud and data records. Ask whether the system still serves a valid purpose and whether a simpler, less risky method now exists.

ISO/IEC 42001 emphasises maintaining and continually improving an AI management system. A trigger framework supports that discipline, but it is not proof of conformity. Its success is visible in operating decisions: changes are detected, routed, evidenced and closed before they quietly enlarge the organisation’s exposure.

Governance should move at the speed of material change. Define the events now, connect them to delivery and operations, and make the safe interim state automatic where consequence warrants it.