What a technical AI privacy review can—and cannot—conclude

A technical privacy review can find that an agent sends full customer records to a model when three fields would do. It can show that deleted conversations remain searchable, that one tenant can retrieve another’s documents or that debug logs retain prompts indefinitely.

It cannot, by itself, determine the correct lawful basis, decide whether a purpose is socially acceptable, interpret every jurisdiction or certify that an organisation complies with data-protection law. Those conclusions require accountable legal, policy and business judgement informed by the technical evidence.

The distinction is not a weakness. A well-scoped technical review gives specialists a truthful picture of what the system actually does and gives engineers requirements they can implement and test.

Define the decision and system boundary

Begin with the decision the review must support: approve a pilot with restricted data, release a memory feature, connect a new retrieval source, select a provider, respond to a finding or expand the agent to another user group.

Freeze the reviewed configuration: model and provider, prompts, retrieval, memory, tools, identities, logging, regions and significant dependencies. Name intended purpose, users, affected people and environments. Without that boundary, a privacy conclusion can be reused for a system the review never observed.

State roles and assumptions as inputs to be validated by qualified owners. Which organisation determines purposes and means? Which suppliers process data? Which data is believed to be personal, sensitive, confidential or anonymous? The technical team can map behaviour and test transformations, but should flag rather than silently resolve uncertain legal classifications.

Trace data end to end

Build a data-flow map from collection to deletion. Follow both content and metadata through:

• user input, attachments and source systems;
• preprocessing, classification and redaction;
• embeddings, indexes and retrieval caches;
• prompt and context assembly;
• model and tool providers;
• agent memory and authoritative business records;
• telemetry, debugging and analytics;
• evaluation, feedback and improvement datasets;
• replicas, backups and archives;
• subprocessors and cross-border transfers.

For every transition, record purpose, fields, identity, recipient, region, protection, retention and deletion route. Verify the map against code, configuration, traffic and provider settings; workshops alone tend to miss retries, error paths and observability exports.

The NIST Privacy Framework treats privacy risk as arising from data processing, not only security incidents, and provides a common language across a data-processing ecosystem. That is a useful review frame because an agent’s privacy effects can occur even when no unauthorised attacker is present.

Convert privacy goals into testable requirements

The review needs concrete criteria agreed with privacy and workflow owners. Examples include:

• the retrieval service returns only resources authorised for the active principal;
• the model provider receives no direct identifiers for a defined task;
• durable memory stores only user-confirmed preferences with provenance and expiry;
• prompt logs redact stated fields and expire within the approved period;
• a deletion request invalidates source records, chunks and embeddings within a defined interval;
• external tool calls include only allowlisted fields necessary for the action;
• protected attributes are not inferred or used for an unapproved purpose.

Requirements should include a failure response. If redaction fails, does the request stop or proceed? If policy cannot be resolved, does retrieval fail closed? If deletion is partial, is the request kept open and data restricted?

The ICO AI and data-protection risk toolkit can help teams consider lifecycle risks and practical controls. It is regulatory guidance, not a substitute for assessing the organisation’s specific processing.

Test the controls, not just their configuration

Review design and implementation evidence, then exercise the boundary. A mature test plan covers:

Area	Example evidence
Minimisation	Field ablation shows which inputs the task actually needs
Access	Cross-role and cross-tenant retrieval attempts are denied before context assembly
Disclosure	Prompt, output and tool payload inspections find no prohibited fields
Memory	Unapproved facts are not written; expiry and correction propagate
Deletion	Seeded records disappear from active stores and derivatives within stated scope
Provider handling	Runtime settings and contracts align with observed endpoints and retention
Leakage resistance	Reasonable extraction and prompt-injection cases do not expose protected content
Observability	Logs retain enough accountability without unnecessary payload content

Use synthetic records where possible and protect test artefacts. Test normal workflows, error paths, retries, caches and administrative access. A front-end check can pass while a background trace exports the same personal data.

CNIL’s AI development recommendations discuss purpose, minimisation, data-source documentation, security, rights and impact assessment. They also show why technical and legal work must remain connected: a measure such as synthetic data or unlearning has to be selected for a defined processing context and evaluated for effectiveness.

Make findings precise and repairable

Write findings as observed conditions, not broad accusations. Include:

1. the affected requirement and data flow;
2. evidence and reproduction steps;
3. people or records plausibly affected;
4. consequence and exploitability;
5. current containment;
6. recommended treatment and owner;
7. retest condition;
8. unresolved legal or policy question.

For example: “Conversation traces include unredacted email addresses in the monitoring provider for 30 days despite a seven-day requirement” is actionable. “The system violates privacy” goes beyond the technical observation and may obscure the fix.

Severity should combine likelihood, sensitivity, scale, duration, identifiability and individual impact. Do not automatically rate every personal-data finding the same. Preserve uncertainty and avoid inventing affected-record counts when telemetry cannot support them.

State what the review can conclude

A technical review can credibly conclude, within its tested boundary, that:

• mapped data flows match or differ from declared design;
• controls are present, absent or misconfigured;
• selected tests demonstrate behaviour under stated conditions;
• data is broader, longer-lived or more widely shared than the requirement permits;
• deletion, access or recovery mechanisms succeed or fail in the tested environment;
• evidence is insufficient for a proposed technical claim;
• specific changes should trigger retesting.

It can also provide a residual-risk view: what remains exposed, which controls depend on suppliers, what was not testable and how operation should be restricted while gaps remain.

State what it cannot conclude alone

The review should not claim:

• that processing has a valid lawful basis;
• that notices, consent or contractual terms are legally sufficient;
• that a model is anonymous in all circumstances;
• that a deployment complies with every applicable law;
• that technical minimisation makes the purpose fair or acceptable;
• that no personal data can ever be extracted;
• that passing sampled tests guarantees future behaviour;
• that the reviewer provides certification, legal advice or regulatory approval.

The EDPB’s Opinion 28/2024 illustrates the fact-specific nature of assessing anonymity, legitimate interest and unlawfully processed data in AI models. Technical extraction tests can inform an anonymity assessment, but they do not replace the competent authority’s or counsel’s analysis.

Join technical evidence to accountable decisions

Close the review with a traceable decision pack: reviewed version, requirements, data-flow map, tests, findings, treatment owners, residual uncertainty, operating restrictions and change triggers. Route legal questions to privacy counsel, ethical trade-offs to accountable leadership and remediation to technical owners.

The NIST AI RMF emphasises contextual, lifecycle risk management and documented roles. Use the technical review as one evidence-producing part of that broader governance process, not as a badge.

Revisit the review when purpose, data, provider, model, retrieval, memory, logging, users or geography changes. A static report will decay; a maintained requirement-and-evidence map can tell the organisation exactly what the change reopens.

The best outcome is not a declaration that privacy is complete. It is a system whose data behaviour is visible, whose controls can be tested, whose gaps have owners and whose leaders understand which questions still require judgement beyond engineering.