How to build an evidence register for an AI system

AI teams rarely lack documents. They lack a reliable answer to a harder question: what does this artefact prove, about which system, under which conditions, and who accepted the remaining uncertainty?

An evidence register supplies that answer. It is not a folder of test reports or a dashboard of scores. It is a controlled index connecting requirements and claims to versioned evidence, findings, decisions and review triggers. Built well, it lets a product owner judge release, an assessor trace a claim, an incident responder reconstruct context and an engineer see what must be rerun after a change.

Start with decisions and claims

Do not begin by uploading every available artefact. Begin with the decisions the organisation must support: release an agent, approve a new action, accept a supplier component, expand a user population, continue operating after a change or retire a control.

For each decision, list the material claims that need evidence. Examples include:

• the agent acts only within an approved authority envelope;
• retrieval does not expose resources outside the active user’s permissions;
• required human confirmation occurs before an irreversible action;
• representative tasks meet defined quality thresholds;
• a stop control prevents new execution within the required interval;
• retained personal data matches the stated purpose and schedule.

Write claims narrowly enough to be challenged. “The agent is secure” is not register-ready. “Unauthorised tool requests are denied by the gateway before execution and recorded with policy version” points to an observable control and test.

The NIST AI RMF links mapping, measurement and management to documented context and risk. An evidence register makes those links navigable without pretending every risk can be reduced to one metric.

Define the entry before collecting evidence

A practical register entry needs enough structure to prevent an artefact from floating free of its meaning:

Field	Purpose
Evidence ID and title	Stable reference for decisions and findings
Claim or requirement	What the artefact is expected to support
System and configuration	Exact object tested or reviewed
Context and scope	Users, workflow, data, environment and exclusions
Method	How the evidence was produced
Artefact location and integrity	Where the protected record lives and how its version is verified
Result and uncertainty	Observed outcome, sample limits and unresolved ambiguity
Findings and treatment	Defects, risks, owners and current disposition
Reviewer and decision	Who judged sufficiency and for which decision
Validity window or trigger	What change makes the evidence stale
Access and retention	Who may view it and how long it is kept

The register should contain metadata and concise conclusions, not sensitive raw traces by default. Link to an approved evidence store with access control, retention and integrity protection.

Separate evidence types

Different artefacts answer different questions. Label the evidence type so reviewers do not confuse design intention with observed effectiveness.

• Normative evidence: policies, requirements, risk criteria and accepted standards define what should happen.
• Design evidence: architecture, data-flow maps, threat models and control specifications describe intended implementation.
• Implementation evidence: code, configuration, deployment manifests and access policies show what was built.
• Evaluation evidence: test cases, datasets, traces, rubrics and results show behaviour under selected conditions.
• Operational evidence: monitoring, sampled cases, incidents, complaints, overrides and recovery exercises show what happened in use.
• Decision evidence: approvals, restrictions, exceptions and residual-risk acceptance show accountable disposition.

No category replaces the others. A policy cannot demonstrate runtime denial. A passing test cannot show production monitoring. An absence of reported incidents cannot show a control is effective.

The UK government’s Introduction to AI assurance describes assurance as measuring, evaluating and communicating and presents a range of techniques that work in combination across the lifecycle. The register should preserve that plurality instead of forcing every artefact into a single “pass” field.

Preserve the evidence boundary

Every result has a boundary. Record the system version, model endpoint, prompts, policies, tools, retrieval corpus, identities, evaluation harness and significant dependency versions. State the population and cases represented, as well as exclusions.

For quantitative evidence, store numerator, denominator and sampling method—not just a percentage. For qualitative review, store the rubric, reviewer selection, disagreement treatment and sample. For adversarial work, distinguish found failures from an estimated failure rate unless the study supports both.

This protects against evidence laundering: reusing a result from one configuration or workflow to support a broader claim. A result obtained with external actions disabled cannot justify release with those actions enabled. A privacy review of a fixed dataset does not automatically cover newly connected memory.

Connect findings to decisions

Evidence that reveals a problem is still valuable. The register should preserve failed results and link them to findings, treatment and retest rather than replacing the old artefact with a cleaner report.

For each material finding, record:

1. observed behaviour and affected requirement;
2. credible consequence and reproducibility;
3. scope across versions or workflows;
4. immediate containment, if any;
5. corrective action and owner;
6. retest criteria and result;
7. release or operating effect;
8. residual uncertainty accepted by the decision owner.

This creates a chain from observation to management action. It also prevents “resolved” from meaning a ticket was closed without evidence that the behaviour changed.

The NIST AI RMF Playbook offers suggested documentation and risk-management practices. Use it to identify useful activities, but tailor the register to local decisions and avoid presenting framework use as certification.

Treat provenance and access as control requirements

Evidence can contain personal data, confidential prompts, security weaknesses and supplier information. Define classification, access, storage, transfer and retention before centralising it.

Keep provenance intact: who produced the artefact, when, using which method and inputs, and whether it has changed. Use hashes, signed build records, immutable storage or repository history where consequence warrants them. Do not rely on a file name such as final-report-v7.pdf as version control.

Separate public or shareable conclusions from restricted evidence. A supplier or regulator may need a specific slice, while raw traces remain limited to authorised reviewers. Redact deliberately and record the relationship between the redacted edition and the authoritative artefact.

Build expiry into every consequential entry

Evidence should not be “current” forever. Use an expiry date where time alone reduces relevance and event triggers where change is decisive.

Common triggers include:

• model, prompt, policy, retrieval or tool changes;
• new users, data, geography or intended purpose;
• altered permission scope or human review;
• incidents, complaints or credible new attack methods;
• performance or control metrics crossing thresholds;
• a supplier change that invalidates assumptions;
• loss of reproducibility or access to the underlying artefact.

The register can calculate affected claims when a component changes. A model update may invalidate behavioural evaluations while leaving a network-egress test valid. A new retrieval source may reopen permission and privacy claims without invalidating unrelated transaction controls.

Make review queries easy

The register should answer real questions quickly:

• Which release claims lack current evidence?
• Which evidence applies to the exact production configuration?
• Which severe findings remain open or accepted?
• Which controls have design evidence but no effectiveness test?
• What expires if a model, tool or data source changes?
• Which decisions depend on one supplier artefact?
• What evidence would be needed to expand the agent’s authority?

Design views around these questions, not around the org chart. A decision pack can then pull current entries without copying evidence into another unmanaged document.

For certain high-risk AI systems within its scope, the EU AI Act includes technical-documentation requirements, described in Article 11 and Annex IV of Regulation (EU) 2024/1689. Applicability and sufficiency require legal assessment. An internal evidence register can support traceability, but should not be represented as satisfying those requirements by itself.

Govern the register without turning it into a filing cabinet

Assign a register steward to manage schema, quality rules and access. Assign an evidence owner to every entry. Automate ingestion from evaluation and deployment systems where it improves accuracy, but require human judgement for scope, finding severity and decision sufficiency.

Quality checks should flag missing owners, broken artefact links, unversioned configurations, expired evidence, claims supported only by supplier assertions and decisions that refer to unresolved findings. Sample entries against the underlying artefacts to detect optimistic summaries.

ISO/IEC 42001 frames responsible AI through a maintained and continually improved management system. The register can be one supporting mechanism; it is not evidence of conformity or an assurance opinion.

Begin with one consequential decision and perhaps a dozen material claims. Link the evidence you already have, expose what is missing, and define change triggers. That smaller, rigorous register is more useful than a complete-looking repository whose artefacts cannot answer a decision.