When does an agent evaluation dataset become stale?

An evaluation dataset does not become stale on its first birthday. It becomes stale when its cases, labels or test conditions stop representing the claim behind the decision.

A five-year-old policy edge may remain essential. A case collected last week may already be irrelevant after a tool contract changes. Calendar review is useful, but freshness is a relationship between evidence and intended use—not a property of the file’s timestamp.

For agent systems, that relationship changes quickly because the evaluated behaviour emerges from models, prompts, tools, permissions, retrieval, workflow state and people. Dataset maintenance must watch all of them.

Define the decision the dataset is valid for

Attach every evaluation dataset to a decision scope: which agent, workflow, users, environment, authority and risks it is intended to assess. Then record the claims and scenario dimensions it covers.

For example, a dataset may support the decision to let a service agent draft account changes for trained staff, while a person approves every write. It does not automatically support autonomous writes, public users or another jurisdiction. If the decision expands, the old data may still be useful but insufficient.

The NIST AI RMF calls for regular assessment of whether metrics and controls remain appropriate and for tracking emerging risk in deployed contexts. This is the right maintenance question: does the dataset still measure the risk and performance properties that matter now?

Watch six forms of staleness

Staleness has several causes, and each needs a different response.

Shift	Signal	Typical response
Workflow shift	New steps, channels, exceptions or ownership	Sample new paths; retire obsolete cases
Population shift	Different users, languages, regions or access needs	Rebalance and add affected groups
Policy shift	Changed eligibility, approval or retention rules	Relabel and add boundary cases
System shift	New model, prompt, tool, memory or retrieval behaviour	Run change analysis; add regressions
Threat shift	New abuse method or exposed integration	Add adversarial and containment cases
Label shift	Reviewers reinterpret what acceptable means	Adjudicate and version expected results

The generalization literature distinguishes several forms and sources of distribution shift. A Nature Machine Intelligence review argues that the source and locus of shift determine what conclusions an experiment supports. A local agent team does not need the paper’s full taxonomy, but it should stop using “drift” as one undifferentiated alarm.

Use triggers, not only a refresh interval

Set a periodic review—quarterly for some systems, more or less often for others—but define event triggers that open the dataset sooner:

• any change to model, system prompt, tool schema or orchestration;
• a policy or authorisation change;
• a new user group or operating region;
• a production incident, complaint or repeated manual override;
• a new attack or failure mode relevant to the architecture;
• a material change in workflow volume or case mix;
• benchmark saturation, unexpectedly uniform scores or grader disagreement;
• evidence that test items were used directly during tuning.

The NCSC secure development guidance recommends documenting data, models and prompts with intended scope, limitations, retention and review frequency. Add dataset triggers to that record so “review regularly” becomes an operable commitment.

Diagnose freshness with coverage and behaviour

Do not compare only input embeddings or topic counts. An evaluation dataset represents situations, expected behaviour and consequences. Review all three.

Begin with a coverage ledger. Compare recent production cases with the dimensions in the test set: request type, user role, data condition, authority, tool path, language, exception and failure consequence. Look for new combinations, not just new words.

Then compare behaviour. Has the rate or pattern of human intervention changed? Are new tools or fallback paths appearing? Do production incidents cluster outside tested states? Does a previously discriminating case now pass every system version? Does the same expected label still make sense to domain reviewers?

Sample drift needs interpretation. If benign requests shift towards a new product line, add representative cases. If a rare high-consequence authority boundary is absent from recent traffic, keep its constructed tests. Frequency is not the only measure of importance.

Prevent contamination and overfitting

An evaluation set loses diagnostic value when engineers repeatedly tune against every item. The system may improve on the visible cases without improving on the underlying behaviour.

Maintain three layers:

1. development set: visible cases for rapid iteration;
2. release holdout: controlled cases used for decision evidence;
3. challenge set: periodically refreshed cases from incidents, domain experts and adversarial work.

Restrict access proportionately; secrecy is not a substitute for sound design, but it can reduce direct overfitting. Track whether cases or close variants entered prompt examples, fine-tuning data or retrieval sources.

LiveBench responds to benchmark contamination by adding frequently updated questions from recent sources and using objective grading. Dynabench uses dynamic human-and-model interaction to collect new challenging data. An enterprise evaluation need not mimic either benchmark, but should borrow the principle: a static, fully exposed suite will eventually describe yesterday’s weaknesses.

Refresh without destroying the baseline

Replacing the entire dataset makes trends hard to interpret. Use a rolling design:

• retain a stable core of high-value invariant cases;
• add new cases for changed workflow and discovered failure modes;
• reweight cases when production prevalence changes;
• keep rare critical cases as a separate safety or authority slice;
• retire obsolete cases from current scoring without deleting history;
• run old and new versions in parallel for one cycle to understand the break.

Report results by slice and dataset version. A headline score across a changing mix is not a valid trend unless weights and composition are controlled. Preserve both “performance on the stable core” and “performance on the current decision set.”

Treat labels as versioned evidence

Expected outcomes can become stale even when inputs remain relevant. Policies change. Domain reviewers discover ambiguity. An incident shows that an apparently successful end state has an unacceptable side effect.

For each case, record who supplied or approved the expected behaviour, which policy or requirement supports it, the adjudication process and the valid period. The documentation discipline proposed in Datasheets for Datasets—motivation, composition, collection and recommended uses—is especially useful here.

Measure inter-reviewer disagreement on samples. A rise can mean poor instructions, genuine ambiguity or changing practice. Do not hide it with a majority label. Clarify the workflow or mark the case unsuitable for deterministic release gating.

Make retirement a controlled decision

A case can leave the active set because its workflow no longer exists, its expected result is invalid, it duplicates better coverage, it contains data that should no longer be retained, or it is so exposed that it no longer discriminates. Record the reason and replacement.

The maintenance record should answer:

• which decisions the current version supports;
• what changed since the prior version;
• which cases were added, reweighted or retired and why;
• where coverage remains weak;
• whether results remain comparable;
• what will trigger the next review.

Freshness does not mean novelty for its own sake. A current dataset combines durable controls with changing operational reality. Its quality is visible when the team can explain why every retained case still matters—and which release claim would become harder to defend if it disappeared.