Golden traces are useful—but they are not gold standards

A golden trace is a reviewed record of an agent completing a task acceptably. It can show the context provided, decisions made, tools called, states changed, controls applied and outcome reached. That makes it invaluable for debugging.

The trouble begins when the reference journey becomes the definition of correctness. An agent can follow the same visible path and still create the wrong state. It can also take a different, shorter path and satisfy every requirement. Exact imitation rewards familiarity, not necessarily reliability.

Use golden traces as worked examples and regression anchors. Define the standard separately: required end states, forbidden states, policy invariants, resource limits and handoff quality.

Know what the trace actually proves

A reviewed trace can establish that one configuration completed one case acceptably under recorded conditions. It can clarify ambiguous requirements and give engineers a concrete sequence to inspect. It cannot establish that the route is optimal, unique or generally safe.

Treat the trace as several evidence layers:

Layer	What it can show	What it cannot show alone
Inputs and context	What the agent was allowed to observe	Whether other plausible inputs are covered
Decisions and tool calls	How one plan unfolded	Whether the same calls are always required
Control events	That a check occurred here	That every alternate path enforces it
Final state	That this run reached the target	How consistently the agent reaches it
Handoff	What a person received	Whether other users can act on it

That evidence boundary should travel with the trace. A label such as “approved path” easily becomes stronger than the review supports.

Separate path invariants from implementation preferences

Some trace elements are non-negotiable. A protected action may require a valid, current approval. A refund may not exceed the user’s authority. Sensitive records may never enter an unapproved tool. These are invariants: they should hold no matter which plan the agent chooses.

Other details are preferences or artefacts. The agent might search before reading the account, combine two safe queries, ask a clarifying question earlier or select a different equivalent tool. Requiring a fixed sequence can reject useful adaptation and make the test brittle whenever an interface changes.

Annotate a golden trace accordingly:

• must occur: authorisation check bound to the proposed action;
• must not occur: call to a tool outside the user’s scope;
• must be true at completion: ledger and customer record agree;
• bounded choice: one or two retries, then handoff;
• illustrative step: the particular order of safe read operations.

This annotation is more valuable than a raw transcript because it communicates the reasoning behind acceptance.

Grade state before resemblance

For tool-using agents, inspect the external state. Tau-bench compares the database state after an interaction with the annotated goal state. WebArena similarly evaluates functional correctness in realistic web environments. Both point towards a practical hierarchy:

1. Was the intended state reached?
2. Were prohibited states avoided?
3. Did the path satisfy authority and policy invariants?
4. Was the execution efficient and recoverable?
5. How did it differ from reviewed references?

Trace similarity belongs near the bottom unless the path itself is the requirement. A prose explanation can look identical while the system made no change. Conversely, two API sequences can be semantically equivalent.

Build deterministic state checks where possible. Verify record values, created objects, sent messages, approval tokens and audit events. Use semantic graders for genuinely semantic properties, and human review for handoffs or judgement that cannot be reduced safely. Preserve the trace so an unexpected score can be investigated.

Maintain a set of acceptable strategies

One case may have several valid plans. Represent materially different strategies with a small trace set: direct success, clarification before action, safe fallback after a transient fault and human handoff at a boundary. Do not try to enumerate every harmless ordering.

Use a path envelope:

Component	Acceptance rule
Goal	Required final state is satisfied
Authority	Every action is within effective permission
Controls	Mandatory confirmations and checks occur
Resources	Steps, latency and cost remain within limits
Recovery	Partial work is reconciled or handed off
Communication	Final message matches actual state

This envelope lets the agent adapt while keeping the things that matter fixed. It also makes regression diagnosis clearer: the system failed the goal, a policy invariant or an operating bound—not merely “trace similarity.”

The HELM evaluation work argues for multiple scenarios and metrics while acknowledging incomplete coverage. That idea applies at workflow scale. No single trajectory captures correctness, robustness, efficiency and fairness; a trace library should make its coverage and gaps visible.

Protect trace provenance and quality

A trace becomes “golden” through review, not because it happened in production or came from a senior engineer. Record who judged it, against which policy version, under which configuration and for what intended use. Remove or control sensitive data. Version the expected state and annotations alongside the case.

The principles behind Datasheets for Datasets—documenting motivation, composition, collection and recommended uses—translate well to a trace library. For each trace, retain:

• origin and lawful basis for use;
• system and environment version;
• scenario dimensions represented;
• review method and date;
• required invariants;
• known ambiguities;
• retirement or refresh trigger.

A production trace may contain realistic exceptions but a disputed outcome. A synthetic trace may be clean but miss operational mess. Provenance helps a reviewer use each appropriately.

Use trace differences to discover, not just reject

When a new agent diverges from the reference, classify the difference:

• beneficial: fewer calls with the same controls and state;
• equivalent: a different safe ordering;
• concerning: a new dependency, retry pattern or ambiguity;
• violating: a broken invariant or wrong state;
• unknown: the current grader cannot determine equivalence.

Unknown differences are candidates for targeted review. They may reveal a valid strategy worth adding, an underspecified requirement or a failure the reference never exercised. This turns trace comparison into an evaluation tool rather than a conformity ritual.

Dynamic evaluation research such as Dynabench was motivated in part by static benchmarks becoming saturated. A local trace suite faces a smaller version of the same problem: repeated tuning makes familiar cases less informative. Keep a holdout set, add incident-derived cases and periodically ask people to construct cases against the current system.

Retire a golden trace when its assumptions expire

A trace becomes stale when the workflow, policy, tool contract, authorisation model, user population or expected state changes. It can also become misleading when engineers optimise directly against its sequence.

Review triggers should include:

• a tool or schema change;
• a model, prompt or orchestrator change that alters planning;
• a policy or approval change;
• a new incident or valid strategy;
• repeated grader disagreement;
• evidence that the trace no longer represents current work.

Retirement does not mean deletion. Preserve old traces with their valid period for audit and regression analysis, but remove them from current acceptance calculations.

The strongest trace library is not the one with the most recordings. It is the one that makes a clean distinction between examples and rules. Golden traces show how acceptable work can look. Gold standards define what must remain true even when the agent finds another way.