Golden trace development
Define reference traces without prescribing one brittle reasoning path
Reviewers disagree about which intermediate actions make a task acceptable. The decision is how to define reference traces without prescribing one brittle reasoning path. The work begins with your actual workflow and looks for the evidence that separates intended behaviour from a consequential exception. It is particularly relevant to evaluation designers and domain experts. A useful outcome is an explicit account of what was examined, what remains uncertain, and which decision the evidence can support.
The release evidence relationship
- Expert demonstrations
Establish the basis for required milestone.
- Allowed alternative
Permit different lookups that preserve constraints.
- Reference-trace library
Retain the scoped finding and its unresolved dependencies.
Decision
Frame the decision around required milestone
Questions and the evidence they need
- Required milestone
- Inspect expert demonstrations for the relevant constraint or observation.
- Allowed alternative
- Inspect tool contracts for the relevant constraint or observation.
- Forbidden transition
- Inspect permitted alternatives for the relevant constraint or observation.
The decision is how to define reference traces without prescribing one brittle reasoning path. This is relevant to evaluation designers and domain experts when reviewers disagree about which intermediate actions make a task acceptable. Name the affected workflow, the consequence of a wrong decision, and the owner who can act on the result. The initial discussion should establish which evidence is already available and which uncertainty is material enough to investigate.
The specific failure hypothesis is that a reference transcript rewards superficial sequence matching while rejecting an equally valid route. It remains a hypothesis until examined in the deployment. Existing internal tests or operating records may already resolve it; identify the uncovered question before commissioning additional work.
Mechanism
Reference traces should allow more than one valid route
Relationships within release evidence
- Required milestone
- Name the evidence that every accepted route needs.
- Allowed alternative
- Permit different lookups that preserve constraints.
- Forbidden transition
- Flag skipped authority checks despite matching answers.
What this specialty examines
A golden trace is most useful when it distinguishes necessary evidence from incidental sequence. Mark required checks, allowed alternatives and forbidden transitions. Avoid treating a model’s private reasoning as the only valid route; observable actions and supported outcomes are usually more reviewable anchors.
The review should annotate required milestones, forbidden actions and acceptable route variation in representative traces. Identify the normal route, relevant exceptions and the authoritative record of the outcome. Inspect the relevant business-system record, enforcement event or process evidence alongside any reported status. Keep incomplete cases visible, with an explanation of what their absence prevents the owner from concluding.
The tradeoff to make explicit
For this decision, detailed trace expectations improve diagnosis while excessive sequence constraints punish valid adaptation. Compare the available design or operating choices against the same intended outcome. If one option removes a required control or changes the task boundary, describe that difference before treating its apparent simplicity, speed or lower effort as an improvement. The decision should expose the tradeoff instead of burying it in an aggregate score.
Preparation
Connect the working inputs to the boundary being examined
Evidence packet for required milestone
Illustrative working format
- Expert demonstrations
- Keep expected behaviour and the limits of the constructed example explicit.
- Tool contracts
- Check its relationship to the relevant workflow and expose missing context.
- Permitted alternatives
- Name the custodian and the conditions requiring a refreshed version.
- Terminal-state evidence
- Define completion using the authoritative state beyond the conversation.
Start with expert demonstrations, tool contracts, permitted alternatives and terminal-state evidence. Identify the owner, revision and operating context of each source. Sanitised examples may support initial design, but differences from actual conditions should remain explicit. Where direct access is unsuitable, a customer-controlled evidence or execution route may be considered, with agreed handling and retention.
The scope also needs the release owner, agreed expectations for a representative task case, and the authority to inspect the relevant boundary. Specify what evidence would distinguish the failure in which a reference transcript rewards superficial sequence matching while rejecting an equally valid route from acceptable behaviour. If stakeholders disagree about intended behaviour, resolve the interpretation before classifying the observed difference as a defect.
How the work proceeds
Carry required milestone through to an accountable finding
From the scoped material to a decision record
- Expert demonstrations
Separate hard constraints from graded preferences.
- Allowed alternative
Apply the scoped check to forbidden transition.
- Reference-trace library
Record the evidence, open dependencies, and the owner’s next decision.
Prepare the evaluation cases
Work with evaluation designers and domain experts to assemble expert demonstrations, tool contracts, permitted alternatives and terminal-state evidence. Select a normal task, a consequential exception and a case with insufficient information. Identify the relevant requirement and expected result for each, then mark which outcomes need human adjudication. The preparation output is a case packet that separates blocking behaviour from graded quality preferences before any comparison is run.
Run and review the comparisons
Have reviewers explain why each milestone matters. If two experts propose different valid sequences, record both or abstract the shared constraint. Reference traces should evolve through an explicit review process, preserving earlier versions used in past comparisons.
Choose the release or evaluation change
Use the reviewed cases to produce a reference-trace library with rationale, alternatives and reviewer disagreement records. Distinguish a product defect from a missing test, an ambiguous requirement or an unreliable measuring method. For each, name the proposed correction and the case that would check it. The release owner can then decide whether to change the implementation, improve coverage or restrict the intended use while a specific gap remains.
Indicative artefact
Make the reference-trace library usable by its owner
Reference-trace library
Illustrative working format
- Reference-trace library
- Identify the assessed workflow, revision, and decision boundary.
- Rationale
- Link supporting and contrary observations to inspectable evidence.
- Alternatives
- Name the unresolved condition, next action, and accountable owner.
A useful decision artefact is a reference-trace library with rationale, alternatives and reviewer disagreement records. Supporting material may include selected cases, configuration or process records, evidence references, a disagreement log and prioritised actions. Retain the case selection rationale and material disagreements alongside the result; otherwise the result may be mistaken for complete coverage.
Illustrative scenario
Examine forbidden transition through a constructed exception
A constructed case and its decision paths
Illustrative scenario
- Required milestone
Two different lookup sequences identify the same authorised account.
- Allowed alternative
Compare required identity checks across both lookup routes.
- Forbidden transition
Accept legitimate route variation while rejecting skipped authority checks.
Consider a fictional situation: two support agents resolve the same issue through different valid lookups, but a rigid comparator fails one. The scenario is illustrative, with no client engagement, executed test or measured improvement implied.
In the two-lookup illustration, both routes may identify the same authorised account reliably. The comparator should accept either if required evidence and permissions are preserved. A route that skips identity confirmation is a different case, even when its final answer happens to match.
Scope and next action
Keep the conclusion within selected task cases
Confirm fit before the next commitment
- Required milestone
- This review is limited to selected task cases.
- Expert demonstrations
- The relevant owner must confirm access and the intended operating conditions.
- Reference-trace library
- The customer retains the release decision.
A result applies to the recorded configuration and selected cases. It does not establish universal agent competence or authorise release. Any engagement depends on an agreed scope, qualified team, authorised access and confirmed availability. Bring the affected workflow, its current configuration or process, the triggering change and the evidence already available. The immediate scope question is prompted by the situation in which reviewers disagree about which intermediate actions make a task acceptable; confirm which missing observation would change the owner’s next action.
Ways to engage
Related engagement scopes
AI Agent Evaluation Evidence Diagnostic
Find the gaps between your release decision and the evidence behind it. Start with one agent workflow and a bounded set of questions.
AI Agent Evaluation Foundation
Turn scattered tests into a repeatable release discipline. Connect the behaviours that matter to datasets, evaluation criteria and evidence your team can maintain.
AI Long-Horizon Agent Evaluation
Examine what changes when an agent keeps working. Evaluate accumulated state, recovery and goal adherence across extended tasks.
Continue exploring
Related insights and resources
Engineer the harness, not just the prompt
Define the contracts around an agent's decisions: what state it trusts, which actions can execute, how interrupted work resumes, and what evidence an operator needs to recover safely.
What would justify releasing this agent?
Replace a single pass rate with a reviewable chain of requirements, cases, observations and judgements. Distinguish task quality, control failures and uncertainty before increasing an agent's exposure.
The evidence behind the go-live decision
A working paper for leaders who need to explain why an agent workflow is ready for its next release. Includes a proposed evidence-pack structure, decision table and review agenda.