Continuous production evaluation
Refresh evaluation evidence as live workflows and dependencies change
Pre-release tests no longer reflect current requests or tool behaviour. The decision is how to refresh evaluation evidence as live workflows and dependencies change. The work begins with your actual workflow and looks for the evidence that separates intended behaviour from a consequential exception. It is particularly relevant to production owners and evaluation platform teams. A useful outcome is an explicit account of what was examined, what remains uncertain, and which decision the evidence can support.
The release evidence relationship
- Sanitised production samples
Establish the basis for change signal.
- Sampling boundary
Use minimised records with declared coverage limits.
- Continuous evaluation design
Retain the scoped finding and its unresolved dependencies.
Decision
Frame the decision around change signal
Questions and the evidence they need
- Change signal
- Inspect sanitised production samples for the relevant constraint or observation.
- Sampling boundary
- Inspect change events for the relevant constraint or observation.
- Triage loop
- Inspect drift signals for the relevant constraint or observation.
The decision is how to refresh evaluation evidence as live workflows and dependencies change. This is relevant to production owners and evaluation platform teams when pre-release tests no longer reflect current requests or tool behaviour. Name the affected workflow, the consequence of a wrong decision, and the owner who can act on the result. The initial discussion should establish which evidence is already available and which uncertainty is material enough to investigate.
The specific failure hypothesis is that a stable dashboard hides a new task segment for which the test set has no coverage. It remains a hypothesis until examined in the deployment. Existing internal tests or operating records may already resolve it; identify the uncovered question before commissioning additional work.
Mechanism
Production changes should refresh evaluation coverage
Relationships within release evidence
- Change signal
- Detect new workflows and dependency behaviour.
- Sampling boundary
- Use minimised records with declared coverage limits.
- Triage loop
- Connect observed failure to a new case and owner.
What this specialty examines
Production evaluation needs a sampling design and a triage destination. Define which requests may be retained, how cases are sanitised, and which changes trigger targeted replay. Avoid collecting entire conversations by default when selected fields can support the evaluation question.
The review should define sampling, replay, alerting and triage loops tied to material changes and observed failures. Identify the normal route, relevant exceptions and the authoritative record of the outcome. Inspect the relevant business-system record, enforcement event or process evidence alongside any reported status. Keep incomplete cases visible, with an explanation of what their absence prevents the owner from concluding.
The tradeoff to make explicit
For this decision, broad production coverage improves relevance but must respect data minimisation and review capacity. Compare the available design or operating choices against the same intended outcome. If one option removes a required control or changes the task boundary, describe that difference before treating its apparent simplicity, speed or lower effort as an improvement. The decision should expose the tradeoff instead of burying it in an aggregate score.
Preparation
Connect the working inputs to the boundary being examined
Evidence packet for change signal
Illustrative working format
- Sanitised production samples
- Identify provenance and the specific decision this material can inform.
- Change events
- Retain the observed conditions and distinguish gaps from contrary evidence.
- Drift signals
- Name the custodian and the conditions requiring a refreshed version.
- Incident classifications
- Identify provenance and the specific decision this material can inform.
Start with sanitised production samples, change events, drift signals and incident classifications. Identify the owner, revision and operating context of each source. Sanitised examples may support initial design, but differences from actual conditions should remain explicit. Where direct access is unsuitable, a customer-controlled evidence or execution route may be considered, with agreed handling and retention.
The scope also needs the release owner, agreed expectations for a representative task case, and the authority to inspect the relevant boundary. Specify what evidence would distinguish the failure in which a stable dashboard hides a new task segment for which the test set has no coverage from acceptable behaviour. If stakeholders disagree about intended behaviour, resolve the interpretation before classifying the observed difference as a defect.
How the work proceeds
Carry change signal through to an accountable finding
From the scoped material to a decision record
- Sanitised production samples
Separate hard constraints from graded preferences.
- Sampling boundary
Apply the scoped check to triage loop.
- Continuous evaluation design
Record the evidence, open dependencies, and the owner’s next decision.
Prepare the evaluation cases
Work with production owners and evaluation platform teams to assemble sanitised production samples, change events, drift signals and incident classifications. Select a normal task, a consequential exception and a case with insufficient information. Identify the relevant requirement and expected result for each, then mark which outcomes need human adjudication. The preparation output is a case packet that separates blocking behaviour from graded quality preferences before any comparison is run.
Run and review the comparisons
Separate traffic changes from behaviour changes. A new customer segment may expose missing coverage rather than a regression on previously tested work. Maintain feedback from incidents and reviewer corrections so the case set evolves without losing its historical comparison points.
Choose the release or evaluation change
Use the reviewed cases to produce a continuous evaluation design with sampling limits, review ownership and refresh triggers. Distinguish a product defect from a missing test, an ambiguous requirement or an unreliable measuring method. For each, name the proposed correction and the case that would check it. The release owner can then decide whether to change the implementation, improve coverage or restrict the intended use while a specific gap remains.
Indicative artefact
Make the continuous evaluation design usable by its owner
Continuous evaluation design
Illustrative working format
- Continuous evaluation design
- Identify the assessed workflow, revision, and decision boundary.
- Sampling limits
- Link supporting and contrary observations to inspectable evidence.
- Review ownership
- Name the role, decision right, handoff, and unresolved ownership boundary.
A useful decision artefact is a continuous evaluation design with sampling limits, review ownership and refresh triggers. Supporting material may include selected cases, configuration or process records, evidence references, a disagreement log and prioritised actions. Retain the case selection rationale and material disagreements alongside the result; otherwise the result may be mistaken for complete coverage.
Illustrative scenario
Examine triage loop through a constructed exception
A constructed case and its decision paths
Illustrative scenario
- Change signal
A new subscription type appears outside the original fixture set.
- Sampling boundary
Review whether the new subscription changes task or authority expectations.
- Triage loop
Refresh the affected fixtures before generalizing prior results.
Consider a fictional situation: a support agent begins receiving a new subscription type that never appeared in its original evaluation fixtures. The scenario is illustrative, with no client engagement, executed test or measured improvement implied.
For the new subscription type, label the coverage gap before assigning a score. Gather representative permissible cases and agree the intended behaviour with the product owner. The resulting update can expand the test set while keeping old results interpretable.
Scope and next action
Keep the conclusion within selected task cases
Confirm fit before the next commitment
- Change signal
- This review is limited to selected task cases.
- Sanitised production samples
- The relevant owner must confirm access and the intended operating conditions.
- Continuous evaluation design
- The customer retains the release decision.
A result applies to the recorded configuration and selected cases. It does not establish universal agent competence or authorise release. Any engagement depends on an agreed scope, qualified team, authorised access and confirmed availability. Bring the affected workflow, its current configuration or process, the triggering change and the evidence already available. The immediate scope question is prompted by the situation in which pre-release tests no longer reflect current requests or tool behaviour; confirm which missing observation would change the owner’s next action.
Ways to engage
Related engagement scopes
AI Agent Evaluation Evidence Diagnostic
Find the gaps between your release decision and the evidence behind it. Start with one agent workflow and a bounded set of questions.
AI Agent Evaluation Foundation
Turn scattered tests into a repeatable release discipline. Connect the behaviours that matter to datasets, evaluation criteria and evidence your team can maintain.
AI Long-Horizon Agent Evaluation
Examine what changes when an agent keeps working. Evaluate accumulated state, recovery and goal adherence across extended tasks.
Continue exploring
Related insights and resources
Engineer the harness, not just the prompt
Define the contracts around an agent's decisions: what state it trusts, which actions can execute, how interrupted work resumes, and what evidence an operator needs to recover safely.
What would justify releasing this agent?
Replace a single pass rate with a reviewable chain of requirements, cases, observations and judgements. Distinguish task quality, control failures and uncertainty before increasing an agent's exposure.
The evidence behind the go-live decision
A working paper for leaders who need to explain why an agent workflow is ready for its next release. Includes a proposed evidence-pack structure, decision table and review agenda.