Evaluation & assurance6 MIN READ
Golden traces are useful—but they are not gold standards
A reference trace can explain one acceptable journey. Treating it as the only correct path can punish valid behaviour and conceal unsafe alternatives.
THE SOFTLANDER BLOG
Release evidence
Practical methods for turning agent evaluation results into reviewable release, change and operating decisions.
In Release evidence
10 articlesEvaluation & assurance6 MIN READ
Measure the action, not just the answer
A convincing message can conceal a failed or unauthorised operation. Agent evaluation must inspect tool calls, external state and control decisions.
Evaluation & assurance6 MIN READ
Continuous evaluation without alert theatre
Run evaluations often, but interrupt people only when a signal maps to user consequence, a decision owner and a tested response.
Evaluation & assurance6 MIN READ
Turning red-team findings into release criteria
A dramatic exploit is a discovery, not a decision rule. Translate it into a threat condition, required control, repeatable regression and release consequence.
Evaluation & assurance6 MIN READ
What confidence is enough for an agent release decision?
Confidence is not a universal percentage. It is a judgement about evidence quality, uncertainty, consequence and the reversibility of the proposed release.
Evaluation & assurance6 MIN READ
Beyond pass or fail: how to grade agent evaluation findings
Grade findings by consequence, reach, reproducibility and control strength, then keep that technical judgement separate from the release decision.
Evaluation & assurance6 MIN READ
When does an agent evaluation dataset become stale?
Age is only one signal. A dataset becomes stale when its cases, labels or conditions no longer support the decision the evaluation is meant to inform.
Evaluation & assurance6 MIN READ
What belongs in an AI agent release evidence pack?
A useful release pack connects the intended workflow, evaluated behaviour, unresolved risk and operating controls to one accountable decision.
Evaluation & assurance6 MIN READ
Design agent evaluations backwards from the decision
Start with the choice an owner must make, then derive claims, evidence, cases and measures. The result is smaller, sharper and easier to defend.
Evaluation & assurance6 MIN READ
How to test whether an AI agent recovers gracefully
Inject realistic faults, verify state and judge the recovery path. A retry that finishes the task can still duplicate work or hide uncertainty.
KEEP EXPLORING
Take the thinking
into your next discussion.
Read the method, assumptions and questions to ask.