Evaluation & assurance6 MIN READ
Beyond pass or fail: how to grade agent evaluation findings
Grade findings by consequence, reach, reproducibility and control strength, then keep that technical judgement separate from the release decision.
THE SOFTLANDER BLOG
Evaluation design
Methods for connecting evaluation cases, measures and thresholds to a decision.
In Evaluation design
20 articlesEvaluation & assurance6 MIN READ
When does an agent evaluation dataset become stale?
Age is only one signal. A dataset becomes stale when its cases, labels or conditions no longer support the decision the evaluation is meant to inform.
Architecture & engineering6 MIN READ
How to evaluate model-routing decisions
A model router is a policy for allocating quality, cost, latency, privacy and failure risk. Evaluate whether each route serves the workflow—not whether routing is fashionable.
Evaluation & assurance6 MIN READ
How to design an independent agent benchmark that remains useful
A durable agent benchmark needs a declared construct, protected cases, reproducible execution, uncertainty, governance and limits on the claims its scores can support.
Evaluation & assurance6 MIN READ
How should we evaluate an agent that works for days rather than minutes?
Long-running agents accumulate state, dependencies and chances to drift. Evaluation must test checkpoints, recovery, changing conditions and complete outcomes across time.
Evaluation & assurance6 MIN READ
Design agent evaluations backwards from the decision
Start with the choice an owner must make, then derive claims, evidence, cases and measures. The result is smaller, sharper and easier to defend.
Evaluation & assurance6 MIN READ
Testing logistics agents under disruption
A logistics agent earns trust when data becomes late, routes close and plans conflict—not when everything follows the timetable.
Evaluation & assurance6 MIN READ
How to test whether an AI agent recovers gracefully
Inject realistic faults, verify state and judge the recovery path. A retry that finishes the task can still duplicate work or hide uncertainty.
KEEP EXPLORING
Take the thinking
into your next discussion.
Read the method, assumptions and questions to ask.