Evaluation & assurance6 MIN READ
Golden traces are useful—but they are not gold standards
A reference trace can explain one acceptable journey. Treating it as the only correct path can punish valid behaviour and conceal unsafe alternatives.
THE SOFTLANDER BLOG
Agent reliability
Consistency, recovery and production behaviour in agentic systems.
In Agent reliability
10 articlesEvaluation & assurance6 MIN READ
How to evaluate a software-engineering agent on real repository work
A passing patch is only the start. Repository evaluation should test understanding, change quality, security, reviewability and recovery.
Architecture & engineering6 MIN READ
How do you evaluate an agent orchestrator?
An orchestrator should be judged by the workflow it controls: decomposition, routing, state, authority, recovery, verification and stopping—not by a polished final response.
Evaluation & assurance6 MIN READ
Continuous evaluation without alert theatre
Run evaluations often, but interrupt people only when a signal maps to user consequence, a decision owner and a tested response.
Architecture & engineering6 MIN READ
How to evaluate model-routing decisions
A model router is a policy for allocating quality, cost, latency, privacy and failure risk. Evaluate whether each route serves the workflow—not whether routing is fashionable.
Evaluation & assurance6 MIN READ
What belongs in an AI agent release evidence pack?
A useful release pack connects the intended workflow, evaluated behaviour, unresolved risk and operating controls to one accountable decision.
Architecture & engineering6 MIN READ
Idempotency for AI agents: preventing one instruction from becoming three actions
Retries, duplicated messages and uncertain tool outcomes can multiply an agent’s effects. Idempotency turns a logical action into one controlled business operation.
Agent security6 MIN READ
Can you trust an agent kill switch you have never tested?
Stopping one process may leave tools, queued actions, delegated agents and credentials active. Define the safe state, control every action path and rehearse the stop under load.
Evaluation & assurance6 MIN READ
How to test whether an AI agent recovers gracefully
Inject realistic faults, verify state and judge the recovery path. A retry that finishes the task can still duplicate work or hide uncertainty.
Strategy & transformation6 MIN READ
Adoption is part of agent reliability
An agent is not reliable if people avoid it, over-trust it or route around its controls. Adoption evidence belongs beside technical performance evidence.
KEEP EXPLORING
Take the thinking
into your next discussion.
Read the method, assumptions and questions to ask.