Architecture & engineering7 MIN READ
A failure ladder for retrieval-augmented agents
A wrong answer is the final symptom of failures that may begin with source coverage, access control, parsing, retrieval, context assembly, generation or action.
THE SOFTLANDER BLOG
Evaluation design
Methods for connecting evaluation cases, measures and thresholds to a decision.
In Evaluation design
20 articlesStrategy & transformation6 MIN READ
How to write an RFP for agentic AI without buying a promise
Specify the workflow, evidence, responsibility and exit conditions suppliers must meet, then evaluate them through representative scenarios.
Evaluation & assurance6 MIN READ
Golden traces are useful—but they are not gold standards
A reference trace can explain one acceptable journey. Treating it as the only correct path can punish valid behaviour and conceal unsafe alternatives.
Evaluation & assurance6 MIN READ
Product acceptance criteria for non-deterministic systems
Accept an agent against outcome distributions, critical invariants and recovery evidence—not one scripted response or an average score.
Evaluation & assurance6 MIN READ
How to evaluate a software-engineering agent on real repository work
A passing patch is only the start. Repository evaluation should test understanding, change quality, security, reviewability and recovery.
Evaluation & assurance6 MIN READ
Measure the action, not just the answer
A convincing message can conceal a failed or unauthorised operation. Agent evaluation must inspect tool calls, external state and control decisions.
Architecture & engineering6 MIN READ
How do you evaluate an agent orchestrator?
An orchestrator should be judged by the workflow it controls: decomposition, routing, state, authority, recovery, verification and stopping—not by a polished final response.
Evaluation & assurance6 MIN READ
What product leaders should demand from an agent evaluation
Product leaders need evaluation evidence that connects workflow outcomes, user experience and policy behaviour to a release choice—not a leaderboard score.
Evaluation & assurance6 MIN READ
Evaluating a customer-support agent from answer to action
A support agent should be judged on whether it resolves the right problem safely, not on whether its first answer sounds convincing.
Evaluation & assurance6 MIN READ
What confidence is enough for an agent release decision?
Confidence is not a universal percentage. It is a judgement about evidence quality, uncertainty, consequence and the reversibility of the proposed release.
Impact measurement6 MIN READ
Why AI productivity claims need a baseline
Time saved is not an outcome until the original work, quality, case mix and destination of released capacity are measured on comparable terms.
Evaluation & assurance6 MIN READ
Portable evaluations are protection against platform lock-in
Keep workflow cases, graders and decision rules separate from the platform so alternatives can be compared without rebuilding the standard of proof.
KEEP EXPLORING
Take the thinking
into your next discussion.
Read the method, assumptions and questions to ask.