Evaluation & assurance5 MIN READ
Your agent passed the test. What does that prove?
A release decision needs a traceable connection between expected behaviour, test conditions and observed results. Here is a practical way to build that connection.
THE SOFTLANDER BLOG
Evaluation & assurance
How to define, produce and maintain evidence that an agentic system behaves as intended across its scope.
In Evaluation & assurance
44 articlesArchitecture & engineering6 MIN READ
A CTO’s checklist before an agent reaches production
Production readiness for an AI agent spans workflow requirements, system identity, evaluation, recovery and operations—not just a working model endpoint.
Architecture & engineering7 MIN READ
A failure ladder for retrieval-augmented agents
A wrong answer is the final symptom of failures that may begin with source coverage, access control, parsing, retrieval, context assembly, generation or action.
Governance & privacy7 MIN READ
What should trigger an agent governance review?
Review should follow material changes in purpose, authority, data, users, dependencies, evidence or incidents—not an arbitrary calendar alone.
Evaluation & assurance6 MIN READ
Golden traces are useful—but they are not gold standards
A reference trace can explain one acceptable journey. Treating it as the only correct path can punish valid behaviour and conceal unsafe alternatives.
Impact measurement6 MIN READ
How to estimate the cost of an agent failure
Model concrete failure scenarios across detection, containment, correction, disruption and consequence, then use ranges to choose controls.
Evaluation & assurance6 MIN READ
How to design a joint AI evaluation lab
A joint lab should be a governed way to create decision evidence across organisations, with shared methods but explicit ownership of systems, data and conclusions.
Evaluation & assurance6 MIN READ
Product acceptance criteria for non-deterministic systems
Accept an agent against outcome distributions, critical invariants and recovery evidence—not one scripted response or an average score.
Evaluation & assurance6 MIN READ
Ten artefacts that make an AI-agent decision reviewable
Reviewable agent decisions depend on a small chain of artefacts connecting workflow purpose, authority, configuration, evidence, findings, controls and ownership.
Evaluation & assurance6 MIN READ
How to evaluate a software-engineering agent on real repository work
A passing patch is only the start. Repository evaluation should test understanding, change quality, security, reviewability and recovery.
Evaluation & assurance6 MIN READ
Measure the action, not just the answer
A convincing message can conceal a failed or unauthorised operation. Agent evaluation must inspect tool calls, external state and control decisions.
Impact measurement6 MIN READ
How to write an AI success story that survives scrutiny
A credible AI success story separates what was delivered from what changed, defines the baseline, shows the contribution chain and preserves uncertainty and permission.
KEEP EXPLORING
Take the thinking
into your next discussion.
Read the method, assumptions and questions to ask.
A little more clarity
Questions worth asking.
Start with the questions that shape a better decision.
Which enterprise agent questions does the Softlander blog examine?
The blog examines where agents create business value, how teams define release evidence, how authority and tool access should be bounded, and how engineering, security and governance owners can make a shared decision.
Who are the articles written for?
They are written for product, engineering, AI platform, security, governance and transformation leaders responsible for moving an agent workflow from idea or pilot towards accountable operation.
How do the articles relate to Softlander’s services?
Articles explain the problems and decision methods behind the consultancy’s service direction. They are educational material and do not replace a scoped review of an organisation’s actual system.