AI Agent Evaluation Foundation
Turn scattered tests into a repeatable release discipline. Connect the behaviours that matter to datasets, evaluation criteria and evidence your team can maintain.
Make release evidence repeatable after every change
- Behaviour taxonomy
- Connect task, authority and recovery criteria.
- Case provenance
- Record why an input represents the intended workflow.
- Grader calibration
- Compare automated judgements with reviewed examples.
- Regression run
- Keep cases, configuration and findings linked.
Decision and fit
Make release evidence repeatable after every change
Compare the engagement with the decision you face
- A question worth resolving
- Make release evidence repeatable after every change.
- An accountable participant
- AI engineering leads who need a maintainable evaluation harness and release criteria for a tool-using workflow.
- Inspectable source material
- The relevant owners can make workflow evidence and agreed inputs available for examination.
- A scope mismatch
- The requirement is only a one-off model leaderboard comparison.
The engagement
Turn scattered tests into a repeatable release discipline. Connect the behaviours that matter to datasets, evaluation criteria and evidence your team can maintain. A scoped implementation project that builds evaluation assets with your engineering team and hands over their operation.
The decision at stake
An agent is being built or already running, but the only evidence of its behaviour is demos and anecdotes. Nobody can say with confidence how it performs across its scope, how it degrades, or what would justify a release or a rollback.
Suitable situations
- AI engineering leads who need a maintainable evaluation harness and release criteria for a tool-using workflow.
- Organisations that need release criteria a risk or governance function can accept.
- Teams that have prompts and integrations but no regression safety net when models, tools or prompts change.
Less suitable situations
- Single-turn model benchmarking without tools or workflows; this engagement is about systems that act.
- Teams unwilling to invest in maintaining evaluation assets after the engagement ends.
Engagement boundary
Agree the work and its boundaries together
The engagement boundary
- Behaviour taxonomy
- Evaluation taxonomy covering task success, tool use, policy compliance, recovery, cost and latency, and human hand-off.
- Case provenance
- The inputs, authorised access and client participants are confirmed before scheduling.
- Engineering owner
- Retains organisational decisions and risk acceptance.
- Additional implementation
- Your team owns maintenance after handover. Adding new workflows, recalibrating graders after material changes or running an ongoing evaluation programme requires a further agreement.
What the work covers
- Evaluation taxonomy covering task success, tool use, policy compliance, recovery, cost and latency, and human hand-off.
- Dataset and golden-trace design drawn from real workflow cases, including edge cases and failure modes.
- Harness architecture, graders (deterministic, model-assisted and human) and reporting.
- Release criteria and regression policy tied to the taxonomy.
- Knowledge transfer so the team owns and extends the capability.
Scope boundaries
- Claims that passing the criteria guarantees safe behaviour in production; evaluation reduces uncertainty, it does not remove it.
- Independent certification or attestation; the engagement builds capability with your team.
- Ongoing operation of the harness after handover unless separately agreed.
- Your team owns maintenance after handover. Adding new workflows, recalibrating graders after material changes or running an ongoing evaluation programme requires a further agreement.
Working sequence
How the work moves from question to handover
What each stage should leave your team with
- Define behaviours that matter
A workflow-specific behaviour taxonomy.
- Build the evidence base
Versioned cases and traces with provenance.
- Stand up the harness
Runnable graders, reports and execution instructions.
- Set release criteria
Criteria that distinguish passes, failures and uncertainty.
- Transfer ownership
A maintainer able to run and extend the suite.
How the work proceeds
- Define behaviours that matter. Work from the workflow and its failure costs to a taxonomy of behaviours worth measuring, including the ones that are hard to measure.
- Build the evidence base. Assemble datasets and traces from real cases and deliberate adversarial construction, recording where each came from and what it exercises.
- Stand up the harness. Implement graders and reporting that the team can run on every change, with clear handling of non-determinism.
- Set release criteria. Translate the taxonomy into criteria a release decision can be made against, with explicit thresholds and known limitations.
- Transfer ownership. Run the first evaluation with the team, document maintenance practices and leave the capability in their hands.
Decision artefacts
Leave a record the next owner can use
Coverage record
Illustrative working format
- Subject and version
- Identify the workflow, configuration or proposal examined.
- Behaviour taxonomy
- Record the relevant requirement and supporting material.
- Grader calibration
- State the observation, supporting evidence and remaining uncertainty.
- Engineering owner
- Record the next decision, follow-up and unresolved question.
What the work can produce
- Evaluation taxonomy and coverage map for the agent's scope.
- Initial evaluation datasets and golden traces with provenance recorded.
- Running harness with reporting, integrated into the team's delivery pipeline where feasible.
- Release criteria, regression policy and a first evaluation report against them.
Worked situation
Follow one situation through to a bounded next decision
Illustrative working situation
Illustrative scenario
- Starting condition
A prompt change improves answers but alters tool arguments.
- Question to examine
Run unchanged cases and inspect action-level evidence.
- Possible next action
Retain the regression case and review the release criteria.
Engagement notes
Read the detailed engagement considerations
How the detailed notes connect
- Behaviour taxonomy
- Connect task, authority and recovery criteria.
- Case provenance
- Record why an input represents the intended workflow.
- Grader calibration
- Compare automated judgements with reviewed examples.
Why a foundation rather than a test suite
Evaluation for agentic systems tends to fail in one of two ways. Either the team measures what is easy, such as final-answer accuracy on a handful of cases, and misses tool misuse, unsafe recovery or drift; or it builds an elaborate benchmark that nobody maintains and that the agent quietly outgrows. A foundation addresses both: it starts from the behaviours that matter for the workflow and its failure costs, and it leaves behind a capability the team runs on every change.
What the taxonomy typically covers
- Task success across representative and edge cases, judged against workflow-defined outcomes.
- Tool selection and argument correctness, unauthorised tool use, and handling of partial execution.
- Policy compliance, including refusals, escalations and confirmations for high-impact actions.
- Recovery from tool errors, stale context and conflicting instructions.
- Cost, latency and step counts, with regressions treated as findings.
- Quality of human hand-off: what the person receives, and whether they can act on it.
Each behaviour is mapped to the datasets or traces that exercise it, so coverage gaps are visible rather than implied.
Working with your delivery pipeline
Where the team’s pipeline allows it, the harness runs on every change to prompts, tools or models and reports against the release criteria. Where it does not, the engagement documents the manual procedure and the conditions under which it should be automated.
Working together
Your team and our team need a clear working agreement
Who contributes and who decides
- Client sponsor
- Confirms the decision, priorities and people needed to resolve it.
- Client technical or process owner
- Provides the agreed evidence, access and review of findings or deliverables.
- Softlander delivery lead
- Coordinates the agreed activities, review points, handover and open actions.
- Authorised decision-maker
- Accepts deliverables against the agreed criteria and separately decides operational or business action.
What the work depends on
- Provide a code owner, workflow subject-matter expert and delivery-pipeline maintainer. They contribute representative tasks, expected outcomes, approved test data, repository access and review of grader disagreements.
- Access to the agent's code, prompts, tool definitions and traces or logs.
- Subject-matter input to judge what a correct outcome looks like in the workflow.
- An environment in which the agent can be exercised safely, or support to build one.
Engagement availability
The start date, named delivery roles, review points and fee basis are agreed after confirming the inputs and scope. Duration depends on evidence access, system complexity and the time your decision-makers need to review the work.
Engagement and handover
Agree the delivery model and what completion means
A clear agreement from scope to ownership
- Scoped work
- Evaluation taxonomy covering task success, tool use, policy compliance, recovery, cost and latency, and human hand-off.
- Indicative handover
- Evaluation taxonomy and coverage map for the agent's scope; Initial evaluation datasets and golden traces with provenance recorded.
- Review and acceptance
- Review the agreed deliverables and record any open items, owners and conditions.
- Ongoing work
- Assign maintenance and operational ownership; agree additional scope and coverage explicitly.
A scoped implementation project that builds evaluation assets with your engineering team and hands over their operation.
A nominated maintainer runs the agreed evaluation suite from the supplied instructions, traces reported results to cases and changes a case or criterion without relying on the original implementer. The handover records coverage gaps and any pipeline integration that remains manual.
Your team owns maintenance after handover. Adding new workflows, recalibrating graders after material changes or running an ongoing evaluation programme requires a further agreement.