AI Agent Component Evaluation
Find where agent behaviour breaks down. Follow memory, retrieval and delegation through your workflow, then isolate the components that need attention.
Separate the mechanisms behind an inconsistent workflow
- Memory state
- Expose retained facts and their provenance.
- Context assembly
- Identify what evidence reaches the decision.
- Task routing
- Follow delegation, retries and termination.
- Connector effect
- Check partial execution and duplicate side effects.
Decision and fit
Separate the mechanisms behind an inconsistent workflow
Compare the engagement with the decision you face
- A question worth resolving
- Separate the mechanisms behind an inconsistent workflow.
- An accountable participant
- Engineering leads whose end-to-end evaluations show failures but cannot isolate memory, retrieval or coordination as the cause.
- Inspectable source material
- The relevant owners can make workflow evidence and agreed inputs available for examination.
- A scope mismatch
- You only need a high-level release evidence inventory.
The engagement
Find where agent behaviour breaks down. Follow memory, retrieval and delegation through your workflow, then isolate the components that need attention. A targeted component-evaluation project with explicit component boundaries and a controlled set of probes.
The decision at stake
The agent passes its end-to-end tests but behaves inconsistently in use: it forgets or over-remembers, cites the wrong material, loops, hands off badly between sub-agents, or fails in ways that are hard to reproduce.
Suitable situations
- Engineering leads whose end-to-end evaluations show failures but cannot isolate memory, retrieval or coordination as the cause.
- Platform teams choosing between memory, retrieval or orchestration designs and wanting evidence rather than preference.
- Organisations with privacy obligations over what an agent retains and to whom it can surface it.
Less suitable situations
- Simple single-tool assistants without memory or delegation, where the evaluation foundation is sufficient.
Engagement boundary
Agree the work and its boundaries together
The engagement boundary
- Memory state
- Memory: accuracy and relevance, retention and forgetting, cross-user leakage, poisoning resistance, deletion and correction, temporal consistency.
- Context assembly
- The inputs, authorised access and client participants are confirmed before scheduling.
- Agent platform owner
- Retains organisational decisions and risk acceptance.
- Additional implementation
- The engagement does not automatically evaluate every component or certify the full system. Broader workflow evaluation and engineering changes need separate scope and acceptance criteria.
What the work covers
- Memory: accuracy and relevance, retention and forgetting, cross-user leakage, poisoning resistance, deletion and correction, temporal consistency.
- Context and retrieval: relevance and grounding, permission-aware retrieval, citation fidelity, stale and conflicting context, compression effects, position sensitivity.
- Orchestration: routing and planner quality, delegation correctness, hand-off fidelity, loop and deadlock detection, termination, retry and compensation, failure propagation.
- Tools and connectors as they interact with the above, including partial execution and idempotency.
Scope boundaries
- Vendor selection on your behalf; comparative evidence is produced, the decision remains yours.
- Guarantees about behaviour outside the probed scope.
- The engagement does not automatically evaluate every component or certify the full system. Broader workflow evaluation and engineering changes need separate scope and acceptance criteria.
Working sequence
How the work moves from question to handover
What each stage should leave your team with
- Map the components
A selected subsystem and testable behaviour boundaries.
- Design probes
Component probes with controlled inputs and expected states.
- Run and measure
Reproducible failures tied to observed transitions.
- Interpret and recommend
Engineering actions paired with verification cases.
- Integrate
Harness additions and maintenance responsibilities.
How the work proceeds
- Map the components. Document how memory is written and read, how context is assembled, and how work is routed, so that probes target real mechanisms.
- Design probes. Construct datasets and scenarios that isolate each behaviour, including adversarial and privacy-sensitive cases.
- Run and measure. Execute the probes with repetition, capture traces, and measure against behaviour-specific criteria.
- Interpret and recommend. Explain findings in terms of mechanisms, propose changes, and define how each change would be verified.
- Integrate. Add the probes to the team's harness and hand over the maintenance practice.
Decision artefacts
Leave a record the next owner can use
Component probe record
Illustrative working format
- Subject and version
- Identify the workflow, configuration or proposal examined.
- Memory state
- Record the relevant requirement and supporting material.
- Task routing
- State the observation, supporting evidence and remaining uncertainty.
- Agent platform owner
- Record the next decision, follow-up and unresolved question.
What the work can produce
- Component evaluation design with datasets and probes for each behaviour in scope.
- Evidence report with reproducible cases, measurements and interpretation.
- Prioritised engineering recommendations with expected effect and how to verify it.
- Additions to the team's harness so the behaviours stay under test.
Worked situation
Follow one situation through to a bounded next decision
Illustrative working situation
Illustrative scenario
- Starting condition
A stale fact causes a planner to repeat a completed action.
- Question to examine
Isolate memory recall, context assembly and orchestration.
- Possible next action
Target the responsible mechanism and add a regression probe.
Engagement notes
Read the detailed engagement considerations
How the detailed notes connect
- Memory state
- Expose retained facts and their provenance.
- Context assembly
- Identify what evidence reaches the decision.
- Task routing
- Follow delegation, retries and termination.
Why component evaluation matters
End-to-end evaluation tells you whether the agent completed the task. It rarely tells you why it failed, or whether it succeeded for the wrong reason. Memory, context assembly and orchestration each introduce failure modes that only appear under specific conditions: a stale memory that surfaces after a correction, a permission boundary that retrieval quietly crosses, a planner that loops when a tool returns an unexpected shape. Component probes are designed to trigger those conditions deliberately and to make the result reproducible.
Privacy is part of the evaluation
Memory and retrieval decide what the agent knows about people and what it can reveal to whom. Probes for cross-user leakage, deletion and correction, and permission-aware retrieval are treated as first-class behaviours, and findings are written so that a privacy function can use them directly.
Working together
Your team and our team need a clear working agreement
Who contributes and who decides
- Client sponsor
- Confirms the decision, priorities and people needed to resolve it.
- Client technical or process owner
- Provides the agreed evidence, access and review of findings or deliverables.
- Softlander delivery lead
- Coordinates the agreed activities, review points, handover and open actions.
- Authorised decision-maker
- Accepts deliverables against the agreed criteria and separately decides operational or business action.
What the work depends on
- Component maintainers provide traces, versioned configuration, approved datasets and access to isolate the selected subsystem. A workflow expert judges ambiguous outcomes and validates what a component is supposed to contribute.
- Access to memory stores, retrieval pipelines, orchestration code and traces.
- Representative content and user scenarios, with any personal data handled under an agreed protocol.
- An environment where probes can run without affecting live users.
Engagement availability
The start date, named delivery roles, review points and fee basis are agreed after confirming the inputs and scope. Duration depends on evidence access, system complexity and the time your decision-makers need to review the work.
Engagement and handover
Agree the delivery model and what completion means
A clear agreement from scope to ownership
- Scoped work
- Memory: accuracy and relevance, retention and forgetting, cross-user leakage, poisoning resistance, deletion and correction, temporal consistency.
- Indicative handover
- Component evaluation design with datasets and probes for each behaviour in scope; Evidence report with reproducible cases, measurements and interpretation.
- Review and acceptance
- Review the agreed deliverables and record any open items, owners and conditions.
- Ongoing work
- Assign maintenance and operational ownership; agree additional scope and coverage explicitly.
A targeted component-evaluation project with explicit component boundaries and a controlled set of probes.
The evidence pack separates observed component behaviour from downstream effects, records reproduction conditions and supplies the agreed harness additions. A client maintainer demonstrates the selected probes during handover.
The engagement does not automatically evaluate every component or certify the full system. Broader workflow evaluation and engineering changes need separate scope and acceptance criteria.