Agent testing infrastructure
Decide which infrastructure can reproduce meaningful cases without uncontrolled effects
When tests need repeatable execution across changing agent components, this area becomes relevant. For the evaluation infrastructure lead, the concern is specific: a test passes because mocks hide the failure behaviour of the real dependency. The engineering design starts from that consequence and follows the relevant evidence through the customer workflow. Testing infrastructure should support the difference between a fast isolated check and a representative system test. The design should make substitutions visible so reviewers understand which conclusions depend on mocks, synthetic data, or real integrations.
The engineering decisions relationship
- Case inventory
Establish the basis for controlled fixture.
- Effect isolation
Keep test actions within authorised synthetic boundaries.
- Testing architecture
Retain the scoped finding and its unresolved dependencies.
Decision
Frame the decision around controlled fixture
Questions and the evidence they need
- Controlled fixture
- Inspect case inventory for the relevant constraint or observation.
- Effect isolation
- Inspect environment constraints for the relevant constraint or observation.
- Evidence identity
- Inspect component versioning requirements for the relevant constraint or observation.
For the evaluation infrastructure lead, this becomes relevant when tests need repeatable execution across changing agent components. The immediate decision is which infrastructure can reproduce meaningful cases without uncontrolled effects. The scope should be narrow enough that the engineering team can identify a meaningful alternative, including continuing the current approach if the available evidence already supports it.
The practical failure to challenge is that a test passes because mocks hide the failure behaviour of the real dependency. That concern gives the work a concrete direction: establish what the relevant boundary should do, inspect how it behaves or would operate, and identify the owner of any gap. It should not be diluted into a broad transformation exercise before the specific question has been answered.
Mechanism
Testing infrastructure should isolate effects and retain provenance
Relationships within engineering decisions
- Controlled fixture
- Version retrieval, tool responses, and task conditions.
- Effect isolation
- Keep test actions within authorised synthetic boundaries.
- Evidence identity
- Connect each result and artefact to the run configuration.
Fixture control
For this scope, fixture control should be examined alongside case inventory. Control fixtures according to the behaviour each test is intended to examine. Synthetic and mocked dependencies should be labelled explicitly. A fast argument test can be useful without proving real authorisation, retrieval, or recovery behaviour that the fixture does not preserve.
Effect isolation
For this scope, effect isolation should be examined alongside environment constraints. Keep external effects inside an approved test boundary. The infrastructure should support representative behaviour without accidentally using production credentials or records. Tests that require stronger integration evidence should have a defined environment and authorised owner rather than quietly expanding a unit-test harness.
Result and artefact provenance
For this scope, result and artefact provenance should be examined alongside component versioning requirements. Preserve configuration, expected outcomes, traces, and result provenance. A reviewer should be able to reproduce or challenge a finding without guessing which prompt or tool version ran. Fixture changes should retain the test’s meaning or document why the expectation was deliberately revised.
Preparation
Connect the working inputs to the boundary being examined
Evidence packet for controlled fixture
Illustrative working format
- Case inventory
- Match this description to the deployed or proposed boundary being examined.
- Environment constraints
- Check its relationship to the relevant workflow and expose missing context.
- Component versioning requirements
- Confirm the applicable revision and who resolves disputed interpretation.
The starting material is case inventory, environment constraints, and component versioning requirements. Review these together rather than assuming one document tells the whole story. A stated rule may differ from a configured behaviour, and a proposed responsibility may not yet have an available owner. Identify those differences before selecting the detailed work.
Participation should include a workflow owner, an implementation lead, an eventual maintainer, and qualified control reviewers where needed. The customer should define which information and environments may be used. Synthetic or sanitised examples can support preparation when they preserve the relevant behaviour; if they omit a material dependency, record the limitation and agree how the missing evidence would be obtained.
How the work proceeds
Carry controlled fixture through to an accountable finding
From the scoped material to a decision record
- Case inventory
Make the component and operating boundary explicit.
- Effect isolation
Apply the scoped check to evidence identity.
- Testing architecture
Record the evidence, open dependencies, and the owner’s next decision.
Separate test fixtures by the behaviour they preserve. A mocked tool may support argument checks but not prove real permission enforcement or recovery semantics. The proposed infrastructure should record these substitutions with each result and allow representative integration cases to run within an approved boundary when stronger evidence is needed.
A practical first slice would focus on fixture control and then follow its consequences into effect isolation. Keep result and artefact provenance visible as a separate review question rather than assuming it is satisfied by the first result. The engineering team should record observations, competing explanations, and changes to the original assumptions. A second reviewer can challenge whether the evidence supports the recommendation and whether an apparently successful example omitted the condition that matters most.
Indicative artefact
Make the testing architecture usable by its owner
Testing architecture
Illustrative working format
- Testing architecture
- Identify the assessed workflow, revision, and decision boundary.
- Fixture and evidence conventions
- Retain selection rationale, accepted alternatives, and unexamined task conditions.
- Integration test backlog
- Name the missing evidence, responsible owner, and condition for revisiting the decision.
Indicative outputs include testing architecture, fixture and evidence conventions, and integration test backlog. Their form should follow the decision and the customer’s existing operating records. The first should establish the assessed boundary and relevant observations; the second should make material gaps or dependencies inspectable; the third should identify the action, owner, and evidence needed to revisit the decision.
Acceptance should be tested through use: can the responsible owner explain what was examined, identify an unresolved issue, and act on the next step without reconstructing the whole engagement? Findings should preserve counterevidence and limits. A polished document is insufficient if it does not help resolve the customer’s specific question.
Illustrative scenario
Examine evidence identity through a constructed exception
A constructed case and its decision paths
Illustrative scenario
- Controlled fixture
Release checks exercise retrieval, planning, and tools in a controlled environment.
- Effect isolation
Repeat the case with fixed fixtures and inspect its isolated tool effects.
- Evidence identity
Retain configuration and artefact provenance with the acceptance result.
Consider an illustrative situation in which a team wants release checks that exercise retrieval, planning, and tool responses in a controlled environment. This is a constructed example, not a client engagement or an observed Softlander outcome.
For the release-check example, an illustrative suite would combine fast isolated cases with a smaller controlled workflow run. A failure should retain configuration, inputs, traces, and expected behaviour so another reviewer can reproduce it. Acceptance should include the customer’s ability to update a fixture without silently changing the meaning of the test.
The useful learning would be the difference between the expected behaviour and the behaviour or responsibility actually demonstrated. If the exercise leaves result and artefact provenance unresolved, that should remain a named next step. A recommendation to narrow the scope, improve a prerequisite, or hold the decision can be appropriate when the example exposes a material dependency.
Scope and next action
Keep the conclusion within the defined workflow and configuration
Confirm fit before the next commitment
- Controlled fixture
- This review is limited to the defined workflow and configuration.
- Case inventory
- The relevant owner must confirm access and the intended operating conditions.
- Testing architecture
- Production acceptance needs separate scoped evidence.
The method described here is proposed engineering work. Scope, staffing, implementation access, acceptance, and commercial terms require a readiness review. A design or lab result does not establish production suitability, guaranteed reliability, or security certification.
Ways to engage
Related engagement scopes
AI Fractional Agentic AI Office
Create a regular place for agentic AI decisions. Connect initiative priorities, architecture questions, risk ownership and evidence reviews.
AI Fractional Agentic Transformation Office
Coordinate the operating changes behind agent adoption. Align workflow redesign, ownership, dependencies and adoption measures.
AI Harness Engineering Pod
Engineer a bounded AI execution harness with tool adapters, context handling, orchestration, recovery and observable traces.
Continue exploring
Related insights and resources
Engineer the harness, not just the prompt
Define the contracts around an agent's decisions: what state it trusts, which actions can execute, how interrupted work resumes, and what evidence an operator needs to recover safely.
Governing agent decisions
Governance becomes useful when it changes a decision. Connect each agent workflow to an accountable owner, explicit operating conditions, proportionate evidence and a practical route for exceptions and withdrawal.
Do you need an agent centre of excellence?
A centre can concentrate scarce expertise and reusable controls, or become an approval queue detached from real work. Choose the mandate before the structure.