Connector reliability testing
Determine whether the integration remains observable and recoverable under agreed failure conditions
When connector behaviour varies under outages or changing upstream systems, this area becomes relevant. For the platform integration and SRE owner, the concern is specific: a connector converts an upstream failure into an empty result that the agent treats as fact. The deployment evaluation starts from that consequence and follows the relevant evidence through the customer workflow. Assess the meaning of failures delivered to the agent. Reliability includes making a dependency problem visible and recoverable; a wrapper that hides errors may appear stable while undermining the workflow's decisions.
Connect the operating question to inspectable evidence
- Platform integration and SRE owner
Owns the bounded decision and the consequence of acting.
- Connector contracts
Defines the starting context for the proposed work.
- Connection loss
Exposes the first material uncertainty to examine.
- Monitoring and recovery backlog
Carries unresolved conditions into an accountable next action.
Decision at stake
Determine whether the integration remains observable and recoverable under agreed failure conditions
Make the decision explicit
- Platform integration and SRE owner
- Connector behaviour varies under outages or changing upstream systems.
- Decision
- Whether the integration remains observable and recoverable under agreed failure conditions.
- Connection loss
- Use connector contracts to establish the relevant starting condition.
For the platform integration and SRE owner, this becomes relevant when connector behaviour varies under outages or changing upstream systems. The immediate decision is whether the integration remains observable and recoverable under agreed failure conditions. The scope should be narrow enough that the evaluation team can identify a meaningful alternative, including continuing the current approach if the available evidence already supports it.
The practical failure to challenge is that a connector converts an upstream failure into an empty result that the agent treats as fact. That concern gives the work a concrete direction: establish what the relevant boundary should do, inspect how it behaves or would operate, and identify the owner of any gap. It should not be diluted into a broad transformation exercise before the specific question has been answered.
Coverage
Examine connection loss alongside schema changes
Keep the review distinctions visible
- Connection loss
- Interrupt connectivity before and after a possible external effect.
- Schema changes
- Change a response schema within the authorised fixture.
- Rate and transient failure handling
- Exercise throttling and transient faults without hiding retry consequences.
Connection loss
For this scope, connection loss should be examined alongside connector contracts. Simulate connection loss at agreed points and inspect how the connector reports it. Preserve correlation so the team can distinguish a failed request from an unknown outcome. The proposed evaluation should connect the technical failure to the agent’s business interpretation and recovery behaviour.
Schema changes
For this scope, schema changes should be examined alongside timeout and error policies. Introduce an authorised response-shape change that tests compatibility assumptions. A connector may reject it explicitly or silently misread it. The finding should describe which contract changed, what the agent received, and whether the integration owner has a detectable failure and maintainable response path.
Rate and transient failure handling
For this scope, rate and transient failure handling should be examined alongside authorised failure fixtures. Exercise rate or transient failures according to the permitted test boundary. Inspect retry behaviour, resource use, and user status without making universal availability claims. The useful output is a profile of observed behaviour under defined conditions and a backlog tied to the responsible connector or upstream owner.
Prerequisites
Start with connector contracts
Give each input a specific job
- Connector contracts
- Defines the operating boundary.
- Timeout and error policies
- Examine schema changes.
- Authorised failure fixtures
- Shows conditions affecting rate and transient failure handling.
- Product or platform owner
- Confirm access and a reviewer who can resolve gaps.
The starting material is connector contracts, timeout and error policies, and authorised failure fixtures. Review these together rather than assuming one document tells the whole story. A stated rule may differ from a configured behaviour, and a proposed responsibility may not yet have an available owner. Identify those differences before selecting the detailed work.
Participation should include a product or platform decision owner, a technical lead, a data owner, and a competent reviewer for the relevant boundary. The customer should define which information and environments may be used. Synthetic or sanitised examples can support preparation when they preserve the relevant behaviour; if they omit a material dependency, record the limitation and agree how the missing evidence would be obtained.
How the work proceeds
Tie the deployment finding to observable behaviour
Follow the evidence through the work
- Connection loss
Use connector contracts to define the decision criteria.
- Schema changes
Inspect timeout and error policies and retain observations that challenge the expected result.
- Rate and transient failure handling
Use authorised failure fixtures to identify unresolved conditions and a responsible owner.
- Monitoring and recovery backlog
Link the recommendation to its evidence, remaining limits and next validation condition.
Exercise the connector under authorised transient failures and contract variations, preserving a known expected business result. Inspect how it exposes errors, correlates requests, and supports recovery. The proposed evaluation should also distinguish integration faults from upstream service behaviour so remediation is assigned to the component that can address it.
A practical first slice would focus on connection loss and then follow its consequences into schema changes. Keep rate and transient failure handling visible as a separate review question rather than assuming it is satisfied by the first result. The evaluation team should record observations, competing explanations, and changes to the original assumptions. A second reviewer can challenge whether the evidence supports the recommendation and whether an apparently successful example omitted the condition that matters most.
Illustrative scenario
Timeout resembles no records: preserve the distinction
Examine the constructed decision boundary
Illustrative scenario
- Timeout resembles no records
An enterprise query loses access to data during a simulated dependency failure.
- Two distinct responses
Compare an unavailable result with a valid empty result.
- Preserve the distinction
Check response semantics and diagnostic telemetry before choosing connector or recovery changes.
Consider an illustrative situation in which an agent queries an enterprise service during a simulated timeout and must distinguish unavailable data from no matching records. This is a constructed example, not a client engagement or an observed Softlander outcome.
For a simulated timeout, an illustrative case would compare unavailable data with a valid empty result. The agent should not treat them as equivalent. Findings should describe whether the connector preserves that distinction and whether operating telemetry gives the responsible team enough information to diagnose the dependency problem.
The useful learning would be the difference between the expected behaviour and the behaviour or responsibility actually demonstrated. If the exercise leaves rate and transient failure handling unresolved, that should remain a named next step. A recommendation to narrow the scope, improve a prerequisite, or hold the decision can be appropriate when the example exposes a material dependency.
Indicative artefacts
Use monitoring and recovery backlog to make the next action explicit
Every artefact needs an operating use
Illustrative working format
- Connector reliability profile
- Boundary — connector contracts.
- Failure handling cases
- Findings about schema changes.
- Monitoring and recovery backlog
- Action on rate and transient failure handling.
Indicative outputs include connector reliability profile, failure handling cases, and monitoring and recovery backlog. Their form should follow the decision and the customer’s existing operating records. The first should establish the assessed boundary and relevant observations; the second should make material gaps or dependencies inspectable; the third should identify the action, owner, and evidence needed to revisit the decision.
Acceptance should be tested through use: can the responsible owner explain what was examined, identify an unresolved issue, and act on the next step without reconstructing the whole engagement? Findings should preserve counterevidence and limits. A polished document is insufficient if it does not help resolve the customer’s specific question.
Scope and authority
A deployment finding remains bounded to the tested conditions
Keep analytical work separate from decision authority
- Authorised deployment cases and traces
- Can include connector reliability profile.
- Product or platform owner
- The customer retains release authority and owns remediation priorities.
- Universal vendor or component assurance
- Cannot be inferred from failure handling cases.
The proposed evaluation concerns the named deployment, configuration, cases, and authorised environment. Findings do not certify a component vendor, establish universal safety, or approve release. Unexamined provider paths and specialist questions remain explicit limitations.
Ways to engage
Related engagement scopes
AI Agent Evaluation Evidence Diagnostic
Find the gaps between your release decision and the evidence behind it. Start with one agent workflow and a bounded set of questions.
AI Agent Evaluation Foundation
Turn scattered tests into a repeatable release discipline. Connect the behaviours that matter to datasets, evaluation criteria and evidence your team can maintain.
AI Agent Component Evaluation
Find where agent behaviour breaks down. Follow memory, retrieval and delegation through your workflow, then isolate the components that need attention.
Continue exploring
Related insights and resources
Engineer the harness, not just the prompt
Define the contracts around an agent's decisions: what state it trusts, which actions can execute, how interrupted work resumes, and what evidence an operator needs to recover safely.
What would justify releasing this agent?
Replace a single pass rate with a reviewable chain of requirements, cases, observations and judgements. Distinguish task quality, control failures and uncertainty before increasing an agent's exposure.
The evidence behind the go-live decision
A working paper for leaders who need to explain why an agent workflow is ready for its next release. Includes a proposed evidence-pack structure, decision table and review agenda.