Retry and compensation testing
Determine whether recovery preserves the intended business outcome
When external failures trigger retries or compensating actions, this area becomes relevant. For the workflow reliability engineer, the concern is specific: retries duplicate effects or compensation reverses a valid action. The deployment evaluation starts from that consequence and follows the relevant evidence through the customer workflow. Evaluate the business effects of recovery, not only exception handling in code. A timeout may leave an unknown outcome, and a compensating action needs its own authority, success criteria, and failure response.
Connect the operating question to inspectable evidence
- Workflow reliability engineer
Owns the bounded decision and the consequence of acting.
- Failure response contracts
Defines the starting context for the proposed work.
- Retry eligibility
Exposes the first material uncertainty to examine.
- Recovery recommendations
Carries unresolved conditions into an accountable next action.
Decision at stake
Determine whether recovery preserves the intended business outcome
Make the decision explicit
- Workflow reliability engineer
- External failures trigger retries or compensating actions.
- Decision
- Whether recovery preserves the intended business outcome.
- Retry eligibility
- Use failure response contracts to establish the relevant starting condition.
For the workflow reliability engineer, this becomes relevant when external failures trigger retries or compensating actions. The immediate decision is whether recovery preserves the intended business outcome. The scope should be narrow enough that the evaluation team can identify a meaningful alternative, including continuing the current approach if the available evidence already supports it.
The practical failure to challenge is that retries duplicate effects or compensation reverses a valid action. That concern gives the work a concrete direction: establish what the relevant boundary should do, inspect how it behaves or would operate, and identify the owner of any gap. It should not be diluted into a broad transformation exercise before the specific question has been answered.
Coverage
Examine retry eligibility alongside compensation ordering
Keep the review distinctions visible
- Retry eligibility
- Check the known business effect before permitting another attempt.
- Compensation ordering
- Apply compensation in the order required by the completed effects.
- Unresolved outcome handling
- Escalate an uncertain effect instead of assuming it is safe to repeat.
Retry eligibility
For this scope, retry eligibility should be examined alongside failure response contracts. Define which failures are eligible for retry and which require reconciliation first. A timeout may represent an unknown outcome rather than a failed action. The proposed test should inspect the external state before judging a repeated request safe, using controlled cases that isolate each interruption point.
Compensation ordering
For this scope, compensation ordering should be examined alongside retry policy. Order compensating actions according to the business dependencies they must restore. Compensation can itself create effects and fail. The evaluation should record its authority, expected result, and verification path rather than assume a reverse API call automatically returns the workflow to a valid state.
Unresolved outcome handling
For this scope, unresolved outcome handling should be examined alongside compensating action definitions. Preserve uncertainty when neither success nor failure can be established. The agent may need to stop and escalate instead of retrying indefinitely. The result should identify the evidence needed to resolve the outcome and should not report a clean recovery simply because the workflow reached a terminal software state.
Prerequisites
Start with failure response contracts
Give each input a specific job
- Failure response contracts
- Defines the operating boundary.
- Retry policy
- Examine compensation ordering.
- Compensating action definitions
- Shows conditions affecting unresolved outcome handling.
- Product or platform owner
- Confirm access and a reviewer who can resolve gaps.
The starting material is failure response contracts, retry policy, and compensating action definitions. Review these together rather than assuming one document tells the whole story. A stated rule may differ from a configured behaviour, and a proposed responsibility may not yet have an available owner. Identify those differences before selecting the detailed work.
Participation should include a product or platform decision owner, a technical lead, a data owner, and a competent reviewer for the relevant boundary. The customer should define which information and environments may be used. Synthetic or sanitised examples can support preparation when they preserve the relevant behaviour; if they omit a material dependency, record the limitation and agree how the missing evidence would be obtained.
How the work proceeds
Tie the deployment finding to observable behaviour
Follow the evidence through the work
- Retry eligibility
Use failure response contracts to define the decision criteria.
- Compensation ordering
Inspect retry policy and retain observations that challenge the expected result.
- Unresolved outcome handling
Use compensating action definitions to identify unresolved conditions and a responsible owner.
- Recovery recommendations
Link the recommendation to its evidence, remaining limits and next validation condition.
Build a failure matrix around the point of interruption: before dispatch, after dispatch with unknown outcome, after verified effect, and during compensation. Define permitted next actions for each. The proposed evaluation should inspect external state so a recovered conversation is not confused with a recovered business process.
A practical first slice would focus on retry eligibility and then follow its consequences into compensation ordering. Keep unresolved outcome handling visible as a separate review question rather than assuming it is satisfied by the first result. The evaluation team should record observations, competing explanations, and changes to the original assumptions. A second reviewer can challenge whether the evidence supports the recommendation and whether an apparently successful example omitted the condition that matters most.
Illustrative scenario
Reservation without local record: failed compensation branch
Examine the constructed decision boundary
Illustrative scenario
- Reservation without local record
The resource is reserved, but recording that fact in another system fails.
- Reconcile completed effect
Inspect authoritative state before choosing completion or authorised compensation.
- Failed compensation branch
If undo also fails, preserve the unresolved reservation and escalate without claiming rollback.
Consider an illustrative situation in which an agent reserves a resource successfully but fails while recording the reservation in another system. This is a constructed example, not a client engagement or an observed Softlander outcome.
For resource reservation, an illustrative test would let the reservation succeed while the local record fails. A safe response depends on reconciliation and an authorised choice to complete or compensate. If compensation also fails, the workflow should preserve an unresolved state and escalate rather than claim a clean rollback.
The useful learning would be the difference between the expected behaviour and the behaviour or responsibility actually demonstrated. If the exercise leaves unresolved outcome handling unresolved, that should remain a named next step. A recommendation to narrow the scope, improve a prerequisite, or hold the decision can be appropriate when the example exposes a material dependency.
Indicative artefacts
Use recovery recommendations to make the next action explicit
Every artefact needs an operating use
Illustrative working format
- Retry and compensation matrix
- Boundary — failure response contracts.
- Effect reconciliation findings
- Findings about compensation ordering.
- Recovery recommendations
- Action on unresolved outcome handling.
Indicative outputs include retry and compensation matrix, effect reconciliation findings, and recovery recommendations. Their form should follow the decision and the customer’s existing operating records. The first should establish the assessed boundary and relevant observations; the second should make material gaps or dependencies inspectable; the third should identify the action, owner, and evidence needed to revisit the decision.
Acceptance should be tested through use: can the responsible owner explain what was examined, identify an unresolved issue, and act on the next step without reconstructing the whole engagement? Findings should preserve counterevidence and limits. A polished document is insufficient if it does not help resolve the customer’s specific question.
Scope and authority
A deployment finding remains bounded to the tested conditions
Keep analytical work separate from decision authority
- Authorised deployment cases and traces
- Can include retry and compensation matrix.
- Product or platform owner
- The customer retains release authority and owns remediation priorities.
- Universal vendor or component assurance
- Cannot be inferred from effect reconciliation findings.
The proposed evaluation concerns the named deployment, configuration, cases, and authorised environment. Findings do not certify a component vendor, establish universal safety, or approve release. Unexamined provider paths and specialist questions remain explicit limitations.
Ways to engage
Related engagement scopes
AI Agent Evaluation Evidence Diagnostic
Find the gaps between your release decision and the evidence behind it. Start with one agent workflow and a bounded set of questions.
AI Agent Evaluation Foundation
Turn scattered tests into a repeatable release discipline. Connect the behaviours that matter to datasets, evaluation criteria and evidence your team can maintain.
AI Agent Component Evaluation
Find where agent behaviour breaks down. Follow memory, retrieval and delegation through your workflow, then isolate the components that need attention.
Continue exploring
Related insights and resources
Engineer the harness, not just the prompt
Define the contracts around an agent's decisions: what state it trusts, which actions can execute, how interrupted work resumes, and what evidence an operator needs to recover safely.
What would justify releasing this agent?
Replace a single pass rate with a reviewable chain of requirements, cases, observations and judgements. Distinguish task quality, control failures and uncertainty before increasing an agent's exposure.
The evidence behind the go-live decision
A working paper for leaders who need to explain why an agent workflow is ready for its next release. Includes a proposed evidence-pack structure, decision table and review agenda.