Partial-execution recovery
Determine whether the agent can identify partial completion and recover accurately
When a multi-step task fails after some external effects occur, this area becomes relevant. For the agent operations and integration owner, the concern is specific: the agent repeats completed steps because it cannot determine where execution stopped. The deployment evaluation starts from that consequence and follows the relevant evidence through the customer workflow. Recovery should begin with what actually happened, not what the agent intended. The test should require evidence of completed and unresolved effects before continuation, retry, or compensation is proposed.
Connect the operating question to inspectable evidence
- Agent operations and integration owner
Owns the bounded decision and the consequence of acting.
- Multi-step action traces
Defines the starting context for the proposed work.
- Completed effect detection
Exposes the first material uncertainty to examine.
- Recovery and communication recommendations
Carries unresolved conditions into an accountable next action.
Decision at stake
Determine whether the agent can identify partial completion and recover accurately
Make the decision explicit
- Agent operations and integration owner
- A multi-step task fails after some external effects occur.
- Decision
- Whether the agent can identify partial completion and recover accurately.
- Completed effect detection
- Use multi-step action traces to establish the relevant starting condition.
For the agent operations and integration owner, this becomes relevant when a multi-step task fails after some external effects occur. The immediate decision is whether the agent can identify partial completion and recover accurately. The scope should be narrow enough that the evaluation team can identify a meaningful alternative, including continuing the current approach if the available evidence already supports it.
The practical failure to challenge is that the agent repeats completed steps because it cannot determine where execution stopped. That concern gives the work a concrete direction: establish what the relevant boundary should do, inspect how it behaves or would operate, and identify the owner of any gap. It should not be diluted into a broad transformation exercise before the specific question has been answered.
Coverage
Examine completed effect detection alongside safe continuation
Keep the review distinctions visible
- Completed effect detection
- Inspect authoritative external state after an interrupted action chain.
- Safe continuation
- Resume only the steps still needed after confirmed effects.
- User explanation of unresolved work
- Tell the user which effects remain uncertain or need reconciliation.
Completed effect detection
For this scope, completed effect detection should be examined alongside multi-step action traces. Determine which external effects actually completed before recovery begins. Use authoritative records or agreed verification calls rather than the agent’s plan alone. The proposed cases should include missing acknowledgments so the workflow must reconcile reality instead of assuming a step failed because its response was not received.
Safe continuation
For this scope, safe continuation should be examined alongside external state inspection. Resume only the missing authorised work once state is established. Test whether the recovery path accidentally repeats completed effects or overwrites later legitimate changes. The evaluation should preserve the operation identity and relevant state versions so a successful continuation can be explained and reproduced.
User explanation of unresolved work
For this scope, user explanation of unresolved work should be examined alongside recovery and compensation expectations. Review the explanation given to the user about completed and unresolved work. The response should not imply full success or a clean rollback when some effects remain. A useful finding connects misleading communication to the underlying state gap and specifies what verification or escalation is needed.
Prerequisites
Start with multi-step action traces
Give each input a specific job
- Multi-step action traces
- Defines the operating boundary.
- External state inspection
- Examine safe continuation.
- Recovery and compensation expectations
- Shows conditions affecting user explanation of unresolved work.
- Product or platform owner
- Confirm access and a reviewer who can resolve gaps.
The starting material is multi-step action traces, external state inspection, and recovery and compensation expectations. Review these together rather than assuming one document tells the whole story. A stated rule may differ from a configured behaviour, and a proposed responsibility may not yet have an available owner. Identify those differences before selecting the detailed work.
Participation should include a product or platform decision owner, a technical lead, a data owner, and a competent reviewer for the relevant boundary. The customer should define which information and environments may be used. Synthetic or sanitised examples can support preparation when they preserve the relevant behaviour; if they omit a material dependency, record the limitation and agree how the missing evidence would be obtained.
How the work proceeds
Tie the deployment finding to observable behaviour
Follow the evidence through the work
- Completed effect detection
Use multi-step action traces to define the decision criteria.
- Safe continuation
Inspect external state inspection and retain observations that challenge the expected result.
- User explanation of unresolved work
Use recovery and compensation expectations to identify unresolved conditions and a responsible owner.
- Recovery and communication recommendations
Link the recommendation to its evidence, remaining limits and next validation condition.
Define the authoritative source for each step’s completion and the evidence needed to resume. Test interruptions after different external effects, including a delayed acknowledgement that arrives during recovery. The proposed method should inspect whether the agent reconciles state before repeating work and whether it communicates unresolved effects accurately.
A practical first slice would focus on completed effect detection and then follow its consequences into safe continuation. Keep user explanation of unresolved work visible as a separate review question rather than assuming it is satisfied by the first result. The evaluation team should record observations, competing explanations, and changes to the original assumptions. A second reviewer can challenge whether the evidence supports the recommendation and whether an apparently successful example omitted the condition that matters most.
Illustrative scenario
Record exists, attachment fails: complete without duplication
Examine the constructed decision boundary
Illustrative scenario
- Record exists, attachment fails
A case workflow creates a record but does not attach its supporting document.
- Inspect existing state
Locate the created record and identify the missing authorised step.
- Complete without duplication
Attach to the existing record if permitted; do not recreate it or claim premature success.
Consider an illustrative situation in which a case workflow creates a record but fails before attaching the supporting document. This is a constructed example, not a client engagement or an observed Softlander outcome.
For record creation without attachment, an illustrative recovery would locate the existing record and complete only the missing authorised step. It should avoid creating another record or claiming the original task fully succeeded. The findings should identify missing correlation, state inspection, or recovery rules rather than reduce the issue to a generic retry failure.
The useful learning would be the difference between the expected behaviour and the behaviour or responsibility actually demonstrated. If the exercise leaves user explanation of unresolved work unresolved, that should remain a named next step. A recommendation to narrow the scope, improve a prerequisite, or hold the decision can be appropriate when the example exposes a material dependency.
Indicative artefacts
Use recovery and communication recommendations to make the next action explicit
Every artefact needs an operating use
Illustrative working format
- Partial-execution case set
- Boundary — multi-step action traces.
- State reconciliation findings
- Findings about safe continuation.
- Recovery and communication recommendations
- Action on user explanation of unresolved work.
Indicative outputs include partial-execution case set, state reconciliation findings, and recovery and communication recommendations. Their form should follow the decision and the customer’s existing operating records. The first should establish the assessed boundary and relevant observations; the second should make material gaps or dependencies inspectable; the third should identify the action, owner, and evidence needed to revisit the decision.
Acceptance should be tested through use: can the responsible owner explain what was examined, identify an unresolved issue, and act on the next step without reconstructing the whole engagement? Findings should preserve counterevidence and limits. A polished document is insufficient if it does not help resolve the customer’s specific question.
Scope and authority
A deployment finding remains bounded to the tested conditions
Keep analytical work separate from decision authority
- Authorised deployment cases and traces
- Can include partial-execution case set.
- Product or platform owner
- The customer retains release authority and owns remediation priorities.
- Universal vendor or component assurance
- Cannot be inferred from state reconciliation findings.
The proposed evaluation concerns the named deployment, configuration, cases, and authorised environment. Findings do not certify a component vendor, establish universal safety, or approve release. Unexamined provider paths and specialist questions remain explicit limitations.
Ways to engage
Related engagement scopes
AI Agent Evaluation Evidence Diagnostic
Find the gaps between your release decision and the evidence behind it. Start with one agent workflow and a bounded set of questions.
AI Agent Evaluation Foundation
Turn scattered tests into a repeatable release discipline. Connect the behaviours that matter to datasets, evaluation criteria and evidence your team can maintain.
AI Agent Component Evaluation
Find where agent behaviour breaks down. Follow memory, retrieval and delegation through your workflow, then isolate the components that need attention.
Continue exploring
Related insights and resources
Engineer the harness, not just the prompt
Define the contracts around an agent's decisions: what state it trusts, which actions can execute, how interrupted work resumes, and what evidence an operator needs to recover safely.
What would justify releasing this agent?
Replace a single pass rate with a reviewable chain of requirements, cases, observations and judgements. Distinguish task quality, control failures and uncertainty before increasing an agent's exposure.
The evidence behind the go-live decision
A working paper for leaders who need to explain why an agent workflow is ready for its next release. Includes a proposed evidence-pack structure, decision table and review agenda.