Extended-task agent evaluation
Evaluate an agent through the full duration of a demanding task
Extended-task agent evaluation becomes relevant when an agent performs well on short examples but must sustain work through many steps and interruptions. A focused review would examine a complete extended task with checkpoints, interruptions, dependencies and a defined stopping condition. This is most relevant to agent product owners and evaluation engineers, especially where teams must distinguish step success and task completion.
Follow the agent through a declared change sequence
- Multi-step task specifications
Identify the initial state and what can be reconstructed.
- Step success
Introduce or observe the relevant changed condition.
- State, tool and handoff traces
Compare behaviour while retaining version and workload context.
- A duration and authority decision brief
Define the evidence needed before extending the conclusion.
Decision at stake
Evaluate an agent through the full duration of a demanding task
Make the decision explicit
- Agent product owners and evaluation engineers
- An agent performs well on short examples but must sustain work through many steps and interruptions.
- Decision
- Whether the proposed task duration and authority are supported by evidence about completion, recovery and control.
- Step success
- Use multi-step task specifications to establish the relevant starting condition.
For agent product owners and evaluation engineers, the immediate question is whether the proposed task duration and authority are supported by evidence about completion, recovery and control. This is most useful when there is a real choice between proceeding, changing the design, collecting more evidence or narrowing the intended use. The starting conversation should identify that choice and its consequence before deciding how much work the topic warrants.
An agent could continue producing plausible intermediate work after losing the original requirement or failing to recover from an early error. A useful review follows a complete extended task with checkpoints, interruptions, dependencies and a defined stopping condition, rather than stopping at a convenient component metric or document boundary. The evidence should separate observation, interpretation and recommendation so the accountable owner can see what it supports.
Coverage
Step success and task completion need different evidence
Keep the review distinctions visible
- Step success
- Compare with task completion.
- Elapsed duration
- Compare with meaningful work.
- Persistent activity
- Compare with controlled progress.
Step success versus task completion
The task definition would preserve intermediate requirements, dependencies and stopping conditions. Evaluation would inspect whether the agent makes meaningful progress, rather than count prolonged activity as evidence of persistence or capability.
Elapsed duration versus meaningful work
Interruptions would include tool failure, delayed approval and changed source evidence. Checkpoints would identify where state or goal fidelity is lost and whether recovery remains possible before the final outcome.
Persistent activity versus controlled progress
Results would separate unaided completion, human-assisted recovery and limit-triggered termination. The conclusion would identify tested sequences and budgets without extrapolating a successful extended task into indefinite operating reliability.
Prerequisites
Start with multi-step task specifications
Give each input a specific job
- Multi-step task specifications
- Defines the operating boundary.
- State, tool and handoff traces
- Examine elapsed duration.
- Resource budgets and recovery requirements
- Shows conditions affecting persistent activity.
- Agent operating owner
- Confirm access and a reviewer who can resolve gaps.
For Extended-task agent evaluation, the team should be able to identify the relevant model, prompt, tool and retained-state conditions. A controlled environment, representative sequences and explicit stop conditions are needed for active tests. Missing lineage or inaccessible provider behaviour would be recorded as a limitation on reconstruction and comparison.
Useful starting material would include multi-step task specifications; state, tool and handoff traces; resource budgets and recovery requirements. These inputs have different jobs: the first establishes context, the second supplies behaviour or comparison material, and the third exposes operating conditions that may qualify a conclusion. Existing material would be reviewed before requesting new collection, and a missing input would be recorded as an evidence gap rather than treated as a favourable result.
How the work proceeds
Preserve the state sequence before interpreting change
Follow the evidence through the work
- Multi-step task specifications
Fix the state, sequence and stopping conditions for the comparison.
- State, tool and handoff traces
Preserve this evidence with the version and operating context.
- Elapsed duration
Challenge this distinction across the declared change sequence.
- A duration and authority decision brief
State which change should trigger another test or operating restriction.
Define state and sequence
Define extended tasks, completion criteria and allowed resources. The opening step turns the operating concern into a question that can be examined. It would identify a complete extended task with checkpoints, interruptions, dependencies and a defined stopping condition, the outcome the owner cares about and the conditions that would make an observation decision-relevant. Scope changes would remain explicit if the evidence reveals a different problem.
Protect the comparison
Instrument state transitions, decisions, tool actions and human handoffs. The comparison material would include state, tool and handoff traces, with resource budgets and recovery requirements available to explain the operating conditions. Differences between step success and task completion would remain visible beside the observation rather than disappear into a pooled result.
Observe change and failure
Exercise interruption, delay, contradictory evidence and recovery conditions. The review question would explicitly distinguish elapsed duration from meaningful work. A material discrepancy would be traced back to multi-step task specifications and the relevant execution or decision record, so a proposed change addresses the observed cause rather than the most convenient explanation.
Set the next operating decision
Review whole-task outcomes and the point where failures or uncertainty accumulated. The resulting duration and authority decision brief would explain the distinction between persistent activity and controlled progress and identify what further observation would change the recommendation. The agent operating owner would retain the decision, including whether a narrower scope or a different approach is preferable.
Illustrative scenario
Extended supplier assessment: examine what changes the decision
Examine the constructed decision boundary
Illustrative scenario
- Extended supplier assessment
The task spans sources, delayed approval and interrupted tool work.
- Sequence challenge
Introduce changed evidence and inspect preservation of the original objective.
- Bounded completion
Judge the accepted end state within the tested duration and conditions.
A fictional research agent prepares a supplier assessment across several document sources and delayed approvals. The proposed evaluation would insert missing evidence, an interrupted tool call and a changed source, then trace whether the agent preserves the task and reaches an acceptable conclusion. Results would be bounded to the tested duration and conditions, without extrapolating to indefinite reliable operation.
The interrupted tool call could leave a partially completed assessment whose ownership is unclear. A checkpoint review would locate that ambiguity and test recovery without restarting all work. The result would inform the proposed task duration and handoff design, with untested longer sequences remaining outside the conclusion. These are hypothetical decision paths, not observed client outcomes.
Indicative artefacts
Use a duration and authority decision brief to make the next action explicit
Every artefact needs an operating use
Illustrative working format
- An extended-task evaluation protocol
- Boundary — multi-step task specifications.
- An end-to-end trajectory and failure analysis
- Findings about elapsed duration.
- A duration and authority decision brief
- Action on persistent activity.
The final artefact set depends on the agreed decision. Useful outputs can include:
- An extended-task evaluation protocol. The working foundation for the question, including relevant definitions, boundaries and assumptions so another reviewer can understand what was examined.
- An end-to-end trajectory and failure analysis. The substantive observations and interpretation, preserving adverse results, disagreement and limitations that affect how the evidence may be used.
- A duration and authority decision brief. The connection to action: what the owner could do next, what evidence is still missing and what change would justify revisiting the conclusion.
Scope and authority
Evidence from a sequence cannot establish indefinite behaviour
Keep analytical work separate from decision authority
- Versioned states and controlled sequences
- Can include an extended-task evaluation protocol.
- Agent operating owner
- Operating owners authorise experiments, deployment changes and stop conditions.
- Indefinite reliability or alignment guarantees
- Cannot be inferred from an end-to-end trajectory and failure analysis.
These methods provide bounded behavioural evidence, not proof of indefinite reliability, alignment or beneficial self-improvement. Research simulations are not operating history. Active tests require authorised environments, competent reviewers and agreed stop conditions; the customer retains deployment and risk decisions. To discuss extended-task agent evaluation, the first practical step is to identify the relevant owner and bring a sanitised description of the decision, the current alternative and the material uncertainty. That gives scoping a concrete starting point without implying a preapproved package or favourable conclusion.
Ways to engage
Related engagement scopes
AI Fractional Agentic AI Office
Create a regular place for agentic AI decisions. Connect initiative priorities, architecture questions, risk ownership and evidence reviews.
AI Fractional Agentic Transformation Office
Coordinate the operating changes behind agent adoption. Align workflow redesign, ownership, dependencies and adoption measures.
AI Harness Engineering Pod
Engineer a bounded AI execution harness with tool adapters, context handling, orchestration, recovery and observable traces.
Continue exploring
Related insights and resources
Engineer the harness, not just the prompt
Define the contracts around an agent's decisions: what state it trusts, which actions can execute, how interrupted work resumes, and what evidence an operator needs to recover safely.
Governing agent decisions
Governance becomes useful when it changes a decision. Connect each agent workflow to an accountable owner, explicit operating conditions, proportionate evidence and a practical route for exceptions and withdrawal.
Do you need an agent centre of excellence?
A centre can concentrate scarce expertise and reusable controls, or become an approval queue detached from real work. Choose the mandate before the structure.