AI Long-Horizon Agent Evaluation
Examine what changes when an agent keeps working. Evaluate accumulated state, recovery and goal adherence across extended tasks.
Judge progress and recovery across the extended task
- Task objective
- Keep the intended end state explicit throughout the run.
- Accumulated state
- Record memory, checkpoints and completed actions.
- Interruption
- Introduce an agreed pause or dependency failure.
- Recovery decision
- Inspect resumed work against the original objective.
Decision and fit
Judge progress and recovery across the extended task
Compare the engagement with the decision you face
- A question worth resolving
- Judge progress and recovery across the extended task.
- An accountable participant
- Agent engineering teams whose workflows persist through many steps, interruptions or changes in task context.
- Inspectable source material
- The relevant owners can make workflow evidence and agreed inputs available for examination.
- A scope mismatch
- The workflow is a single response with no extended state or recovery.
The engagement
Examine what changes when an agent keeps working. Evaluate accumulated state, recovery and goal adherence across extended tasks. An extended-task evaluation with an agreed run horizon, scenario set, repetition plan and resource limits.
The decision at stake
Short tests do not reveal how an agent behaves after repeated actions, interruptions or context changes.
Suitable situations
- Agent engineering teams whose workflows persist through many steps, interruptions or changes in task context.
Less suitable situations
- The workflow is a single response with no extended state or recovery.
Engagement boundary
Agree the work and its boundaries together
The engagement boundary
- Task objective
- Define extended tasks and the state an agent accumulates along the way.
- Accumulated state
- The inputs, authorised access and client participants are confirmed before scheduling.
- Workflow owner
- Retains organisational decisions and risk acceptance.
- Additional implementation
- Longer horizons, new providers and changes to memory or orchestration require revised scenarios. The engagement does not establish indefinite autonomous reliability or include ongoing runtime operation.
What the work covers
- Define extended tasks and the state an agent accumulates along the way.
- Exercise interruption, recovery, goal drift and context-change scenarios.
- Report behaviour across repeated runs and document limits of coverage.
Scope boundaries
- Formal certification, legal conclusions and guarantees of safe or compliant behaviour.
- Implementation or ongoing operation beyond the agreed scope.
- Longer horizons, new providers and changes to memory or orchestration require revised scenarios. The engagement does not establish indefinite autonomous reliability or include ongoing runtime operation.
Working sequence
How the work moves from question to handover
What each stage should leave your team with
- Specify the extended task
Defined goals, checkpoints and run limits.
- Design interruptions and changes
Interruptions and changing conditions worth exercising.
- Observe repeated trajectories
Trajectories showing state, recovery and accumulated errors.
- Set bounded operating criteria
Operating criteria bounded by the tested horizon.
How the work proceeds
- Specify the extended task. Define the goal, completion conditions, checkpoints and state that must persist across the selected horizon.
- Design interruptions and changes. Select tool failures, resumed sessions, stale context and shifting prerequisites that could alter the task.
- Observe repeated trajectories. Run the agreed trials and inspect accumulated errors, recovery choices, resource use and human escalation.
- Set bounded operating criteria. Document the conditions supported by the observations and provide regression scenarios for unresolved long-running failures.
Decision artefacts
Leave a record the next owner can use
Extended-run record
Illustrative working format
- Subject and version
- Identify the workflow, configuration or proposal examined.
- Task objective
- Record the relevant requirement and supporting material.
- Interruption
- State the observation, supporting evidence and remaining uncertainty.
- Workflow owner
- Record the next decision, follow-up and unresolved question.
What the work can produce
- Long-running evaluation scenarios
- State and recovery findings
- Long-horizon acceptance criteria
Worked situation
Follow one situation through to a bounded next decision
Illustrative working situation
Illustrative scenario
- Starting condition
An interrupted agent repeats an action recorded before the pause.
- Question to examine
Compare checkpoint state with the external action history.
- Possible next action
Define recovery and acceptance criteria for the repeated action.
Working together
Your team and our team need a clear working agreement
Who contributes and who decides
- Client sponsor
- Confirms the decision, priorities and people needed to resolve it.
- Client technical or process owner
- Provides the agreed evidence, access and review of findings or deliverables.
- Softlander delivery lead
- Coordinates the agreed activities, review points, handover and open actions.
- Authorised decision-maker
- Accepts deliverables against the agreed criteria and separately decides operational or business action.
What the work depends on
- Provide checkpoint and state traces, representative long-running tasks and a safe way to interrupt or resume them. Workflow experts define acceptable recovery, goal adherence and escalation; engineers agree runtime and cost constraints.
Engagement availability
The start date, named delivery roles, review points and fee basis are agreed after confirming the inputs and scope. Duration depends on evidence access, system complexity and the time your decision-makers need to review the work.
Engagement and handover
Agree the delivery model and what completion means
A clear agreement from scope to ownership
- Scoped work
- Define extended tasks and the state an agent accumulates along the way.
- Indicative handover
- Long-running evaluation scenarios; State and recovery findings
- Review and acceptance
- Review the agreed deliverables and record any open items, owners and conditions.
- Ongoing work
- Assign maintenance and operational ownership; agree additional scope and coverage explicitly.
An extended-task evaluation with an agreed run horizon, scenario set, repetition plan and resource limits.
The report states the observed time/step horizon, repetitions, state transitions and recovery results. Handover includes scenario definitions and evidence needed to reproduce material failures without implying behaviour beyond the tested horizon.
Longer horizons, new providers and changes to memory or orchestration require revised scenarios. The engagement does not establish indefinite autonomous reliability or include ongoing runtime operation.