Lost-in-the-middle evaluation
Determine whether evidence position changes task performance in this deployment
When important evidence may be overlooked inside lengthy inputs, this area becomes relevant. For the long-context evaluation owner, the concern is specific: the agent answers correctly when a fact is near the start but misses it in another position. The deployment evaluation starts from that consequence and follows the relevant evidence through the customer workflow. Use matched inputs that vary evidence position while preserving meaning. The evaluation should record model and assembly configuration and should avoid turning a result on one task or context shape into a general claim about all long-context behaviour.
Connect the operating question to inspectable evidence
- Long-context evaluation owner
Owns the bounded decision and the consequence of acting.
- Controlled context variants
Defines the starting context for the proposed work.
- Position sensitivity
Exposes the first material uncertainty to examine.
- Context assembly recommendations
Carries unresolved conditions into an accountable next action.
Decision at stake
Determine whether evidence position changes task performance in this deployment
Make the decision explicit
- Long-context evaluation owner
- Important evidence may be overlooked inside lengthy inputs.
- Decision
- Whether evidence position changes task performance in this deployment.
- Position sensitivity
- Use controlled context variants to establish the relevant starting condition.
For the long-context evaluation owner, this becomes relevant when important evidence may be overlooked inside lengthy inputs. The immediate decision is whether evidence position changes task performance in this deployment. The scope should be narrow enough that the evaluation team can identify a meaningful alternative, including continuing the current approach if the available evidence already supports it.
The practical failure to challenge is that the agent answers correctly when a fact is near the start but misses it in another position. That concern gives the work a concrete direction: establish what the relevant boundary should do, inspect how it behaves or would operate, and identify the owner of any gap. It should not be diluted into a broad transformation exercise before the specific question has been answered.
Coverage
Examine position sensitivity alongside ordering confounds
Keep the review distinctions visible
- Position sensitivity
- Move the same critical fact through matched context positions.
- Ordering confounds
- Hold distractors and task wording constant during the position change.
- Retained evidence after truncation
- Separate position effects from evidence removed by truncation.
Position sensitivity
For this scope, position sensitivity should be examined alongside controlled context variants. Move the task-critical evidence among controlled positions while keeping wording and overall meaning fixed. Record the supplied context for each variant. The proposed comparison should include repeat runs appropriate to observed variability so a single changed answer does not become an unsupported general effect.
Ordering confounds
For this scope, ordering confounds should be examined alongside task-critical evidence. Check whether ordering changes also alter retrieval, assembly, or truncation. These are potential confounds rather than evidence of position sensitivity itself. The evaluation should identify which inputs actually reached the agent and preserve any differences that prevent a clean comparison.
Retained evidence after truncation
For this scope, retained evidence after truncation should be examined alongside matched distractor material. Inspect whether the decisive evidence survives every variant before interpreting the result. If the assembly system drops it, the finding concerns the input path. If it remains present but is not used, the owner has a different question about attention, interpretation, or task design to investigate.
Prerequisites
Start with controlled context variants
Give each input a specific job
- Controlled context variants
- Defines the operating boundary.
- Task-critical evidence
- Examine ordering confounds.
- Matched distractor material
- Shows conditions affecting retained evidence after truncation.
- Product or platform owner
- Confirm access and a reviewer who can resolve gaps.
The starting material is controlled context variants, task-critical evidence, and matched distractor material. Review these together rather than assuming one document tells the whole story. A stated rule may differ from a configured behaviour, and a proposed responsibility may not yet have an available owner. Identify those differences before selecting the detailed work.
Participation should include a product or platform decision owner, a technical lead, a data owner, and a competent reviewer for the relevant boundary. The customer should define which information and environments may be used. Synthetic or sanitised examples can support preparation when they preserve the relevant behaviour; if they omit a material dependency, record the limitation and agree how the missing evidence would be obtained.
How the work proceeds
Tie the deployment finding to observable behaviour
Follow the evidence through the work
- Position sensitivity
Use controlled context variants to define the decision criteria.
- Ordering confounds
Inspect task-critical evidence and retain observations that challenge the expected result.
- Retained evidence after truncation
Use matched distractor material to identify unresolved conditions and a responsible owner.
- Context assembly recommendations
Link the recommendation to its evidence, remaining limits and next validation condition.
Use position-controlled variants that keep wording, evidence, and task intent constant. Record context assembly and truncation so an apparent position effect is not actually caused by omitted text. The proposed analysis should compare task outcomes and supporting rationale across repeated runs appropriate to the system’s variability.
A practical first slice would focus on position sensitivity and then follow its consequences into ordering confounds. Keep retained evidence after truncation visible as a separate review question rather than assuming it is satisfied by the first result. The evaluation team should record observations, competing explanations, and changes to the original assumptions. A second reviewer can challenge whether the evidence supports the recommendation and whether an apparently successful example omitted the condition that matters most.
Illustrative scenario
Exception changes position: assembly retest
Examine the constructed decision boundary
Illustrative scenario
- Exception changes position
A decisive case fact moves within otherwise equivalent long inputs.
- Matched context variants
Place it in different positions while preserving the rest of the task.
- Assembly retest
Investigate sensitivity and retest other evidence shapes before accepting a context change.
Consider an illustrative situation in which a long case file contains a decisive exception that is moved among otherwise equivalent context variants. This is a constructed example, not a client engagement or an observed Softlander outcome.
For the decisive case exception, an illustrative evaluation would place it near different parts of the same long input. The reviewer would identify patterns without assuming a universal model limitation. Any assembly change proposed in response should be retested on other evidence shapes to check that it does not merely move the problem.
The useful learning would be the difference between the expected behaviour and the behaviour or responsibility actually demonstrated. If the exercise leaves retained evidence after truncation unresolved, that should remain a named next step. A recommendation to narrow the scope, improve a prerequisite, or hold the decision can be appropriate when the example exposes a material dependency.
Indicative artefacts
Use context assembly recommendations to make the next action explicit
Every artefact needs an operating use
Illustrative working format
- Position-controlled case set
- Boundary — controlled context variants.
- Sensitivity findings
- Findings about ordering confounds.
- Context assembly recommendations
- Action on retained evidence after truncation.
Indicative outputs include position-controlled case set, sensitivity findings, and context assembly recommendations. Their form should follow the decision and the customer’s existing operating records. The first should establish the assessed boundary and relevant observations; the second should make material gaps or dependencies inspectable; the third should identify the action, owner, and evidence needed to revisit the decision.
Acceptance should be tested through use: can the responsible owner explain what was examined, identify an unresolved issue, and act on the next step without reconstructing the whole engagement? Findings should preserve counterevidence and limits. A polished document is insufficient if it does not help resolve the customer’s specific question.
Scope and authority
A deployment finding remains bounded to the tested conditions
Keep analytical work separate from decision authority
- Authorised deployment cases and traces
- Can include position-controlled case set.
- Product or platform owner
- The customer retains release authority and owns remediation priorities.
- Universal vendor or component assurance
- Cannot be inferred from sensitivity findings.
The proposed evaluation concerns the named deployment, configuration, cases, and authorised environment. Findings do not certify a component vendor, establish universal safety, or approve release. Unexamined provider paths and specialist questions remain explicit limitations.
Ways to engage
Related engagement scopes
AI Agent Evaluation Evidence Diagnostic
Find the gaps between your release decision and the evidence behind it. Start with one agent workflow and a bounded set of questions.
AI Agent Evaluation Foundation
Turn scattered tests into a repeatable release discipline. Connect the behaviours that matter to datasets, evaluation criteria and evidence your team can maintain.
AI Agent Component Evaluation
Find where agent behaviour breaks down. Follow memory, retrieval and delegation through your workflow, then isolate the components that need attention.
Continue exploring
Related insights and resources
Engineer the harness, not just the prompt
Define the contracts around an agent's decisions: what state it trusts, which actions can execute, how interrupted work resumes, and what evidence an operator needs to recover safely.
What would justify releasing this agent?
Replace a single pass rate with a reviewable chain of requirements, cases, observations and judgements. Distinguish task quality, control failures and uncertainty before increasing an agent's exposure.
The evidence behind the go-live decision
A working paper for leaders who need to explain why an agent workflow is ready for its next release. Includes a proposed evidence-pack structure, decision table and review agenda.