Context relevance evaluation
Determine whether included context is relevant to the current task
When answers depend on a large mix of retrieved and supplied context, this area becomes relevant. For the AI application evaluation owner, the concern is specific: the agent receives relevant-looking text that does not support the requested decision. The deployment evaluation starts from that consequence and follows the relevant evidence through the customer workflow. Evaluate context selection separately from final answer fluency. A well-written response can conceal irrelevant inputs, while a poor answer may result from missing evidence rather than weak reasoning over the evidence supplied.
Connect the operating question to inspectable evidence
- AI application evaluation owner
Owns the bounded decision and the consequence of acting.
- Assembled prompts
Defines the starting context for the proposed work.
- Necessary evidence inclusion
Exposes the first material uncertainty to examine.
- Assembly improvement backlog
Carries unresolved conditions into an accountable next action.
Decision at stake
Determine whether included context is relevant to the current task
Make the decision explicit
- AI application evaluation owner
- Answers depend on a large mix of retrieved and supplied context.
- Decision
- Whether included context is relevant to the current task.
- Necessary evidence inclusion
- Use assembled prompts to establish the relevant starting condition.
For the AI application evaluation owner, this becomes relevant when answers depend on a large mix of retrieved and supplied context. The immediate decision is whether included context is relevant to the current task. The scope should be narrow enough that the evaluation team can identify a meaningful alternative, including continuing the current approach if the available evidence already supports it.
The practical failure to challenge is that the agent receives relevant-looking text that does not support the requested decision. That concern gives the work a concrete direction: establish what the relevant boundary should do, inspect how it behaves or would operate, and identify the owner of any gap. It should not be diluted into a broad transformation exercise before the specific question has been answered.
Coverage
Examine necessary evidence inclusion alongside distracting material
Keep the review distinctions visible
- Necessary evidence inclusion
- Check that the context contains evidence required for the decision.
- Distracting material
- Add matched irrelevant material and observe whether task behaviour changes.
- Downstream decision contribution
- Connect a retrieved passage to its effect on the final task decision.
Necessary evidence inclusion
For this scope, necessary evidence inclusion should be examined alongside assembled prompts. Identify facts necessary to complete the task before judging the assembled context. Test whether those facts are included and accessible in the form supplied to the agent. A final-answer error may begin with omission, so retrieval and assembly evidence should remain separate from model interpretation.
Distracting material
For this scope, distracting material should be examined alongside source relevance judgements. Add plausible but irrelevant material while preserving the required facts. Inspect whether it distracts the agent or changes an action without support. The proposed cases should include near-topic documents, since obvious nonsense alone may not reveal how context selection fails in realistic business tasks.
Downstream decision contribution
For this scope, downstream decision contribution should be examined alongside expected task outcomes. Remove or replace one source at a time where practical and observe its effect on the outcome. This creates evidence about contribution rather than appearance. A source that sounds related may add no useful information, while a short exception can be essential to the correct decision.
Prerequisites
Start with assembled prompts
Give each input a specific job
- Assembled prompts
- Defines the operating boundary.
- Source relevance judgements
- Examine distracting material.
- Expected task outcomes
- Shows conditions affecting downstream decision contribution.
- Product or platform owner
- Confirm access and a reviewer who can resolve gaps.
The starting material is assembled prompts, source relevance judgements, and expected task outcomes. Review these together rather than assuming one document tells the whole story. A stated rule may differ from a configured behaviour, and a proposed responsibility may not yet have an available owner. Identify those differences before selecting the detailed work.
Participation should include a product or platform decision owner, a technical lead, a data owner, and a competent reviewer for the relevant boundary. The customer should define which information and environments may be used. Synthetic or sanitised examples can support preparation when they preserve the relevant behaviour; if they omit a material dependency, record the limitation and agree how the missing evidence would be obtained.
How the work proceeds
Tie the deployment finding to observable behaviour
Follow the evidence through the work
- Necessary evidence inclusion
Use assembled prompts to define the decision criteria.
- Distracting material
Inspect source relevance judgements and retain observations that challenge the expected result.
- Downstream decision contribution
Use expected task outcomes to identify unresolved conditions and a responsible owner.
- Assembly improvement backlog
Link the recommendation to its evidence, remaining limits and next validation condition.
Label context items according to whether they are necessary, helpful, irrelevant, contradictory, or outside the task’s permission boundary. Use those labels to construct controlled inclusion and exclusion cases. The proposed evaluation should inspect the assembled input as well as the output so missing evidence is not confused with failure to use available evidence.
A practical first slice would focus on necessary evidence inclusion and then follow its consequences into distracting material. Keep downstream decision contribution visible as a separate review question rather than assuming it is satisfied by the first result. The evaluation team should record observations, competing explanations, and changes to the original assumptions. A second reviewer can challenge whether the evidence supports the recommendation and whether an apparently successful example omitted the condition that matters most.
Illustrative scenario
Wrong source assembled: evidence-chain repair
Examine the constructed decision boundary
Illustrative scenario
- Wrong source assembled
An assistant receives account notes for a question requiring approved product policy.
- Fixed question, varied evidence
Change source inclusion and inspect whether insufficient support causes abstention or improvisation.
- Evidence-chain repair
Locate the gap in retrieval, assembly or answer handling before prescribing a change.
Consider an illustrative situation in which an internal assistant receives account notes when the question requires an approved product policy. This is a constructed example, not a client engagement or an observed Softlander outcome.
For account notes versus product policy, an illustrative run would vary which source is included and preserve the same question. The reviewer would examine whether the agent notices insufficient support or improvises an answer. The resulting recommendation could concern retrieval, assembly, or abstention, depending on where the evidence chain failed.
The useful learning would be the difference between the expected behaviour and the behaviour or responsibility actually demonstrated. If the exercise leaves downstream decision contribution unresolved, that should remain a named next step. A recommendation to narrow the scope, improve a prerequisite, or hold the decision can be appropriate when the example exposes a material dependency.
Indicative artefacts
Use assembly improvement backlog to make the next action explicit
Every artefact needs an operating use
Illustrative working format
- Context relevance case set
- Boundary — assembled prompts.
- Source contribution analysis
- Findings about distracting material.
- Assembly improvement backlog
- Action on downstream decision contribution.
Indicative outputs include context relevance case set, source contribution analysis, and assembly improvement backlog. Their form should follow the decision and the customer’s existing operating records. The first should establish the assessed boundary and relevant observations; the second should make material gaps or dependencies inspectable; the third should identify the action, owner, and evidence needed to revisit the decision.
Acceptance should be tested through use: can the responsible owner explain what was examined, identify an unresolved issue, and act on the next step without reconstructing the whole engagement? Findings should preserve counterevidence and limits. A polished document is insufficient if it does not help resolve the customer’s specific question.
Scope and authority
A deployment finding remains bounded to the tested conditions
Keep analytical work separate from decision authority
- Authorised deployment cases and traces
- Can include context relevance case set.
- Product or platform owner
- The customer retains release authority and owns remediation priorities.
- Universal vendor or component assurance
- Cannot be inferred from source contribution analysis.
The proposed evaluation concerns the named deployment, configuration, cases, and authorised environment. Findings do not certify a component vendor, establish universal safety, or approve release. Unexamined provider paths and specialist questions remain explicit limitations.
Ways to engage
Related engagement scopes
AI Agent Evaluation Evidence Diagnostic
Find the gaps between your release decision and the evidence behind it. Start with one agent workflow and a bounded set of questions.
AI Agent Evaluation Foundation
Turn scattered tests into a repeatable release discipline. Connect the behaviours that matter to datasets, evaluation criteria and evidence your team can maintain.
AI Agent Component Evaluation
Find where agent behaviour breaks down. Follow memory, retrieval and delegation through your workflow, then isolate the components that need attention.
Continue exploring
Related insights and resources
Engineer the harness, not just the prompt
Define the contracts around an agent's decisions: what state it trusts, which actions can execute, how interrupted work resumes, and what evidence an operator needs to recover safely.
What would justify releasing this agent?
Replace a single pass rate with a reviewable chain of requirements, cases, observations and judgements. Distinguish task quality, control failures and uncertainty before increasing an agent's exposure.
The evidence behind the go-live decision
A working paper for leaders who need to explain why an agent workflow is ready for its next release. Includes a proposed evidence-pack structure, decision table and review agenda.