Retrieval quality evaluation
Decide which retrieval failures explain missing or misleading answers
When the agent cannot reliably find evidence needed to answer tasks, this area becomes relevant. For the search and agent quality owner, the concern is specific: the answer model is blamed when the necessary evidence was never retrieved. The deployment evaluation starts from that consequence and follows the relevant evidence through the customer workflow. Define relevance in the customer's task context before comparing retrieval configurations. Include ambiguous queries and plausible distractors, and keep retrieval performance distinct from whether the downstream agent interprets the evidence correctly.
Connect the operating question to inspectable evidence
- Search and agent quality owner
Owns the bounded decision and the consequence of acting.
- Representative queries
Defines the starting context for the proposed work.
- Missing relevant items
Exposes the first material uncertainty to examine.
- Retrieval remediation priorities
Carries unresolved conditions into an accountable next action.
Decision at stake
Decide which retrieval failures explain missing or misleading answers
Make the decision explicit
- Search and agent quality owner
- The agent cannot reliably find evidence needed to answer tasks.
- Decision
- Which retrieval failures explain missing or misleading answers.
- Missing relevant items
- Use representative queries to establish the relevant starting condition.
For the search and agent quality owner, this becomes relevant when the agent cannot reliably find evidence needed to answer tasks. The immediate decision is which retrieval failures explain missing or misleading answers. The scope should be narrow enough that the evaluation team can identify a meaningful alternative, including continuing the current approach if the available evidence already supports it.
The practical failure to challenge is that the answer model is blamed when the necessary evidence was never retrieved. That concern gives the work a concrete direction: establish what the relevant boundary should do, inspect how it behaves or would operate, and identify the owner of any gap. It should not be diluted into a broad transformation exercise before the specific question has been answered.
Coverage
Examine missing relevant items alongside irrelevant ranking
Keep the review distinctions visible
- Missing relevant items
- Compare the returned set with the task's relevant reference items.
- Irrelevant ranking
- Inspect distractors ranked above material evidence.
- Retrieval variation across query forms
- Paraphrase the same request and retain query-specific retrieval differences.
Missing relevant items
For this scope, missing relevant items should be examined alongside representative queries. Create cases with known relevant evidence that should be retrieved, including alternate valid sources. Inspect omissions by query type and configuration. The goal is to identify whether missing information originates in indexing, query formulation, ranking, filtering, or another observable step rather than assume the model failed to reason.
Irrelevant ranking
For this scope, irrelevant ranking should be examined alongside relevance judgements. Include plausible distractors that share vocabulary with the correct source but omit a decisive condition. Review their ranking and effect on downstream answers. This helps distinguish useful semantic matching from topic similarity that does not actually support the business question being asked.
Retrieval variation across query forms
For this scope, retrieval variation across query forms should be examined alongside index and retrieval configuration. Use equivalent requests expressed in different ways and record retrieval variation. Keep the intended information need fixed so comparisons remain meaningful. The evaluation should report where clarification is appropriate and where a reasonable reformulation should still find the same authorised evidence.
Prerequisites
Start with representative queries
Give each input a specific job
- Representative queries
- Defines the operating boundary.
- Relevance judgements
- Examine irrelevant ranking.
- Index and retrieval configuration
- Shows conditions affecting retrieval variation across query forms.
- Product or platform owner
- Confirm access and a reviewer who can resolve gaps.
The starting material is representative queries, relevance judgements, and index and retrieval configuration. Review these together rather than assuming one document tells the whole story. A stated rule may differ from a configured behaviour, and a proposed responsibility may not yet have an available owner. Identify those differences before selecting the detailed work.
Participation should include a product or platform decision owner, a technical lead, a data owner, and a competent reviewer for the relevant boundary. The customer should define which information and environments may be used. Synthetic or sanitised examples can support preparation when they preserve the relevant behaviour; if they omit a material dependency, record the limitation and agree how the missing evidence would be obtained.
How the work proceeds
Tie the deployment finding to observable behaviour
Follow the evidence through the work
- Missing relevant items
Use representative queries to define the decision criteria.
- Irrelevant ranking
Inspect relevance judgements and retain observations that challenge the expected result.
- Retrieval variation across query forms
Use index and retrieval configuration to identify unresolved conditions and a responsible owner.
- Retrieval remediation priorities
Link the recommendation to its evidence, remaining limits and next validation condition.
Build relevance judgements around the information a task needs, including acceptable alternative sources and insufficient evidence cases. Compare query forms, ranking, and retrieval configuration while controlling other changes. The proposed analysis should retain false positives and false negatives rather than report only examples where a relevant document was found.
A practical first slice would focus on missing relevant items and then follow its consequences into irrelevant ranking. Keep retrieval variation across query forms visible as a separate review question rather than assuming it is satisfied by the first result. The evaluation team should record observations, competing explanations, and changes to the original assumptions. A second reviewer can challenge whether the evidence supports the recommendation and whether an apparently successful example omitted the condition that matters most.
Illustrative scenario
Buried policy exception: locate the remedy
Examine the constructed decision boundary
Illustrative scenario
- Buried policy exception
The required exception sits among similar documents that omit it.
- Retrieval and downstream use
Check whether the correct source appears and supports the resulting answer.
- Locate the remedy
Choose a query, ranking, indexing or generation change according to the failed stage.
Consider an illustrative situation in which an assistant must find a specific exception buried among similar policy documents. This is a constructed example, not a client engagement or an observed Softlander outcome.
For the buried policy exception, an illustrative case would include similar documents that omit the exception. The reviewer would examine both whether the correct source appears and whether it is usable downstream. The next decision could be a query, ranking, indexing, or answer-generation change, depending on the observed failure.
The useful learning would be the difference between the expected behaviour and the behaviour or responsibility actually demonstrated. If the exercise leaves retrieval variation across query forms unresolved, that should remain a named next step. A recommendation to narrow the scope, improve a prerequisite, or hold the decision can be appropriate when the example exposes a material dependency.
Indicative artefacts
Use retrieval remediation priorities to make the next action explicit
Every artefact needs an operating use
Illustrative working format
- Retrieval quality case set
- Boundary — representative queries.
- Query and ranking findings
- Findings about irrelevant ranking.
- Retrieval remediation priorities
- Action on retrieval variation across query forms.
Indicative outputs include retrieval quality case set, query and ranking findings, and retrieval remediation priorities. Their form should follow the decision and the customer’s existing operating records. The first should establish the assessed boundary and relevant observations; the second should make material gaps or dependencies inspectable; the third should identify the action, owner, and evidence needed to revisit the decision.
Acceptance should be tested through use: can the responsible owner explain what was examined, identify an unresolved issue, and act on the next step without reconstructing the whole engagement? Findings should preserve counterevidence and limits. A polished document is insufficient if it does not help resolve the customer’s specific question.
Scope and authority
A deployment finding remains bounded to the tested conditions
Keep analytical work separate from decision authority
- Authorised deployment cases and traces
- Can include retrieval quality case set.
- Product or platform owner
- The customer retains release authority and owns remediation priorities.
- Universal vendor or component assurance
- Cannot be inferred from query and ranking findings.
The proposed evaluation concerns the named deployment, configuration, cases, and authorised environment. Findings do not certify a component vendor, establish universal safety, or approve release. Unexamined provider paths and specialist questions remain explicit limitations.
Ways to engage
Related engagement scopes
AI Agent Evaluation Evidence Diagnostic
Find the gaps between your release decision and the evidence behind it. Start with one agent workflow and a bounded set of questions.
AI Agent Evaluation Foundation
Turn scattered tests into a repeatable release discipline. Connect the behaviours that matter to datasets, evaluation criteria and evidence your team can maintain.
AI Agent Component Evaluation
Find where agent behaviour breaks down. Follow memory, retrieval and delegation through your workflow, then isolate the components that need attention.
Continue exploring
Related insights and resources
Engineer the harness, not just the prompt
Define the contracts around an agent's decisions: what state it trusts, which actions can execute, how interrupted work resumes, and what evidence an operator needs to recover safely.
What would justify releasing this agent?
Replace a single pass rate with a reviewable chain of requirements, cases, observations and judgements. Distinguish task quality, control failures and uncertainty before increasing an agent's exposure.
The evidence behind the go-live decision
A working paper for leaders who need to explain why an agent workflow is ready for its next release. Includes a proposed evidence-pack structure, decision table and review agenda.