A failure ladder for retrieval-augmented agents

When a retrieval-augmented agent gives a wrong answer, the immediate reaction is often to change the prompt, swap the embedding model or increase the number of chunks. Any of those may alter the symptom without fixing the failure.

The wrong answer may have begun much earlier: the approved corpus never contained the fact; the parser dropped a table; an access filter removed the only current document; retrieval found the right passage but context assembly buried it; the model ignored it; or the agent used a supported answer to take an unsupported action.

A failure ladder makes those stages explicit. Test from the bottom up, and stop blaming the generation model for errors created elsewhere.

The seven rungs from source to effect

Rung	Question	Failure signal
Source authority	Does the approved corpus contain a current, authoritative answer?	Gold evidence is absent, stale or contradicted
Ingestion	Was the source represented faithfully?	Tables, headings, footnotes or effective dates disappear
Eligibility	May this principal use this material for this purpose?	Relevant evidence is wrongly exposed or wrongly filtered
Retrieval	Did the system find the necessary evidence?	Required passages miss the candidate set or rank too low
Context assembly	Did the model receive a usable evidence package?	Evidence is truncated, duplicated, reordered or overwhelmed
Generation	Is the response supported and appropriately qualified?	Claims exceed evidence, citations mismatch or abstention fails
Action	Did the agent apply the answer within authority and verify the result?	A correct answer produces a prohibited or unverified effect

The stages interact, so this is not a claim that every error has one cause. It is a diagnostic order. A failed lower rung constrains what any higher rung can achieve.

Rung one: prove the answer exists

Build the evaluation set from decisions users actually need to make, then identify the authoritative evidence for each. Record document owner, version, effective date, location and the passage or structured field that supports the answer. Add cases with conflicting versions and cases that are deliberately unanswerable from the approved collection.

This produces three important labels:

• answerable and current;
• answerable only with a named qualification or conflict; and
• not answerable from this source boundary.

Without these labels, the team may reward an agent for answering from model pretraining when it should have relied on the corpus. Research on unanswerability in RAG highlights that many evaluations focus on answerable queries and under-test the system’s ability to reject requests that its knowledge base cannot support.

Rung two: inspect what ingestion changed

Retrieval never sees the original document. It sees a representation created by extraction, normalisation, chunking and metadata enrichment. Compare that representation with the source.

For sampled documents, check:

• whether titles and heading hierarchy survive;
• whether tables retain row and column relationships;
• whether footnotes remain attached to the claims they qualify;
• whether dates, versions and superseded status become metadata;
• whether images that carry meaning have an approved text equivalent; and
• whether chunks preserve definitions and exceptions together.

A useful ingestion test asks specific questions against both the source and its indexed representation. If a person can answer from the document but cannot identify the required material in the index, the failure is not retrieval relevance. It is information loss.

Rung three: test authorisation as part of relevance

In enterprise retrieval, the highest-scoring passage may be one the requester must not see. Run each evaluation case as a defined principal with purpose, group membership and data boundary. Assert both positive and negative access outcomes before measuring ranking.

This yields four possibilities: relevant and allowed, relevant and denied, irrelevant and allowed, irrelevant and denied. Only the first belongs in context. Treating access filtering as a late redaction step risks leaking sensitive content to the model or allowing it to influence an answer even if the final text is masked.

Also measure false denials. A perfectly private system that consistently removes the only valid policy is not a useful retrieval system; it should route the unresolved question rather than invent an answer.

Rung four: separate candidate recall from ranking quality

For answerable cases, measure whether all necessary evidence appears anywhere in the retrieved candidate set. Then measure where it ranks and how much irrelevant material arrives with it. Multi-hop questions may require passages from different documents, so “the gold chunk was found” may still be incomplete.

The survey of RAG evaluation organizes evaluation across retrieval and generation rather than treating output quality as one undifferentiated metric. A practical retrieval scorecard includes:

• evidence recall at the candidate cutoff;
• precision or noise in the selected context;
• coverage of every fact needed for multi-part answers;
• ranking stability under paraphrases and spelling variants;
• performance on entities, dates, tables and rare terms; and
• latency and failure behaviour under the intended index size.

Inspect misses by failure class. A chunking change that helps narrative text may make table retrieval worse. A larger candidate set may improve recall while overloading the next rung.

Rung five: vary the assembled context

Even when retrieval succeeds, the final context can fail through truncation, ordering, deduplication or prompt construction. The “Lost in the Middle” experiments showed that models may use relevant information differently depending on its position and that adding more retrieved documents need not produce proportional answer gains.

Test context assembly with controlled perturbations:

1. move the decisive passage among early, middle and late positions;
2. add plausible but irrelevant passages;
3. add an older document that conflicts with the current one;
4. vary the number and length of chunks;
5. repeat a passage with different metadata; and
6. remove one required hop from a multi-document answer.

Retain the exact assembled context with the result. Otherwise, a later investigation cannot distinguish a retrieval miss from a packing decision.

Rung six: grade support, not fluency

At generation, score each material claim against the supplied evidence. Check whether citations point to the passage that actually supports the claim, whether qualifications survived summarization and whether the agent abstained when evidence was insufficient.

The RAGAs paper proposes reference-free measures for aspects such as faithfulness and relevance. Automated graders can make regression testing practical, but they should be calibrated against human-labelled cases and versioned like any other model component. Use deterministic checks for citation identifiers, dates, quantities and structured outputs wherever possible.

Include a “right answer, wrong reason” label. If the response is correct only because the model already knew the subject, it does not establish that the retrieval pipeline works.

Rung seven: follow the answer into the action

A retrieval-augmented agent may use its answer to approve a claim, change a record or send an instruction. Evaluate that transition separately. Was the evidence current at action time? Did the policy permit this principal to act? Was required human confirmation bound to the exact proposal? Did the downstream system confirm the new state?

The NIST Generative AI Profile emphasizes lifecycle risk management and documented evaluation. For an acting RAG system, the lifecycle extends beyond cited prose to the real-world effect and its recovery path.

Turn the ladder into a triage routine

For every failed end-to-end case, locate the lowest rung that cannot support the expected outcome. Fix that stage, rerun its local checks, then rerun the whole path. Keep stage-level metrics and traces together so improvements are not hidden trade-offs.

A release review can then answer precise questions: Which sources are covered? Which document structures remain unreliable? Which principals experience false denials? Which query classes miss evidence? When does context packing fail? How well does the system abstain? Which actions have independent verification?

That evidence is more useful than “RAG accuracy improved.” It tells an owner what changed, what remains uncertain and whether the next investment belongs in content operations, access policy, retrieval engineering, model behaviour or workflow control.