Citation and grounding evaluation
Determine whether claims are supported by the cited evidence
When users need to verify the agent's factual answers, this area becomes relevant. For the product quality and knowledge owner, the concern is specific: a real citation is attached to a claim the source does not support. The deployment evaluation starts from that consequence and follows the relevant evidence through the customer workflow. Assess individual claims rather than treating the presence of links as grounding. The proposed method should distinguish absent citations, incorrect references, unsupported interpretations, and cases where the available evidence is insufficient for a confident answer.
Connect the operating question to inspectable evidence
- Product quality and knowledge owner
Owns the bounded decision and the consequence of acting.
- Agent answers
Defines the starting context for the proposed work.
- Citation existence
Exposes the first material uncertainty to examine.
- Citation failure taxonomy
Carries unresolved conditions into an accountable next action.
Decision at stake
Determine whether claims are supported by the cited evidence
Make the decision explicit
- Product quality and knowledge owner
- Users need to verify the agent's factual answers.
- Decision
- Whether claims are supported by the cited evidence.
- Citation existence
- Use agent answers to establish the relevant starting condition.
For the product quality and knowledge owner, this becomes relevant when users need to verify the agent’s factual answers. The immediate decision is whether claims are supported by the cited evidence. The scope should be narrow enough that the evaluation team can identify a meaningful alternative, including continuing the current approach if the available evidence already supports it.
The practical failure to challenge is that a real citation is attached to a claim the source does not support. That concern gives the work a concrete direction: establish what the relevant boundary should do, inspect how it behaves or would operate, and identify the owner of any gap. It should not be diluted into a broad transformation exercise before the specific question has been answered.
Coverage
Examine citation existence alongside claim entailment
Keep the review distinctions visible
- Citation existence
- Verify that a displayed citation resolves to an inspectable source.
- Claim entailment
- Check whether the cited passage supports the particular assertion.
- Missing or contradictory evidence
- Preserve contradictory passages rather than counting a citation as support.
Citation existence
For this scope, citation existence should be examined alongside agent answers. Verify that a citation resolves to the intended source and relevant passage under the reviewer’s permitted access. Broken or mismatched references are distinct from unsupported reasoning. The proposed evaluation should retain source version and context so later source changes do not silently alter the meaning of the finding.
Claim entailment
For this scope, claim entailment should be examined alongside cited source passages. Break the answer into claims and ask whether the cited material supports each one. A source about the same topic may not justify a specific exception or conclusion. Include qualifications and combined claims so a generally correct answer cannot conceal an unsupported material statement.
Missing or contradictory evidence
For this scope, missing or contradictory evidence should be examined alongside claim-level expected support. Provide cases with missing or contradictory evidence and define the expected qualification or escalation. The agent should not manufacture certainty to complete the answer. The result should identify when retrieval is insufficient, when interpretation overreaches, and when the task itself needs an authorised source or human decision.
Prerequisites
Start with agent answers
Give each input a specific job
- Agent answers
- Defines the operating boundary.
- Cited source passages
- Examine claim entailment.
- Claim-level expected support
- Shows conditions affecting missing or contradictory evidence.
- Product or platform owner
- Confirm access and a reviewer who can resolve gaps.
The starting material is agent answers, cited source passages, and claim-level expected support. Review these together rather than assuming one document tells the whole story. A stated rule may differ from a configured behaviour, and a proposed responsibility may not yet have an available owner. Identify those differences before selecting the detailed work.
Participation should include a product or platform decision owner, a technical lead, a data owner, and a competent reviewer for the relevant boundary. The customer should define which information and environments may be used. Synthetic or sanitised examples can support preparation when they preserve the relevant behaviour; if they omit a material dependency, record the limitation and agree how the missing evidence would be obtained.
How the work proceeds
Tie the deployment finding to observable behaviour
Follow the evidence through the work
- Citation existence
Use agent answers to define the decision criteria.
- Claim entailment
Inspect cited source passages and retain observations that challenge the expected result.
- Missing or contradictory evidence
Use claim-level expected support to identify unresolved conditions and a responsible owner.
- Citation failure taxonomy
Link the recommendation to its evidence, remaining limits and next validation condition.
Break answers into material claims and compare each with the cited passage. Include unsupported inferences, contradictory sources, and claims requiring more than one document. The proposed rubric should distinguish a citation formatting defect from a substantive support failure and should allow an insufficient-evidence outcome.
A practical first slice would focus on citation existence and then follow its consequences into claim entailment. Keep missing or contradictory evidence visible as a separate review question rather than assuming it is satisfied by the first result. The evaluation team should record observations, competing explanations, and changes to the original assumptions. A second reviewer can challenge whether the evidence supports the recommendation and whether an apparently successful example omitted the condition that matters most.
Illustrative scenario
Citation misses the exception: qualified answer
Examine the constructed decision boundary
Illustrative scenario
- Citation misses the exception
An assistant cites a policy while borrowing an exception from an unrelated note.
- Claim-level support
Inspect whether the policy contains that exception or only the general topic.
- Qualified answer
Require supported wording or another authorised source instead of counting the link as grounding.
Consider an illustrative situation in which an assistant cites a policy document while making an exception that appears only in an unrelated note. This is a constructed example, not a client engagement or an observed Softlander outcome.
For the policy exception, an illustrative review would ask whether the cited policy actually contains the exception or merely discusses the general topic. The agent might need to qualify the claim or cite another authorised source. Findings should explain the unsupported reasoning rather than count every link as successful grounding.
The useful learning would be the difference between the expected behaviour and the behaviour or responsibility actually demonstrated. If the exercise leaves missing or contradictory evidence unresolved, that should remain a named next step. A recommendation to narrow the scope, improve a prerequisite, or hold the decision can be appropriate when the example exposes a material dependency.
Indicative artefacts
Use citation failure taxonomy to make the next action explicit
Every artefact needs an operating use
Illustrative working format
- Claim-to-source review
- Boundary — agent answers.
- Grounding evaluation cases
- Findings about claim entailment.
- Citation failure taxonomy
- Action on missing or contradictory evidence.
Indicative outputs include claim-to-source review, grounding evaluation cases, and citation failure taxonomy. Their form should follow the decision and the customer’s existing operating records. The first should establish the assessed boundary and relevant observations; the second should make material gaps or dependencies inspectable; the third should identify the action, owner, and evidence needed to revisit the decision.
Acceptance should be tested through use: can the responsible owner explain what was examined, identify an unresolved issue, and act on the next step without reconstructing the whole engagement? Findings should preserve counterevidence and limits. A polished document is insufficient if it does not help resolve the customer’s specific question.
Scope and authority
A deployment finding remains bounded to the tested conditions
Keep analytical work separate from decision authority
- Authorised deployment cases and traces
- Can include claim-to-source review.
- Product or platform owner
- The customer retains release authority and owns remediation priorities.
- Universal vendor or component assurance
- Cannot be inferred from grounding evaluation cases.
The proposed evaluation concerns the named deployment, configuration, cases, and authorised environment. Findings do not certify a component vendor, establish universal safety, or approve release. Unexamined provider paths and specialist questions remain explicit limitations.
Ways to engage
Related engagement scopes
AI Agent Evaluation Evidence Diagnostic
Find the gaps between your release decision and the evidence behind it. Start with one agent workflow and a bounded set of questions.
AI Agent Evaluation Foundation
Turn scattered tests into a repeatable release discipline. Connect the behaviours that matter to datasets, evaluation criteria and evidence your team can maintain.
AI Agent Component Evaluation
Find where agent behaviour breaks down. Follow memory, retrieval and delegation through your workflow, then isolate the components that need attention.
Continue exploring
Related insights and resources
Engineer the harness, not just the prompt
Define the contracts around an agent's decisions: what state it trusts, which actions can execute, how interrupted work resumes, and what evidence an operator needs to recover safely.
What would justify releasing this agent?
Replace a single pass rate with a reviewable chain of requirements, cases, observations and judgements. Distinguish task quality, control failures and uncertainty before increasing an agent's exposure.
The evidence behind the go-live decision
A working paper for leaders who need to explain why an agent workflow is ready for its next release. Includes a proposed evidence-pack structure, decision table and review agenda.