Context-conflict resolution testing
Determine whether conflicts are resolved using explicit authority and context rules
When the agent receives sources that disagree, this area becomes relevant. For the AI product and knowledge governance owner, the concern is specific: the agent silently chooses the most convenient source without exposing the disagreement. The deployment evaluation starts from that consequence and follows the relevant evidence through the customer workflow. A conflict can be legitimate rather than an error in retrieval. Tests should examine when one source governs, when different facts apply to different contexts, and when the agent should stop and ask for qualified judgement.
Connect the operating question to inspectable evidence
- AI product and knowledge governance owner
Owns the bounded decision and the consequence of acting.
- Conflicting source pairs
Defines the starting context for the proposed work.
- Authority precedence
Exposes the first material uncertainty to examine.
- Source policy recommendations
Carries unresolved conditions into an accountable next action.
Decision at stake
Determine whether conflicts are resolved using explicit authority and context rules
Make the decision explicit
- AI product and knowledge governance owner
- The agent receives sources that disagree.
- Decision
- Whether conflicts are resolved using explicit authority and context rules.
- Authority precedence
- Use conflicting source pairs to establish the relevant starting condition.
For the AI product and knowledge governance owner, this becomes relevant when the agent receives sources that disagree. The immediate decision is whether conflicts are resolved using explicit authority and context rules. The scope should be narrow enough that the evaluation team can identify a meaningful alternative, including continuing the current approach if the available evidence already supports it.
The practical failure to challenge is that the agent silently chooses the most convenient source without exposing the disagreement. That concern gives the work a concrete direction: establish what the relevant boundary should do, inspect how it behaves or would operate, and identify the owner of any gap. It should not be diluted into a broad transformation exercise before the specific question has been answered.
Coverage
Examine authority precedence alongside temporal distinctions
Keep the review distinctions visible
- Authority precedence
- Present conflicting sources with declared authority levels.
- Temporal distinctions
- Separate a later effective fact from a merely later document timestamp.
- Uncertainty and escalation
- Check whether unresolved conflicts trigger clarification or escalation.
Authority precedence
For this scope, authority precedence should be examined alongside conflicting source pairs. Use source pairs with an agreed authority relationship and inspect whether the agent applies it. Record who supplied the precedence rule. The test should avoid asking the model to invent a hierarchy between a general policy, a customer-specific agreement, and an informal note.
Temporal distinctions
For this scope, temporal distinctions should be examined alongside source precedence rules. Separate apparent contradictions caused by time or scope from genuine unresolved conflicts. Two statements may both be correct for different contexts. The proposed cases should ask the agent to identify the relevant distinction and preserve it in the answer or action rather than flatten both sources into one rule.
Uncertainty and escalation
For this scope, uncertainty and escalation should be examined alongside expected escalation cases. Include a conflict without an authorised resolution and define the escalation path. Inspect whether the agent exposes the disagreement and seeks the required judgement. A silent choice may look efficient but leave the business owner unaware that the action rests on unresolved authority or evidence.
Prerequisites
Start with conflicting source pairs
Give each input a specific job
- Conflicting source pairs
- Defines the operating boundary.
- Source precedence rules
- Examine temporal distinctions.
- Expected escalation cases
- Shows conditions affecting uncertainty and escalation.
- Product or platform owner
- Confirm access and a reviewer who can resolve gaps.
The starting material is conflicting source pairs, source precedence rules, and expected escalation cases. Review these together rather than assuming one document tells the whole story. A stated rule may differ from a configured behaviour, and a proposed responsibility may not yet have an available owner. Identify those differences before selecting the detailed work.
Participation should include a product or platform decision owner, a technical lead, a data owner, and a competent reviewer for the relevant boundary. The customer should define which information and environments may be used. Synthetic or sanitised examples can support preparation when they preserve the relevant behaviour; if they omit a material dependency, record the limitation and agree how the missing evidence would be obtained.
How the work proceeds
Tie the deployment finding to observable behaviour
Follow the evidence through the work
- Authority precedence
Use conflicting source pairs to define the decision criteria.
- Temporal distinctions
Inspect source precedence rules and retain observations that challenge the expected result.
- Uncertainty and escalation
Use expected escalation cases to identify unresolved conditions and a responsible owner.
- Source policy recommendations
Link the recommendation to its evidence, remaining limits and next validation condition.
Create conflict pairs that differ in source authority, effective time, customer scope, and uncertainty. Predefine which cases permit resolution and which require escalation. The proposed test should inspect the agent’s explanation of the conflict as well as its selected answer, since a silent choice can hide material ambiguity.
A practical first slice would focus on authority precedence and then follow its consequences into temporal distinctions. Keep uncertainty and escalation visible as a separate review question rather than assuming it is satisfied by the first result. The evaluation team should record observations, competing explanations, and changes to the original assumptions. A second reviewer can challenge whether the evidence supports the recommendation and whether an apparently successful example omitted the condition that matters most.
Illustrative scenario
Policy conflicts with agreement: escalation boundary
Examine the constructed decision boundary
Illustrative scenario
- Policy conflicts with agreement
Two sources give different directions for the same customer request.
- Known and unknown precedence
Test one owner-authorized rule and another case with precedence deliberately unresolved.
- Escalation boundary
Follow the established rule; preserve uncertainty rather than invent authority for the unresolved case.
Consider an illustrative situation in which a policy document and a customer-specific agreement give different instructions for the same request. This is a constructed example, not a client engagement or an observed Softlander outcome.
For policy versus customer agreement, an illustrative case would establish the applicable precedence rule through authorised owners. Another case would deliberately leave precedence unresolved. The agent should distinguish them. Findings can then target source metadata, policy expression, retrieval, or escalation rather than encourage the model to invent an authority hierarchy.
The useful learning would be the difference between the expected behaviour and the behaviour or responsibility actually demonstrated. If the exercise leaves uncertainty and escalation unresolved, that should remain a named next step. A recommendation to narrow the scope, improve a prerequisite, or hold the decision can be appropriate when the example exposes a material dependency.
Indicative artefacts
Use source policy recommendations to make the next action explicit
Every artefact needs an operating use
Illustrative working format
- Conflict resolution test set
- Boundary — conflicting source pairs.
- Precedence and escalation findings
- Findings about temporal distinctions.
- Source policy recommendations
- Action on uncertainty and escalation.
Indicative outputs include conflict resolution test set, precedence and escalation findings, and source policy recommendations. Their form should follow the decision and the customer’s existing operating records. The first should establish the assessed boundary and relevant observations; the second should make material gaps or dependencies inspectable; the third should identify the action, owner, and evidence needed to revisit the decision.
Acceptance should be tested through use: can the responsible owner explain what was examined, identify an unresolved issue, and act on the next step without reconstructing the whole engagement? Findings should preserve counterevidence and limits. A polished document is insufficient if it does not help resolve the customer’s specific question.
Scope and authority
A deployment finding remains bounded to the tested conditions
Keep analytical work separate from decision authority
- Authorised deployment cases and traces
- Can include conflict resolution test set.
- Product or platform owner
- The customer retains release authority and owns remediation priorities.
- Universal vendor or component assurance
- Cannot be inferred from precedence and escalation findings.
The proposed evaluation concerns the named deployment, configuration, cases, and authorised environment. Findings do not certify a component vendor, establish universal safety, or approve release. Unexamined provider paths and specialist questions remain explicit limitations.
Ways to engage
Related engagement scopes
AI Agent Evaluation Evidence Diagnostic
Find the gaps between your release decision and the evidence behind it. Start with one agent workflow and a bounded set of questions.
AI Agent Evaluation Foundation
Turn scattered tests into a repeatable release discipline. Connect the behaviours that matter to datasets, evaluation criteria and evidence your team can maintain.
AI Agent Component Evaluation
Find where agent behaviour breaks down. Follow memory, retrieval and delegation through your workflow, then isolate the components that need attention.
Continue exploring
Related insights and resources
Engineer the harness, not just the prompt
Define the contracts around an agent's decisions: what state it trusts, which actions can execute, how interrupted work resumes, and what evidence an operator needs to recover safely.
What would justify releasing this agent?
Replace a single pass rate with a reviewable chain of requirements, cases, observations and judgements. Distinguish task quality, control failures and uncertainty before increasing an agent's exposure.
The evidence behind the go-live decision
A working paper for leaders who need to explain why an agent workflow is ready for its next release. Includes a proposed evidence-pack structure, decision table and review agenda.