Workflow-routing evaluation
Determine whether routing selects an appropriate path for the task and authority
When requests are sent to different workflows or agents, this area becomes relevant. For the orchestration product owner, the concern is specific: the agent selects a capable workflow that lacks authority for the requested action. The deployment evaluation starts from that consequence and follows the relevant evidence through the customer workflow. Routing correctness includes the decision to seek clarification. A route should be judged against task needs and permission context, not simply whether the selected branch eventually produces a plausible response.
Connect the operating question to inspectable evidence
- Orchestration product owner
Owns the bounded decision and the consequence of acting.
- Routing rules
Defines the starting context for the proposed work.
- Ambiguous intent
Exposes the first material uncertainty to examine.
- Clarification and fallback cases
Carries unresolved conditions into an accountable next action.
Decision at stake
Determine whether routing selects an appropriate path for the task and authority
Make the decision explicit
- Orchestration product owner
- Requests are sent to different workflows or agents.
- Decision
- Whether routing selects an appropriate path for the task and authority.
- Ambiguous intent
- Use routing rules to establish the relevant starting condition.
For the orchestration product owner, this becomes relevant when requests are sent to different workflows or agents. The immediate decision is whether routing selects an appropriate path for the task and authority. The scope should be narrow enough that the evaluation team can identify a meaningful alternative, including continuing the current approach if the available evidence already supports it.
The practical failure to challenge is that the agent selects a capable workflow that lacks authority for the requested action. That concern gives the work a concrete direction: establish what the relevant boundary should do, inspect how it behaves or would operate, and identify the owner of any gap. It should not be diluted into a broad transformation exercise before the specific question has been answered.
Coverage
Examine ambiguous intent alongside boundary categories
Keep the review distinctions visible
- Ambiguous intent
- Give the router a request that reasonably fits more than one branch.
- Boundary categories
- Test examples immediately around the declared routing threshold.
- Route changes after clarification
- Add clarification and verify that the route can change coherently.
Ambiguous intent
For this scope, ambiguous intent should be examined alongside routing rules. Use requests whose intent is clear alongside requests that legitimately require clarification. The expected route should account for what the user is trying to achieve, not just keywords. The proposed evaluation should preserve the ambiguity that made a routing decision premature or unsupported.
Boundary categories
For this scope, boundary categories should be examined alongside representative request categories. Construct neighboring categories that look similar but lead to different permissions or business effects. Inspect the chosen branch and resulting action. A route that can technically complete the task may still be inappropriate if it crosses an authority boundary or bypasses a required decision.
Route changes after clarification
For this scope, route changes after clarification should be examined alongside expected branch and fallback labels. Change the information available after clarification and inspect whether routing updates correctly. A preliminary route should not become permanent when new facts alter the task. The result should identify whether the failure belongs to intent interpretation, branch selection, state handling, or the downstream workflow contract.
Prerequisites
Start with routing rules
Give each input a specific job
- Routing rules
- Defines the operating boundary.
- Representative request categories
- Examine boundary categories.
- Expected branch and fallback labels
- Shows conditions affecting route changes after clarification.
- Product or platform owner
- Confirm access and a reviewer who can resolve gaps.
The starting material is routing rules, representative request categories, and expected branch and fallback labels. Review these together rather than assuming one document tells the whole story. A stated rule may differ from a configured behaviour, and a proposed responsibility may not yet have an available owner. Identify those differences before selecting the detailed work.
Participation should include a product or platform decision owner, a technical lead, a data owner, and a competent reviewer for the relevant boundary. The customer should define which information and environments may be used. Synthetic or sanitised examples can support preparation when they preserve the relevant behaviour; if they omit a material dependency, record the limitation and agree how the missing evidence would be obtained.
How the work proceeds
Tie the deployment finding to observable behaviour
Follow the evidence through the work
- Ambiguous intent
Use routing rules to define the decision criteria.
- Boundary categories
Inspect representative request categories and retain observations that challenge the expected result.
- Route changes after clarification
Use expected branch and fallback labels to identify unresolved conditions and a responsible owner.
- Clarification and fallback cases
Link the recommendation to its evidence, remaining limits and next validation condition.
Include requests that sit near routing boundaries and requests that need clarification before a route is chosen. Record the intended task, permission context, selected path, and eventual effect. The proposed rubric should distinguish an inefficient route from one that creates an unauthorised or materially incorrect outcome.
A practical first slice would focus on ambiguous intent and then follow its consequences into boundary categories. Keep route changes after clarification visible as a separate review question rather than assuming it is satisfied by the first result. The evaluation team should record observations, competing explanations, and changes to the original assumptions. A second reviewer can challenge whether the evidence supports the recommendation and whether an apparently successful example omitted the condition that matters most.
Illustrative scenario
Information or account change: route-specific repair
Examine the constructed decision boundary
Illustrative scenario
- Information or account change
A service request omits the detail needed to distinguish those intentions.
- Clarification case
Withhold that detail and inspect whether the agent asks before routing.
- Route-specific repair
Identify the missing distinction and repair classification, policy or workflow accordingly.
Consider an illustrative situation in which a service request could be an information question or an account-change request depending on missing details. This is a constructed example, not a client engagement or an observed Softlander outcome.
For the ambiguous service request, an illustrative case would withhold whether the user wants information or an account change. The appropriate behaviour may be a question rather than immediate routing. A finding should explain which missing distinction led to the misroute and whether the remedy belongs in classification, policy, or workflow design.
The useful learning would be the difference between the expected behaviour and the behaviour or responsibility actually demonstrated. If the exercise leaves route changes after clarification unresolved, that should remain a named next step. A recommendation to narrow the scope, improve a prerequisite, or hold the decision can be appropriate when the example exposes a material dependency.
Indicative artefacts
Use clarification and fallback cases to make the next action explicit
Every artefact needs an operating use
Illustrative working format
- Routing evaluation matrix
- Boundary — routing rules.
- Misroute findings
- Findings about boundary categories.
- Clarification and fallback cases
- Action on route changes after clarification.
Indicative outputs include routing evaluation matrix, misroute findings, and clarification and fallback cases. Their form should follow the decision and the customer’s existing operating records. The first should establish the assessed boundary and relevant observations; the second should make material gaps or dependencies inspectable; the third should identify the action, owner, and evidence needed to revisit the decision.
Acceptance should be tested through use: can the responsible owner explain what was examined, identify an unresolved issue, and act on the next step without reconstructing the whole engagement? Findings should preserve counterevidence and limits. A polished document is insufficient if it does not help resolve the customer’s specific question.
Scope and authority
A deployment finding remains bounded to the tested conditions
Keep analytical work separate from decision authority
- Authorised deployment cases and traces
- Can include routing evaluation matrix.
- Product or platform owner
- The customer retains release authority and owns remediation priorities.
- Universal vendor or component assurance
- Cannot be inferred from misroute findings.
The proposed evaluation concerns the named deployment, configuration, cases, and authorised environment. Findings do not certify a component vendor, establish universal safety, or approve release. Unexamined provider paths and specialist questions remain explicit limitations.
Ways to engage
Related engagement scopes
AI Agent Evaluation Evidence Diagnostic
Find the gaps between your release decision and the evidence behind it. Start with one agent workflow and a bounded set of questions.
AI Agent Evaluation Foundation
Turn scattered tests into a repeatable release discipline. Connect the behaviours that matter to datasets, evaluation criteria and evidence your team can maintain.
AI Agent Component Evaluation
Find where agent behaviour breaks down. Follow memory, retrieval and delegation through your workflow, then isolate the components that need attention.
Continue exploring
Related insights and resources
Engineer the harness, not just the prompt
Define the contracts around an agent's decisions: what state it trusts, which actions can execute, how interrupted work resumes, and what evidence an operator needs to recover safely.
What would justify releasing this agent?
Replace a single pass rate with a reviewable chain of requirements, cases, observations and judgements. Distinguish task quality, control failures and uncertainty before increasing an agent's exposure.
The evidence behind the go-live decision
A working paper for leaders who need to explain why an agent workflow is ready for its next release. Includes a proposed evidence-pack structure, decision table and review agenda.