Planner quality assessment
Determine whether plans are feasible, sufficiently complete, and bounded
When a planner produces steps that look plausible but fail in execution, this area becomes relevant. For the agent planning and evaluation lead, the concern is specific: a plan assumes a permission or tool result that has not been established. The deployment evaluation starts from that consequence and follows the relevant evidence through the customer workflow. Assess the plan against executable constraints rather than style. A concise plan can be sufficient, and a detailed plan can still be invalid if it relies on unavailable tools, missing authority, or unsupported assumptions.
Connect the operating question to inspectable evidence
- Agent planning and evaluation lead
Owns the bounded decision and the consequence of acting.
- Task goals
Defines the starting context for the proposed work.
- Missing prerequisites
Exposes the first material uncertainty to examine.
- Replanning test cases
Carries unresolved conditions into an accountable next action.
Decision at stake
Determine whether plans are feasible, sufficiently complete, and bounded
Make the decision explicit
- Agent planning and evaluation lead
- A planner produces steps that look plausible but fail in execution.
- Decision
- Whether plans are feasible, sufficiently complete, and bounded.
- Missing prerequisites
- Use task goals to establish the relevant starting condition.
For the agent planning and evaluation lead, this becomes relevant when a planner produces steps that look plausible but fail in execution. The immediate decision is whether plans are feasible, sufficiently complete, and bounded. The scope should be narrow enough that the evaluation team can identify a meaningful alternative, including continuing the current approach if the available evidence already supports it.
The practical failure to challenge is that a plan assumes a permission or tool result that has not been established. That concern gives the work a concrete direction: establish what the relevant boundary should do, inspect how it behaves or would operate, and identify the owner of any gap. It should not be diluted into a broad transformation exercise before the specific question has been answered.
Coverage
Examine missing prerequisites alongside invalid step order
Keep the review distinctions visible
- Missing prerequisites
- Check that a plan obtains information or authority before needing it.
- Invalid step order
- Verify prerequisite order against the actual tool and dependency constraints.
- Replanning after new evidence
- Introduce new evidence and inspect whether the plan revises its next steps.
Missing prerequisites
For this scope, missing prerequisites should be examined alongside task goals. Give the planner tasks with explicit prerequisites and inspect whether it establishes them before proposing dependent actions. A missing permission, unavailable tool, or unverified fact can make a plausible plan infeasible. The proposed rubric should identify the specific unsupported dependency rather than score plan detail as quality.
Invalid step order
For this scope, invalid step order should be examined alongside tool and dependency constraints. Test step ordering where one action changes the validity of another. Preserve expected dependencies and inspect whether execution follows the plan or improvises around it. This helps distinguish a planning defect from an execution defect and reveals plans that succeed only because later components repair their mistakes.
Replanning after new evidence
For this scope, replanning after new evidence should be examined alongside accepted plan and outcome criteria. Introduce new evidence that invalidates an assumption and observe replanning. The agent should revise, clarify, or stop according to the task contract. The evaluation should inspect whether obsolete steps remain active or whether the planner can explain why the revised path is now appropriate.
Prerequisites
Start with task goals
Give each input a specific job
- Task goals
- Defines the operating boundary.
- Tool and dependency constraints
- Examine invalid step order.
- Accepted plan and outcome criteria
- Shows conditions affecting replanning after new evidence.
- Product or platform owner
- Confirm access and a reviewer who can resolve gaps.
The starting material is task goals, tool and dependency constraints, and accepted plan and outcome criteria. Review these together rather than assuming one document tells the whole story. A stated rule may differ from a configured behaviour, and a proposed responsibility may not yet have an available owner. Identify those differences before selecting the detailed work.
Participation should include a product or platform decision owner, a technical lead, a data owner, and a competent reviewer for the relevant boundary. The customer should define which information and environments may be used. Synthetic or sanitised examples can support preparation when they preserve the relevant behaviour; if they omit a material dependency, record the limitation and agree how the missing evidence would be obtained.
How the work proceeds
Tie the deployment finding to observable behaviour
Follow the evidence through the work
- Missing prerequisites
Use task goals to define the decision criteria.
- Invalid step order
Inspect tool and dependency constraints and retain observations that challenge the expected result.
- Replanning after new evidence
Use accepted plan and outcome criteria to identify unresolved conditions and a responsible owner.
- Replanning test cases
Link the recommendation to its evidence, remaining limits and next validation condition.
Evaluate plans against available tools, required evidence, dependencies, and authorised actions. Include tasks where an early assumption is contradicted later so replanning can be observed. The proposed rubric should distinguish a plan that is incomplete but recoverable from one that cannot succeed without violating a constraint.
A practical first slice would focus on missing prerequisites and then follow its consequences into invalid step order. Keep replanning after new evidence visible as a separate review question rather than assuming it is satisfied by the first result. The evaluation team should record observations, competing explanations, and changes to the original assumptions. A second reviewer can challenge whether the evidence supports the recommendation and whether an apparently successful example omitted the condition that matters most.
Illustrative scenario
Resolution planned too early: revise, clarify or stop
Examine the constructed decision boundary
Illustrative scenario
- Resolution planned too early
An agent plans an update before obtaining the required eligibility evidence.
- Unavailable prerequisite
Withhold eligibility and inspect replanning and any later attempted action.
- Revise, clarify or stop
Follow the task contract instead of carrying the unsupported assumption into execution.
Consider an illustrative situation in which an agent plans to resolve a case before checking the evidence needed for an authorised update. This is a constructed example, not a client engagement or an observed Softlander outcome.
For the prematurely planned case resolution, an illustrative run would make eligibility evidence unavailable. The planner should revise, clarify, or stop according to the task contract. The reviewer would inspect whether later execution compensates for a weak plan or simply carries an unsupported assumption into a tool action.
The useful learning would be the difference between the expected behaviour and the behaviour or responsibility actually demonstrated. If the exercise leaves replanning after new evidence unresolved, that should remain a named next step. A recommendation to narrow the scope, improve a prerequisite, or hold the decision can be appropriate when the example exposes a material dependency.
Indicative artefacts
Use replanning test cases to make the next action explicit
Every artefact needs an operating use
Illustrative working format
- Planner quality rubric
- Boundary — task goals.
- Feasibility and sequencing findings
- Findings about invalid step order.
- Replanning test cases
- Action on replanning after new evidence.
Indicative outputs include planner quality rubric, feasibility and sequencing findings, and replanning test cases. Their form should follow the decision and the customer’s existing operating records. The first should establish the assessed boundary and relevant observations; the second should make material gaps or dependencies inspectable; the third should identify the action, owner, and evidence needed to revisit the decision.
Acceptance should be tested through use: can the responsible owner explain what was examined, identify an unresolved issue, and act on the next step without reconstructing the whole engagement? Findings should preserve counterevidence and limits. A polished document is insufficient if it does not help resolve the customer’s specific question.
Scope and authority
A deployment finding remains bounded to the tested conditions
Keep analytical work separate from decision authority
- Authorised deployment cases and traces
- Can include planner quality rubric.
- Product or platform owner
- The customer retains release authority and owns remediation priorities.
- Universal vendor or component assurance
- Cannot be inferred from feasibility and sequencing findings.
The proposed evaluation concerns the named deployment, configuration, cases, and authorised environment. Findings do not certify a component vendor, establish universal safety, or approve release. Unexamined provider paths and specialist questions remain explicit limitations.
Ways to engage
Related engagement scopes
AI Agent Evaluation Evidence Diagnostic
Find the gaps between your release decision and the evidence behind it. Start with one agent workflow and a bounded set of questions.
AI Agent Evaluation Foundation
Turn scattered tests into a repeatable release discipline. Connect the behaviours that matter to datasets, evaluation criteria and evidence your team can maintain.
AI Agent Component Evaluation
Find where agent behaviour breaks down. Follow memory, retrieval and delegation through your workflow, then isolate the components that need attention.
Continue exploring
Related insights and resources
Engineer the harness, not just the prompt
Define the contracts around an agent's decisions: what state it trusts, which actions can execute, how interrupted work resumes, and what evidence an operator needs to recover safely.
What would justify releasing this agent?
Replace a single pass rate with a reviewable chain of requirements, cases, observations and judgements. Distinguish task quality, control failures and uncertainty before increasing an agent's exposure.
The evidence behind the go-live decision
A working paper for leaders who need to explain why an agent workflow is ready for its next release. Includes a proposed evidence-pack structure, decision table and review agenda.