Tool-use evaluation
Verify that tool actions match the request and the agent's authority
An agent gains a write-capable connector or its tool schema changes. The decision is how to verify that tool actions match the request and the agent's authority. The work begins with your actual workflow and looks for the evidence that separates intended behaviour from a consequential exception. It is particularly relevant to application engineers and tool owners. A useful outcome is an explicit account of what was examined, what remains uncertain, and which decision the evidence can support.
The release evidence relationship
- Tool schemas
Establish the basis for tool selection.
- Argument binding
Tie the resource identifier to the approved target.
- Tool-use matrix
Retain the scoped finding and its unresolved dependencies.
Decision
Frame the decision around tool selection
Questions and the evidence they need
- Tool selection
- Inspect tool schemas for the relevant constraint or observation.
- Argument binding
- Inspect permission rules for the relevant constraint or observation.
- Observed effect
- Inspect request examples for the relevant constraint or observation.
The decision is how to verify that tool actions match the request and the agent’s authority. This is relevant to application engineers and tool owners when an agent gains a write-capable connector or its tool schema changes. Name the affected workflow, the consequence of a wrong decision, and the owner who can act on the result. The initial discussion should establish which evidence is already available and which uncertainty is material enough to investigate.
The specific failure hypothesis is that the agent selects a plausible tool but supplies the wrong record identifier or repeats a non-idempotent action. It remains a hypothesis until examined in the deployment. Existing internal tests or operating records may already resolve it; identify the uncovered question before commissioning additional work.
Mechanism
Correct tool syntax can still target the wrong record
Relationships within release evidence
- Tool selection
- Match the operation to the requested job.
- Argument binding
- Tie the resource identifier to the approved target.
- Observed effect
- Check persistence and duplicate execution.
What this specialty examines
Tool evaluation separates selection from execution. Check whether the chosen operation fits the task, whether arguments identify the correct resource, and whether the caller is permitted to act. Then verify the tool’s response and actual side effect rather than trusting an optimistic success message.
The review should check tool choice, arguments, sequencing, authorisation, response handling and observed side effects. Identify the normal route, relevant exceptions and the authoritative record of the outcome. Inspect the relevant business-system record, enforcement event or process evidence alongside any reported status. Keep incomplete cases visible, with an explanation of what their absence prevents the owner from concluding.
The tradeoff to make explicit
For this decision, narrow tools simplify evaluation while broad tools may reduce integration effort. Compare the available design or operating choices against the same intended outcome. If one option removes a required control or changes the task boundary, describe that difference before treating its apparent simplicity, speed or lower effort as an improvement. The decision should expose the tradeoff instead of burying it in an aggregate score.
Preparation
Connect the working inputs to the boundary being examined
Evidence packet for tool selection
Illustrative working format
- Tool schemas
- Match this description to the deployed or proposed boundary being examined.
- Permission rules
- Specify the subject, operation, resource, and context at the enforcing boundary.
- Request examples
- Keep expected behaviour and the limits of the constructed example explicit.
- Tool-side execution logs
- Retain the observed conditions and distinguish gaps from contrary evidence.
Start with tool schemas, permission rules, request examples and tool-side execution logs. Identify the owner, revision and operating context of each source. Sanitised examples may support initial design, but differences from actual conditions should remain explicit. Where direct access is unsuitable, a customer-controlled evidence or execution route may be considered, with agreed handling and retention.
The scope also needs the release owner, agreed expectations for a representative task case, and the authority to inspect the relevant boundary. Specify what evidence would distinguish the failure in which the agent selects a plausible tool but supplies the wrong record identifier or repeats a non-idempotent action from acceptable behaviour. If stakeholders disagree about intended behaviour, resolve the interpretation before classifying the observed difference as a defect.
How the work proceeds
Carry tool selection through to an accountable finding
From the scoped material to a decision record
- Tool schemas
Separate hard constraints from graded preferences.
- Argument binding
Apply the scoped check to observed effect.
- Tool-use matrix
Record the evidence, open dependencies, and the owner’s next decision.
Prepare the evaluation cases
Work with application engineers and tool owners to assemble tool schemas, permission rules, request examples and tool-side execution logs. Select a normal task, a consequential exception and a case with insufficient information. Identify the relevant requirement and expected result for each, then mark which outcomes need human adjudication. The preparation output is a case packet that separates blocking behaviour from graded quality preferences before any comparison is run.
Run and review the comparisons
Include malformed arguments, ambiguous identifiers, timeouts and duplicate calls. A valid JSON object may still express the wrong business action. Test assumptions in wrappers and adapters, especially where defaults or previously viewed records can silently supply missing values.
Choose the release or evaluation change
Use the reviewed cases to produce a tool-use matrix with action-level findings and proposed negative cases. Distinguish a product defect from a missing test, an ambiguous requirement or an unreliable measuring method. For each, name the proposed correction and the case that would check it. The release owner can then decide whether to change the implementation, improve coverage or restrict the intended use while a specific gap remains.
Indicative artefact
Make the tool-use matrix usable by its owner
Tool-use matrix
Illustrative working format
- Tool-use matrix
- Identify the assessed workflow, revision, and decision boundary.
- Action-level findings
- Name the missing evidence, responsible owner, and condition for revisiting the decision.
- Negative cases
- Name the unresolved condition, next action, and accountable owner.
A useful decision artefact is a tool-use matrix with action-level findings and proposed negative cases. Supporting material may include selected cases, configuration or process records, evidence references, a disagreement log and prioritised actions. Retain the case selection rationale and material disagreements alongside the result; otherwise the result may be mistaken for complete coverage.
Illustrative scenario
Examine observed effect through a constructed exception
A constructed case and its decision paths
Illustrative scenario
- Tool selection
An update reaches the most recently viewed account instead of the approved one.
- Argument binding
Bind the approved synthetic customer identifier to the executed request.
- Observed effect
Correct target selection before interpreting the fluent response as success.
Consider a fictional situation: a support agent identifies the right customer but applies an update to the most recently viewed account. The scenario is illustrative, with no client engagement, executed test or measured improvement implied.
For the account update, introduce two synthetic records with similar names. Verify that the identifier used at execution matches the approved target. If the tool accepts the wrong identifier, improving answer fluency would not resolve the underlying action-selection problem.
Scope and next action
Keep the conclusion within selected task cases
Confirm fit before the next commitment
- Tool selection
- This review is limited to selected task cases.
- Tool schemas
- The relevant owner must confirm access and the intended operating conditions.
- Tool-use matrix
- The customer retains the release decision.
A result applies to the recorded configuration and selected cases. It does not establish universal agent competence or authorise release. Any engagement depends on an agreed scope, qualified team, authorised access and confirmed availability. Bring the affected workflow, its current configuration or process, the triggering change and the evidence already available. The immediate scope question is prompted by the situation in which an agent gains a write-capable connector or its tool schema changes; confirm which missing observation would change the owner’s next action.
Ways to engage
Related engagement scopes
AI Agent Evaluation Evidence Diagnostic
Find the gaps between your release decision and the evidence behind it. Start with one agent workflow and a bounded set of questions.
AI Agent Evaluation Foundation
Turn scattered tests into a repeatable release discipline. Connect the behaviours that matter to datasets, evaluation criteria and evidence your team can maintain.
AI Long-Horizon Agent Evaluation
Examine what changes when an agent keeps working. Evaluate accumulated state, recovery and goal adherence across extended tasks.
Continue exploring
Related insights and resources
Engineer the harness, not just the prompt
Define the contracts around an agent's decisions: what state it trusts, which actions can execute, how interrupted work resumes, and what evidence an operator needs to recover safely.
What would justify releasing this agent?
Replace a single pass rate with a reviewable chain of requirements, cases, observations and judgements. Distinguish task quality, control failures and uncertainty before increasing an agent's exposure.
The evidence behind the go-live decision
A working paper for leaders who need to explain why an agent workflow is ready for its next release. Includes a proposed evidence-pack structure, decision table and review agenda.