Measure the action, not just the answer

“Done—I updated the customer’s address.” The sentence is clear, polite and wrong. The tool call failed. Or it updated the wrong record. Or it succeeded twice after a retry. Or the user never had permission to make the change.

Answer grading would call the response convincing. Agent evaluation must ask what happened in the world.

Once a model can call tools, the unit of value and risk moves from generated text to a sequence of decisions and state changes. The answer still matters, but it is a receipt. A receipt should match the transaction.

Define the action as a state transition

For each evaluated task, record the relevant state before and the required state after. Include invariants that must remain true.

For an address change:

• before: authenticated user owns account A; address is X; no pending approval;
• requested transition: set account A address to Y;
• required controls: verify authorisation; validate address; request confirmation;
• after: address is Y exactly once; audit event exists; unrelated fields are unchanged;
• communication: tell the user success only after the committed state is confirmed.

This structure makes grading independent of eloquence. It also permits more than one valid plan while rejecting a plausible response that did nothing.

Tau-bench formalizes a related idea by comparing the final database state with the annotated goal state in tool-agent-user interactions. WebArena evaluates functional correctness in realistic web tasks. Their domains and metrics may not match an enterprise workflow, but the design lesson does: inspect the environment, not merely the transcript.

Grade four layers independently

A useful action evaluation separates outcome, process, authority and communication.

Layer	Core question	Example check
Outcome	Did the intended external state result?	Correct record has the new value
Process	Did execution handle dependencies and faults?	No duplicate after timeout and retry
Authority	Was every action permitted and approved?	Effective scope covered exact operation
Communication	Did the answer match state and next steps?	Success stated only after verification

Do not average these into one score too early. A perfect outcome reached through an unauthorised action is not a qualified success. A safe refusal may be the correct policy result even though the requested business outcome did not occur. A successful action paired with a false failure message creates rework and may cause a user to repeat it.

Report the joint result with separate failure reasons. That gives engineering a diagnostic signal and gives the decision owner a policy-relevant result.

Instrument the boundary around every tool

Capture tool name and version, arguments, caller identity, authorisation decision, approval token, start and completion time, response, retry relation and resulting audit event. Do not rely on the agent’s own narrated reasoning as the authoritative log.

At the boundary, enforce and record controls independently of the model:

• parameter validation and canonicalization;
• least-privilege identity and scope;
• action-specific approval;
• idempotency for retried writes;
• rate, cost and step limits;
• timeout and cancellation behaviour;
• sensitive-data filtering;
• immutable audit events appropriate to the workflow.

The point of instrumentation is not maximum telemetry. It is enough trustworthy evidence to reconstruct the action and decide whether controls operated.

Test the dangerous gap between request and effect

Agent failures concentrate in partial states. A tool may commit a change but return a timeout. One of several steps may succeed. An approval can expire between planning and execution. Retrieved data can change after the agent reads it.

Build cases for:

• response lost after successful commit;
• success response before eventual write failure;
• stale record between read and update;
• partial completion of a multi-step action;
• duplicate user request;
• tool returning an unexpected schema;
• permission revoked mid-workflow;
• cancellation while a write is in flight.

The ToolEmu paper uses an emulated tool environment to search for long-tail risky behaviour. Its authors also validate emulator findings with people, a useful reminder that simulation is evidence with a fidelity boundary. Use real sandbox integrations for release-critical state transitions where practical, and label emulated results as such.

Evaluate hostile influence over action

An agent may read email, documents, tickets or web pages before acting. Those data sources can contain instructions that conflict with the user’s intent. Answer-only testing often misses the security consequence because the final prose remains helpful.

Create paired tasks: one with benign data and one with an injected instruction. Verify that the legitimate task can still complete while the attacker goal does not. Inspect tool calls and state for data exfiltration, unauthorised messages, changed permissions or poisoned memory.

AgentDojo evaluates both utility tasks and security outcomes in dynamic tool environments. That dual view matters. A defence that stops every action is secure only in a narrow sense; a production control also needs to preserve legitimate work.

Use the right grader for each layer

Prefer deterministic checks for structured state, authorisation and required audit events. Examples include database queries, API reads, ledger reconciliation, policy-engine logs and invariant assertions.

Use semantic or human grading for communication, rationale and handoff quality. Where a model grades, calibrate it against trained reviewers and preserve disagreement. Do not let an LLM judge override an objective state mismatch.

A practical result object might contain:

{
  "goal_state": "fail",
  "policy": "pass",
  "recovery": "fail",
  "communication": "fail",
  "reason": "write committed after timeout; retry duplicated; response reported one success"
}

That result is more actionable than “response quality: 4/5.”

Connect action evidence to release authority

Evaluation depth should grow with the agent’s authority. Recommendation-only systems need evidence that suggestions and citations are useful, plus checks that people can review them. Systems with external write access need end-state verification, policy enforcement, replay-safe operations and tested containment.

The NIST AI RMF calls for testing before deployment and during operation, with measurement of reliability, security, resilience and human-AI configuration. Translate that broad guidance into local release criteria:

• zero unauthorised actions in the critical authority set;
• no duplicate state change under defined retry faults;
• qualified completion rate above the agreed threshold, with uncertainty;
• every reported success reconciles with confirmed state;
• safe handoff when state cannot be determined;
• production alerts for control failures that require action.

The thresholds are context-specific. The structure should not be.

Make the user-facing answer accountable to reality

Design response rules around verified state. The agent should distinguish “I completed,” “I requested,” “I could not confirm,” and “a person must review.” Avoid optimistic language based on an intended call or ambiguous tool response.

Test those distinctions as part of the contract. Deliberately return accepted-but-not-completed responses, delayed consistency and ambiguous timeouts, then verify that the wording changes with the known state. Where a system cannot confirm completion, the interface should make retry risk visible and direct the user to a safe reconciliation path.

In a partial failure, the best answer may explain what succeeded, what did not, whether the action is safe to retry and who owns the next step. That communication is part of recovery, not cosmetic copy.

An agent’s answer is evidence of what it said. The tool log is evidence of what it attempted. The authoritative external state is evidence of what happened. A credible evaluation aligns all three—and refuses to call the task successful when they disagree.