How do you evaluate an agent orchestrator?

An agent orchestrator can produce an excellent final answer after wasting work, crossing an authority boundary or recovering by accident. It can also produce a poor answer even though decomposition and routing worked, because a specialist agent or external tool failed. Scoring only the last message makes both diagnoses impossible.

Evaluate the orchestrator as a workflow control plane. Its job is to translate a goal into bounded work, assign that work, preserve state and authority, respond to evidence, verify completion and stop. The test should follow those responsibilities from request to business state.

Define what the orchestrator owns

Orchestration is an overloaded term. In one system it means choosing a prompt template. In another it creates sub-agents, allocates tools, retries tasks and approves a final action. Write the responsibility boundary before choosing metrics.

A useful contract answers:

• What goals may the orchestrator accept?
• Which steps may it create, reorder or cancel?
• Which workers, models and tools may it select?
• What authority can it delegate, and can a worker delegate again?
• Which state transitions are authoritative?
• What evidence completes a step?
• Which failures may it retry, compensate or escalate?
• What conditions end the workflow?

Without that contract, the same behaviour may be called intelligent adaptation in one review and unauthorised scope expansion in another.

Establish a simpler baseline

Multi-agent or dynamic orchestration is not automatically better than a single agent or deterministic workflow. Compare against the simplest credible implementation using the same task set and, where possible, the same underlying models and tools.

The baseline might be:

1. one agent with all permitted tools;
2. a fixed sequence of specialized calls;
3. ordinary rules plus a model at one ambiguous step; or
4. a human-run workflow with measured time and rework.

The orchestrator should demonstrate a decision-relevant gain—such as stronger outcome quality, safer separation of authority, better recovery or lower quality-adjusted cost. “More agents collaborated” is activity, not value.

MultiAgentBench evaluates collaboration through milestones and compares coordination topologies, illustrating why system structure belongs in the evaluation rather than being treated as an invisible implementation detail.

Score seven control responsibilities

Use a rubric that separates outcomes from coordination quality.

Responsibility	Passing behaviour	Evidence
Decomposition	Creates necessary, non-overlapping work with clear completion criteria	Plan versions, task graph and rationale
Routing	Assigns work to a capable, permitted worker without avoidable escalation	Route decision, alternatives, cost and result
Context	Gives each worker sufficient, current, authorised information	Handoff payload, provenance and omissions
State	Records accepted, active, completed and failed work consistently	Durable transitions and reconciliation
Authority	Delegates only the permissions required for the assigned step	Principal, grant, scope, expiry and policy result
Recovery	Handles timeout, partial effect and worker failure without duplication	Fault trace, retry or compensation and end-state check
Verification	Confirms the workflow outcome and stops at an appropriate terminal state	Independent checks, unresolved exceptions and final record

Add end-to-end quality, latency and cost, but do not collapse the rubric into one number too early. A weighted aggregate can hide a prohibited action behind strong average task performance.

Build scenarios around coordination hazards

Representative happy paths establish basic function. The useful evidence comes from controlled disruptions:

• two workers claim the same task;
• a worker completes after its lease expires;
• a tool times out after changing external state;
• a sub-agent returns a plausible result without required evidence;
• the requested goal changes while work is active;
• two sources conflict on a material fact;
• a worker asks for broader permission than its task requires;
• a dependency becomes unavailable;
• the best route is to stop or ask a human; and
• a late message arrives after the workflow is closed.

For each scenario, define the expected state transitions and prohibited outcomes before running it. Then repeat runs because model-mediated coordination varies. Report distributions and severe failures, not only average completion.

Research on reasoning-and-action evaluation similarly argues for observing intermediate progress and tool interaction, not merely the final result (AgentBoard).

Distinguish model error from orchestration error

When a worker returns a wrong result, ask whether the orchestrator selected an unsuitable worker, omitted necessary context, ignored a failed check or behaved correctly given the worker contract. When duplicate work occurs, ask whether the assignment protocol lacked a durable claim, the worker violated its lease or the orchestrator processed a completion twice.

The MAST study groups observed multi-agent failures into specification and system design, inter-agent misalignment, and task verification or termination. That taxonomy is a useful challenge to the reflex of blaming “the model.” Many failures arise because roles, protocols or completion conditions are underspecified.

Use fault attribution labels such as:

• invalid task specification;
• incorrect decomposition;
• wrong route;
• insufficient or excessive context;
• worker execution error;
• lost, duplicated or reordered message;
• authority-policy failure;
• verification failure; and
• termination failure.

Allow more than one label. The goal is not to allocate blame; it is to locate the control that should change.

Test state and recovery like a distributed system

An orchestrator coordinates work across processes and external systems. Crashes, retries and late messages are normal operating conditions. Durable-workflow research such as ExoFlow treats logged workflow structure and coordinated checkpoint/recovery as core fault-tolerance mechanisms. Agent orchestration inherits the same problem even when task decisions are model-generated.

During fault injection, inspect whether the system:

1. persists the task assignment before dispatch;
2. distinguishes accepted from completed work;
3. deduplicates completion and external effects;
4. checks authoritative state after ambiguous timeouts;
5. resumes from a known checkpoint rather than reconstructing from prose;
6. expires delegated credentials and leases; and
7. reconciles orphaned work after restart.

A fluent explanation of recovery is not evidence. Force the fault and inspect the resulting state.

Make stopping a first-class result

Orchestrators are often rewarded for finishing, which encourages needless retries, invented subtasks and premature claims of completion. The rubric should recognise appropriate stops:

• the goal is outside permitted scope;
• required evidence is unavailable;
• a conflict needs an accountable human decision;
• the action is irreversible and lacks confirmation;
• the budget or time limit is reached; or
• a verification check fails.

The NIST AI RMF Core calls for defined human-AI roles and oversight responsibilities. In an orchestrated workflow, that translates into named escalation owners, structured handoff data and a state that clearly distinguishes “waiting for decision” from “failed” or “complete.”

Use a decision table for release

Summarise evidence by consequence, not by demo appeal.

Finding	Release implication
Better final quality, but duplicate external actions occur under timeout	Block write authority; repair idempotency and recovery
Reliable routing, but no gain over a single-agent baseline	Prefer the simpler design unless another benefit is evidenced
Correct normal paths, but conflicting instructions never escalate	Narrow scope and add a tested decision gate
Higher cost with materially better verified outcomes on complex cases	Route only eligible complex cases; retain the cheaper baseline elsewhere
Workers complete correctly, but the orchestrator cannot verify final state	Do not treat completion messages as workflow completion

The release owner should see scenario coverage, repeated-run variance, severe failures, baseline comparison, configuration versions and remaining unknowns. That is enough to make a bounded decision: release, narrow, remediate or stop.

An orchestrator earns its place when it makes complex work more controllable and inspectable. If evaluation only shows that it can generate an elaborate plan, the most important part of the system remains untested.