What "good" looks like
A multi-agent system is scored on two axes at once: whether the team hit its milestones, and where the collaboration broke when it did not. The primary metric is a milestone-based task score with partial credit, paired with a failure-mode profile of sampled traces. Either one alone is uninterpretable.
The ship targets:
- A milestone-based task score with partial credit, so partial completion earns partial credit rather than a zero.
- A MAST failure-mode profile over sampled traces, tallied by category.
- A cost-adjusted result that beats a single strong agent at equal token budget.
MARBLE is the reference for the first axis. MultiAgentBench (MARBLE) scores explicit sub-goal KPIs alongside collaboration quality, communication efficiency, and protocol compliance[1].
Two of its findings carry into your own harness. Graph topology performs best in the research scenarios, and cognitive planning improves milestone achievement by about 3%[1]. Topology and planning strategy are eval variables, not fixed architecture decisions.
This page covers what is specific to N agents sharing a task: credit assignment, communication failure, orchestrator quality, and cost normalization. Pass^k, deterministic state checks, and single-agent error attribution live in Long-horizon planning; do not re-derive them here.
flowchart TD
RUN["Multi-agent run: per-agent traces"] --> MS["Milestone score with partial credit"]
RUN --> MAST["MAST profile of sampled traces"]
RUN --> ABL["Ablation: single strong agent, equal budget"]
MS --> GATE{"Beats single agent on cost-adjusted score?"}
ABL --> GATE
MAST --> CAT{"Failures mostly system-design or coordination?"}
GATE -->|yes| SHIP["Ship"]
GATE -->|no| REGR["Architecture is the regression: fix wiring"]
CAT -->|yes| FIXORG["Fix orchestration, not the model"]
CAT -->|no| FIXWORK["Fix the worker or verifier"]
Secondary metrics
| Metric | What it catches | How to score |
|---|
| Cost-adjusted success vs single agent | Single-agent quality matched at a multiple of the cost, which quality-only reporting hides | Compare the same tasks against a strong single agent at equal token budget |
| Communication rounds per milestone | Agents talking past each other: a rising round count with flat milestone completion | Count rounds per milestone from per-agent spans |
| Per-agent error attribution | Which agent introduced the error that sank the run | No standard automatic metric exists, so trace annotation carries the load. Agent-as-judge gives step-level verdicts on samples at reasonable cost[2] |
Offline eval design
Start from the MAST taxonomy. It defines 14 failure modes in 3 categories (specification and system design, inter-agent misalignment, task verification), built from more than 1,600 annotated traces across 7 frameworks, with human agreement at kappa 0.88[3]. The authors ship an open LLM-as-judge annotator and the dataset, so you can profile your own traces without building a rubric from scratch[4].
Be honest about credit assignment: no standard automatic metric exists, and MAST-style trace annotation is current best practice, not a stopgap.
Evaluate the orchestrator separately from the workers. Magentic-One is the reference design: an Orchestrator maintains a task ledger and a progress ledger over WebSurfer, FileSurfer, Coder, and ComputerTerminal workers, and the system is statistically competitive with strong single-agent systems on GAIA, AssistantBench, and WebArena[5] [6]. Score the orchestrator on task decomposition and re-planning; score workers on step execution. A sound plan with a botched worker step is a different bug from a sound execution of a bad plan.
Figure: Orchestrator vs worker evaluation split in a Magentic-One style multi-agent system. The orchestrator maintains a task ledger and a progress ledger and is scored on task decomposition and re-planning, while the four workers (WebSurfer, FileSurfer, Coder, ComputerTerminal) are scored on step execution.
Compare topologies per scenario, not globally. Evaluate star, chain, tree, and graph per scenario; MARBLE's scenario-dependent results say the best wiring changes with the task[1].
Every offline table needs an ablation row: same task, single strong agent, same budget. If that row wins, the multi-agent architecture is the regression.
Online eval design
Log full traces with per-agent spans so every message, tool call, and hand-off is attributable to its agent. Dashboards should slice by agent:
- Per-agent error attribution.
- Token spend per milestone.
- Communication-round counts per milestone.
Alert on the two signatures of a failing team. Retry loops (the same agent pair exchanging near-identical messages) and cost-per-milestone drift, which is where multi-agent budgets quietly explode.
Adopt observability rather than rebuilding it. Frameworks for stage-attributable scoring of agentic workflows are worth adopting[7].
Run periodic MAST annotation over a trace sample with the open annotator; the category tally over time is your system-design health metric[4].
Top benchmarks
- MARBLE: milestone-scored scenarios across topologies.
- MAST annotator: failure-mode profiling of your own traces, the fastest path to a taxonomy-grounded diagnosis[4].
- Langfuse: production trace logging with per-agent spans and annotation queues.
- Inspect AI: harness-level control for isolated, repeatable runs of the same scenario.
- Agent-as-judge pipelines: step-level verdicts on samples where deterministic checks do not reach[2].
Common pitfalls
| Pitfall | Why it bites | Fix |
|---|
| Reading end-task pass rate as model quality | Most failures are system-design and coordination problems, not model problems, so swapping in a stronger model to fix a specification bug changes nothing[3] | Profile with MAST, then fix orchestration |
| Trusting the checker agent | Verification weakness is its own MAST category; a wrong answer approved by a weak checker scores as success in a naive harness | Give the verifier its own eval |
| Comparing runs without isolation | Agent side-effects contaminate reruns, so two runs of the same task are not comparable and deltas are noise | Use isolated environments plus repetition |
| Reporting quality without cost | A system that matches a single agent at several times the token spend is a worse system | Report the equal-budget ablation row |
Canonical reading
MAST (Cemri et al., 2025) is the canonical read because it is the first systematic answer to "why do multi-agent systems fail": 14 failure modes from 1,600+ traces across 7 frameworks, annotated at kappa 0.88, with an open annotator you can run today[3]. Its core finding, that most failures trace to system design and inter-agent coordination rather than model capability, reframes the whole eval problem: profile the organization, not just the employees.
Day-1 starter
- Log 20 full multi-agent traces end to end, with every message and tool call attributed to the agent that produced it.
- Annotate those 20 traces with the MAST taxonomy using the open annotator and tally the three categories; the tally is your first diagnosis[4].
- Run the same 20 tasks through your strongest single agent and compare pass rate and total tokens. If the single agent wins the cost-adjusted comparison, fix the architecture before the prompts.
Cross-cutting links