What "good" looks like
An MCP eval scores an agent on real third-party servers, not mocks, and grades whether the end-to-end task completed. The unit of evaluation is not the function call, it is the agent-plus-registry system. Report success per server and per tool: an aggregate number over a registry hides which server, which tool, and which layer failed.
The reference targets on MCP-Universe (paper-time scores, August 2025):
- GPT-5 at 43.72%, the best frontier model, still failing a majority of tasks.
- Grok-4 at 33.33%.
- Claude-4.0-Sonnet at 29.44%.
Where those numbers stand (fetched as of 2026-10-01). The MCP-Universe leaderboard has barely moved: Gemini-3-Pro-Preview leads its function-calling track at 44.59%, GPT-5-High leads the ReAct track at 44.16%, and the paper's 43.72% is still listed. Its newest entries are late-2025 models, so it no longer tracks the frontier[1]. Other MCP suites have run ahead of it. Toolathlon-Verified, released June 30, 2026, lists GLM 5.3 Flash (max) at 78.4% Pass@1[2]. MCPMark Verified, the default task set since June 12, 2026, lists gpt-5.5 (xhigh) at 92.9%[3]. These are different task sets with different graders, so do not compare them with each other, and MCPMark itself says scores from its earlier task versions are not comparable to Verified. Two lessons survive the churn. A published MCP score is only as old as its task version. And the Pass@1 to Pass^3 drop is the reliability cost: Kimi K3 goes from 76.5 to 68.5 on Toolathlon-Verified[2].
Scope first. AST-match versus execution-grounded scoring, hallucinated function names, and strict-mode decoding are single-registry problems, covered in Tool use and function calling[4][5][6]. This page owns what changes at registry scale with third-party servers: tool discovery across dozens of servers, schemas you do not control, and tool results you did not write.
The reference benchmark. MCP-Universe covers 6 domains across 11 real MCP servers: maps, repo management, financial analysis, 3D design, browser automation, and web search. Two findings matter more than the leaderboard. Cursor, the enterprise agent the paper tested, did not beat a plain ReAct loop, and the dominant failure drivers are long-context token growth and unfamiliarity with unknown tools[7].
The evaluator taxonomy is the design lesson. MCP-Universe uses three evaluator types: format evaluators check output compliance, static evaluators match content, and dynamic evaluators fetch real-time ground truth at grading time. The dynamic evaluator is the key trick for live-server evals, because the correct answer to "cheapest flight Tuesday" changes hourly. Date-stamp every result and link the live leaderboard rather than quoting stale numbers[8].
flowchart TD
TASK["Task against a real MCP registry"] --> ROUTE["Route: pick the server and tool"]
ROUTE -->|retrieval error| FAILR["Fix routing first"]
ROUTE --> CALL["Construct call, run against live server"]
CALL --> EVAL{"Which evaluator fits?"}
EVAL -->|structure| FMT["Format check"]
EVAL -->|stable fact| STAT["Static check"]
EVAL -->|time-sensitive| DYN["Dynamic ground-truth fetch"]
FMT --> GATE{"Per-server, per-tool pass?"}
STAT --> GATE
DYN --> GATE
GATE -->|yes| SHIP["Ship, date-stamp result"]
GATE -->|no| FIXLAYER["Fix the failing layer, not the aggregate"]
Figure: MCP-Universe pass rates for agents on 11 real MCP servers (paper-time scores, August 2025): GPT-5 leads at 43.72%, ahead of Grok-4 at 33.33% and Claude-4.0-Sonnet at 29.44%, yet every frontier model still fails a majority of tasks.
Secondary metrics
| Metric | What it catches | How to score |
|---|
| Tool routing at registry scale | Retrieval errors when the agent sees many servers | LiveMCPBench runs 95 tasks over 70 servers and 527 tools: retrieval errors are nearly half of all failures; Claude-Sonnet-4 reaches 78.95% while most models land in the 30-50% band[9]. Its leaderboard sheet still lists only August 2025 results (fetched as of 2026-10-01), so read 78.95% as a 2025 ceiling |
| Per-server operational matrix | A specific server being slow, costly, or wrong | Task accuracy crossed with latency and token consumption under a fixed agent and LLM, the MCPBench design; compares alternative servers across STDIO versus remote SSE transports[10]; the spec has deprecated HTTP+SSE in favor of Streamable HTTP since 2025-03-26 and the 2026-07-28 revision formalizes it, so expect the SSE leg to age out[11] |
| Security triple | Behavior when a tool result carries an injected instruction | AgentDojo across 97 realistic tasks and 629 security cases: report benign utility, utility under attack, and targeted attack success rate; frontier models fail many tasks even unattacked[12] |
If your agent sees more than a handful of servers, measure routing before anything else. If you defer tool loading behind a search step (Anthropic's tool search tool, OpenAI's tool_search on gpt-5.4 and later), score the search and the call separately: did the right tool come back, and did the model then call it correctly. Anthropic's docs put the point where selection degrades at 30 to 50 loaded tools and say tool search typically loads only 3 to 5[13][14]. Report all three security numbers: a defense that halves utility is not a defense.
Offline eval design
Build a per-tool eval matrix as a comparison table. Columns: tool, accuracy, latency, tokens, and date of last schema change. Three tasks per tool with static or dynamic evaluators is enough to make the matrix real. The matrix is what turns "the agent got worse" into "the repo server's search tool got worse."
Then add schema-drift regression. The steps:
- Snapshot the full tool schemas from every server you depend on.
- Diff the snapshots in CI.
- Replay recorded traces after any server, schema, or prompt change.
This is tooling practice rather than an academic benchmark. No published benchmark measures drift, but third-party servers change under you without a release note, and the diff job is the only warning you get.
The spec moved under your harness. The 2026-07-28 MCP revision, released July 28, 2026, drops the initialize handshake and protocol-level sessions: every request now carries its protocol version and client capabilities in _meta, and servers advertise themselves through a new server/discover call[11]. Four consequences for the snapshot job:
- Record server identity from
server/discover or the result _meta, not from a handshake your client no longer sends.
- Snapshot tool lists per credential.
tools/list must not vary per connection but may vary with the authorization on the request[15].
- Sort tools by name before diffing. Servers SHOULD return a deterministic order, which is not a guarantee, and
tools/list results now carry ttlMs and cacheScope freshness hints, so log both with each snapshot[11].
- Resolve
$ref before comparing. The revision loosens inputSchema and outputSchema to any JSON Schema 2020-12 keyword[11].
The same revision deprecates Roots, Sampling, and Logging, so a server that needs sampling to work is a migration risk, not a stable dependency[11].
Grade with MCP-Universe's evaluator split. Format checks for structure, static checks for stable facts, dynamic ground-truth fetches for anything time-sensitive[7].
Online eval design
Three production signals.
Schema-drift alerts. Run the snapshot-diff job on a schedule against live servers, not just in CI, and page when a depended-on tool changes shape.
Injection canaries. Seed a small stream of AgentDojo-style poisoned tool results through the real agent and alert on any canary action executing[12]. Put some canaries in tool descriptions and annotations too, not only in results. MCPTox plants the payload in tool metadata across 45 live servers and 353 tools: o1-mini reached a 72.8% attack success rate, and the best refusal rate (Claude-3.7-Sonnet) was under 3%[16]. The current spec says clients MUST treat tool annotations as untrusted unless the server is trusted[15].
Token and latency budgets per server. MCP-Universe's failure analysis says long-context growth is a primary killer, so track tokens consumed per server per task and alert on budget breaches before they become truncation failures[7].
Top benchmarks
- MCP-Universe, https://github.com/SalesforceAIResearch/MCP-Universe. 6 domains, 11 real MCP servers, execution-verified with format, static, and dynamic evaluators; maintains a live leaderboard[8].
- LiveMCPBench, https://arxiv.org/abs/2508.01780. 70 servers and 527 tools; the registry-scale routing benchmark[9].
- MCPMark, https://arxiv.org/abs/2509.24002. 127 CRUD-heavy tasks across Notion, GitHub, Filesystem, Postgres, and Playwright servers with programmatic verifiers; the Verified set pins every environment version[17][3].
- Tool Decathlon (Toolathlon), https://arxiv.org/abs/2510.25726. 108 tasks over 32 applications and 604 tools (ICLR 2026); Toolathlon-Verified is the current release and reports Pass@1, Pass@3, and Pass^3[18][2].
- AgentDojo, https://arxiv.org/abs/2406.13352. 97 tasks and 629 security cases; the standard for utility-under-attack measurement[12].
- MCPTox, https://arxiv.org/abs/2508.14925. 1,348 tool-poisoning cases over 45 live servers; the benchmark for malicious instructions hidden in tool metadata[16].
- MCP-Universe harness: execution-verified tasks against real servers; it also runs MCPMark tasks[8].
- LiveMCPBench: routing at registry scale, with its copilot-style tool retriever as a baseline.
- MCPBench: the per-server accuracy-latency-tokens matrix across transports[10].
- AgentDojo: the injection-probe suite.
- Inspect AI: the general harness when you write your own tasks; its sandboxing and scorer primitives fit the static-plus-dynamic evaluator pattern well.
Common pitfalls
| Pitfall | Why it bites | Fix |
|---|
| Treating live-server scores as reproducible | The servers are non-stationary; the same task set scores differently next month with zero agent changes | Use dynamic evaluators and date-stamp every result[7]. Pin versions of any server you can run yourself, as MCPMark did for its GitHub and Notion servers[3] |
| Reading only the end-to-end number | Once tool count grows, failures concentrate in tool retrieval and parameter construction; the end-to-end score hides the layer you can actually fix | Break out routing: LiveMCPBench attributes nearly half of failures to retrieval alone[9] |
| Assuming clean-run success implies safety | A benign pass rate says nothing about behavior when a tool result carries an injected instruction | Report the under-attack numbers, not just benign utility[12] |
| Ignoring silent schema drift | A third-party server renames a parameter and your agent breaks with zero code change on your side | Snapshot and diff schemas, or the first alert is a customer |
Canonical reading
MCP-Universe (Luo et al., 2025) is the canonical paper because it measures what this category actually is: agents against 11 real servers with execution-based evaluators, not mocks[7]. Its two sobering launch results, GPT-5 at 43.72% and Cursor failing to beat plain ReAct, set honest expectations for anyone shipping MCP agents. Newer suites show scores climbing, so read it for the evaluator taxonomy as much as the scores.
Day-1 starter
- Pick your top 10 tools across servers and write 3 tasks each, graded by static evaluators for stable facts and dynamic ground-truth fetches for anything time-sensitive.
- Snapshot every tool schema you depend on and add a CI job that diffs the snapshots on each run; a changed schema fails the build until traces are replayed.
- Add 10 AgentDojo-style injection probes to the suite and report the triple: benign utility, utility under attack, and targeted attack success rate. The AgentDojo recipe is the runnable pattern, and the agentic security chapter covers tool poisoning as a class.
Cross-cutting links