What "good" looks like
Tool use looks like a single capability and is actually four nested ones: pick the right function, fill correct arguments, handle the result, and chain across turns. A tool-use eval scores whether the model produced the right call and whether that call actually works, not whether it emitted valid JSON.
The primary metric is a composite of two checks:
- AST-match: does the function name and argument structure match the gold?
- Execution-grounded success: when the mocked or real endpoint runs the call, does it return the right result?
The "good" targets:
- AST-match on simple cases above 90%.
- Honest irrelevance detection: the model should not invoke a tool when none is needed.
- Execution-grounded success on real or mocked endpoints, not just JSON validity.
The public bar. Berkeley's BFCL v4 is the public benchmark; its leaderboard was last updated 2026-04-12 (fetched as of 2026-10-01)[1]. The overall score is a weighted blend: agentic (web search and memory) 40%, multi-turn 30%, live 10%, non-live AST 10%, irrelevance 10%[2]. The leaderboard footnote still calls it an unweighted average, but the blog's weights reproduce the published top score. That top score is Claude-Opus-4-5-20251101 (FC) at 77.47%, with 88.58% on non-live AST, 68.38% on multi-turn, and 73.76% on memory[1]. Single calls are close to solved; multi-turn and memory are where the points go. The newest frontier entries on the board are late-2025 models (Claude Opus 4.5, GPT-5.2), so read 77.47% as a late-2025 ceiling, not today's.
The named failure mode. The Gorilla paper, which introduced API-Bench, showed that a fine-tuned model paired with a document retriever "substantially mitigates the issue of hallucination" in API calls, the most-cited single failure mode in this category[3].
flowchart TD
REQ["User request"] --> CALL["Model tool call"]
CALL --> AST{"AST-match: name and args match gold?"}
AST -->|no| FAIL["Fail: wrong call"]
AST -->|yes| EXEC{"Execution-grounded: endpoint returns right result?"}
EXEC -->|no| FAIL
EXEC -->|yes| IRR{"Irrelevance: tool skipped when none needed?"}
IRR -->|no| FAIL
IRR -->|yes| GATE{"AST-match above 90% on simple cases?"}
GATE -->|yes| SHIP["Ship"]
GATE -->|no| FIX["Fix the weaker axis"]
Secondary metrics
| Metric | What it catches | How to score |
|---|
| Hallucinated function names or parameters | The Gorilla failure mode: invented names or params | Track separately from AST-match[3] |
| Parameter type errors | Wrong numeric, date, and enum fields | Type-check arguments against the schema |
| Cost from extra tool calls | An agent making 4 calls when 2 would do, paying 2x for the same outcome | Count calls per task against a gold minimum |
| Irrelevance detection | The model calling a tool when no tool is needed | Rate of declining to call; without it the AST-match number is misleading |
Offline eval design
Public sets for breadth and scale. Use BFCL v4 for breadth. Use ToolBench from Tsinghua and OpenBMB for scale: 3,451 tools, 16,464 REST APIs, 126,486 instructions[4]. ToolBench calls live RapidAPI endpoints, so scores drift as those APIs change. StableToolBench re-serves the same tasks from a virtual API server (a response cache plus simulators) and keeps only solvable queries; use it for any number you plan to compare across months[5].
Your own harness. On top of public sets, build a harness over your own tools with mocked endpoints. Mocking is what enables deterministic scoring.
A minimum useful artifact is a 50-task set with, per task:
- The explicit gold tool.
- The gold arguments.
- The expected response.
Add 10 "no tool needed" negative cases; the model should answer from its own knowledge. Score AST-equivalence and execution outcome separately so the failure mode is debuggable.
Strict mode. Strict mode (Anthropic's term, used by other providers under different names) eliminates one whole failure class by constraining the schema during decoding[6]. Anthropic's docs say strict tool inputs follow the schema and the tool name is always valid[7], so the hallucinated-name rate should be zero under strict mode; if it is not, look at your registry or harness first. Strict guarantees shape, not correctness: the right function with the wrong date still passes. Evaluate with and without strict mode and report both, and check what each request actually ran. OpenAI's docs say Responses requests try to normalize your schema into strict mode when possible while Chat Completions stays non-strict by default[8], and Gemini exposes a validated mode for schema adherence[9].
Figure: Scoring AST-match and execution-grounded success separately turns every tool-calling failure into a named, debuggable case: both pass is a correct call, AST-match with execution failure points at the mock or gold response, execution success with an AST mismatch means the gold set is too strict, and a double failure is a genuine model error such as a hallucinated function name.
Online eval design
Log everything. Log every tool call, result, and final answer.
Track three rates:
- Schema-conformance: does the call validate?
- Retry rate: how often does the model re-call after a tool error?
- Latency by tool: long-tail tools are usually the next regression.
Default to strict. Strict-mode tool use eliminates schema errors at the cost of some flexibility. The right default is strict mode plus an unstrict fallback for known edge cases.
Alert on two signals. Alert on hallucinated-function-name detections, and on any tool that suddenly drops out of the call distribution (deletion in the registry, schema drift, or a model regression that makes the tool look unattractive).
Top benchmarks
- Instructor (over 8 million PyPI downloads in the 30 days to 2026-10-01[10]): Pydantic-based output validation with auto-retry on schema failure[11].
- Outlines: constraint-grammar-level enforcement during decoding[12].
- BFCL evaluator: parity with public scores.
- Inspect AI: sandboxed mock-endpoint eval with strong assertion primitives.
- Langfuse or Braintrust: production logging with per-tool slicing.
Common pitfalls
| Pitfall | Why it bites | Fix |
|---|
| Treating valid JSON as success when arguments are wrong | The single most common scoring error | AST-equivalence beats JSON validity, but execution-grounded success is the truth |
| No "tool-shouldn't-have-been-called" negatives | Without irrelevance cases, your AST-match number is biased upward. The reverse also bites: on BFCL, Ministral-8B-Instruct-2410 (FC) lists 100.00% on irrelevance, 0.00% on relevance and multi-turn, and 11.10% overall[1] | Add negative cases where the model should not call a tool, and read irrelevance next to call accuracy, never alone |
| Single-turn only when the product is multi-turn | Misses the multi-turn failures the product will hit | BFCL v4 has a multi-turn category for a reason[1] |
| Forgetting strict mode | Provider strict modes eliminate a whole class of schema-shape errors | Compare with and without strict mode (Anthropic and OpenAI strict: true, Gemini validated), and confirm which mode each request ran[6] |
Canonical reading
Gorilla (Patil et al., 2023) is the canonical paper because it introduced API-Bench and articulated the hallucination-of-API-names problem that motivates this whole category[3].
The paper's headline result, that retrieval plus fine-tuning substantially reduces fabricated API calls, is still the operational pattern for large API catalogs; for a small registry, strict mode closes the name problem at decode time. Read it with the BFCL leaderboard for the current numbers[1].
Day-1 starter
- Mock every tool. Build a 50-task set with gold function names and arguments and expected responses. Mocking is what enables deterministic scoring.
- Add 10 "no tool needed" negative cases. The model should answer from its own knowledge or refuse the tool politely.
- Score AST-equivalence and execution outcome separately. When they diverge, the failure mode is named and debuggable; when you report only one composite, it is not.
Cross-cutting links