What "good" looks like
A native multimodal eval scores two things: task accuracy on interleaved benchmarks, and whether the model actually used information from each modality. The second axis is the one aggregate accuracy hides. A model can score well on a benchmark and still ignore image layout in your product by falling back to OCR plus text-only reasoning.
The targets to track:
- Accuracy on MMMU and its higher-difficulty sibling MMMU-Pro, where text-only-solvable items were removed and the drop versus MMMU was 16.8 to 26.9% for the leading models of 2024[1]. That headroom is mostly gone at the top: as of October 2026 (fetched as of 2026-10-02), Artificial Analysis lists Claude Opus 5.5 at 88% on MMMU-Pro and GPT-6 Astra at 87%[2]. Treat both MMMU sets as floor checks, not rankings.
- Cross-modal grounding fidelity: for each output, is the model using information from each modality, or has it fallen back to OCR plus text-only reasoning?
Why grounding matters. A model that scores well on MMMU but ignores image layout in your real product is a failure mode, and the eval has to catch it.
The composite. Benchmark accuracy plus a domain-specific LaaJ check that asks, for each output, "which inputs informed this answer?" If the rubric can be satisfied without the image, the question is poorly designed. If the model's answer ignores the image, the model is leaking.
flowchart TD
IN["Interleaved input: image + text, audio, screen"] --> ACC["Benchmark accuracy: MMMU, MMMU-Pro"]
IN --> LAAJ["Grounding LaaJ: which inputs informed this answer?"]
ACC --> GATE{"Accurate AND every modality used?"}
LAAJ --> GATE
LAAJ --> LEAK["No image use?"]
LEAK -->|"model ignores image"| FIX["Model leaking: OCR + text-only fallback"]
LEAK -->|"rubric passes without image"| DESIGN["Question poorly designed: rewrite"]
GATE -->|yes| SHIP["Ship"]
GATE -->|no| FIX
Secondary metrics
| Metric | What it catches | How to score |
|---|
| Modality leakage | Model reads OCR text but ignores layout, charts, or spatial relations | LaaJ check on which modality informed the output |
| Hallucinated objects or text-in-image | Model "sees" something not present | LaaJ against the actual image contents |
| Refusal-on-image false positives | Model refuses a benign image because it triggered a safety filter | Refusal rate on a benign-image set |
| Per-skill accuracy | Uneven capability the aggregate hides | Benchmark splits; MM-Vet splits into 6 capabilities across 16 integrations and the breakdown is more useful than the aggregate |
Offline eval design
Start with public breadth. MMMU plus MathVista plus VQAv2 for breadth, and MMMU-Pro as the harder variant[1], with the caveat that the top models now sit at 87 to 88% on it. MMBench's CircularEval (rotating multiple-choice options) reduces position bias and is worth running where available[3].
Then build the domain set. On top of the public sets, build a domain-specific interleaved set: 100 paired examples per modality combination you ship (screenshot to action, audio to JSON, chart to caption). Score with a LaaJ rubric that explicitly asks which modality informed each output. This catches single-modality fallback that aggregate accuracy hides.
Add an adversarial probe. Vibe-Eval from Reka is a useful small-N adversarial set for high-difficulty image reasoning[4].
Borrow a document-reasoning template. GDP.pdf is a public model for the OCR-fallback check. It has 100 tasks over ten professional domains, each pairing a prompt with one or more PDFs, graded on 1,275 atomic rubric criteria across five attempts per task. Artificial Analysis has GPT-6 Astra (xhigh) first at 32.2% all-pass[5]. The authors name the failures that recur: misaligned tables, misread charts, skipped footnotes and exclusions, miscounted floor-plan symbols, scan noise, and amendments that supersede earlier text[6]. Seed your domain set with the same failure types from your own documents.
Online eval design
Sample by modality combination, not by query in aggregate. The failure rates differ across (text+image), (text+audio), and (image-only) inputs.
Run a grounding LaaJ on samples with the prompt: "did the response use information from each input modality?"
Track refusal rate stratified by modality. Many vision safety filters trigger on benign content.
Alert on two signatures:
- Per-combination accuracy drops.
- The OCR-fallback signature: response cites text from the image but ignores diagrams or layout.
Top benchmarks
- MMMU, https://arxiv.org/abs/2311.16502. 11,500 college-level questions across 30 subjects; the standard for college-level multimodal.
- MMMU-Pro, https://arxiv.org/abs/2409.02813. Text-only-solvable items removed. Claude Opus 5.5 leads Artificial Analysis at 88%, GPT-6 Astra at 87%, so it no longer separates the top models[2].
- BLINK, https://arxiv.org/abs/2404.12390. 3,807 multiple-choice from 14 classical computer-vision tasks; humans 95.7%, GPT-4V 51.26% in the original paper[7]. Those are 2024 figures, and the repo's own leaderboard stops at mid-2024 models, so run it on the models you compare instead of quoting the gap.
- GDP.pdf, https://arxiv.org/abs/2607.11192. 100 professional-document tasks across ten domains from Surge AI, published in July 2026; the paper reports the best of seventeen frontier models passing 30.7% of items and the worst 2%[6].
Figure: Where multimodal benchmarks discriminated in the original 2024 papers: removing text-only-solvable items cost leading models 16.8 to 26.9 percentage points from MMMU to MMMU-Pro, and BLINK's classical perception tasks showed a 44.4-point gap between humans (95.7%) and GPT-4V (51.26%). Top models now score 87 to 88% on MMMU-Pro, so read the MMMU-Pro drop as history.
- lmms-eval: cross-model benchmark runs at scale across image, video, and audio tasks; v0.7 (February 2026) added 25+ tasks and agentic task evaluation[8].
- MMBench harness: CircularEval with rotating multiple-choice options.
- Inspect AI: sandboxed reproducible runs with image logging.
- Anthropic Console Evaluate: side-by-side prompt iteration on a fixed image set.
- Hugging Face leaderboards: model selection at the macro level, but the domain set is what matters for the decision.
Common pitfalls
| Pitfall | Why it bites | Fix |
|---|
| Single-modality fallback | The most common failure mode and the hardest to catch with aggregate accuracy | The "which modality informed this answer?" check exists for it |
| No fine-grained skill breakdown | An aggregate hides uneven capability | MM-Vet's 6 capability splits are more useful than an aggregate, and so are the per-skill scores on MMMU |
| Reporting on saturated benchmarks | VQAv2 is largely saturated, and the top models score 87 to 88% on MMMU-Pro[2] | Gate on them as floors; rank on perception sets such as BLINK, document sets such as GDP.pdf (best all-pass 32.2%), and your own domain set |
| Skipping the domain set | Public benchmarks correlate weakly with whether your product works on your customer's screenshots | Pair every public number with a domain golden set[9] |
Canonical reading
MMMU is the most useful single read because it remains the standard reference for college-level multimodal capability and its construction is the template most domain extensions follow[10].
Read MMMU-Pro alongside it: the follow-up filters out questions that text-only models can answer, augments the candidate options, and adds a vision-only setting where the question is embedded in the image[1].
BLINK is the right second read for evaluating visual perception specifically, separate from visual question answering[7].
Day-1 starter
- 100 paired examples per modality combination you ship in production. The domain set is non-negotiable.
- A LaaJ rubric with an explicit "which inputs informed this answer?" field; bind it to your CI.
- Track refusal rate stratified by modality. Many over-refusals are modality-specific and only visible when you split.
Cross-cutting links