MMMU-Pro Benchmark

MMMU-Pro (A More Robust Multi-discipline Multimodal Understanding Benchmark) is a benchmark test for evaluating multimodal models — large multimodal models (LMMs) that jointly process images and text — designed as a more rigorous and difficult version of the MMMU benchmark. It was introduced in September 2024 in the paper MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark by Xiang Yue, Tianyu Zheng, Yuansheng Ni and colleagues, and was published at the ACL conference in 2025.

MMMU-Pro was created to address weaknesses in the original MMMU: the observation that some questions could be answered by text-only language models without using the image, and that the four-option multiple-choice format allowed models to reach correct answers through guessing or shortcuts within the options. Its goal is to measure a model's genuine multimodal understanding and reasoning across college-level academic disciplines, rather than its ability to exploit textual or option-based cues.

Background

MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI) is a college-level multimodal benchmark of 11,550 image-text questions covering six disciplines, 30 subjects, and 183 subfields, published at CVPR 2024. After its release, MMMU rapidly became a standard evaluation for new multimodal models. However, the MMMU-Pro authors found that text-only LLMs could answer some MMMU questions without any visual input — either because a question was largely independent of its image, or because models exploited correlations within the four answer options. Combined with the limited option space, this raised doubts about whether high MMMU scores reflected genuine multimodal reasoning, and motivated a more robust benchmark.

Construction

MMMU-Pro is built from MMMU through a three-step process:

• Filtering. Four strong open-source text-only LLMs — Llama-3-70B-Instruct, Qwen2-72B-Instruct, Yi-1.5-34B-Chat, and Mixtral-8×22B-Instruct — each attempted the MMMU questions ten times without access to the images. A question was treated as text-only answerable and excluded if at least three of the four models answered it correctly in the majority of trials. From the surviving pool, the authors sampled 1,800 questions, evenly distributed across the 30 subjects (60 per subject).
• Option augmentation. The number of answer options was increased from four to (usually) ten, making it harder for models to rely on guessing. The additional options were generated by GPT-4o, filtered by Claude 3.5, and then refined through two rounds of human review; during this step 70 questions were removed, leaving 1,730.
• Vision-only input. Each question was also rendered as a screenshot or photo, manually captured by annotators under varied backgrounds, fonts, and sizes, with the question text embedded in the image and no separate text prompt. The model must therefore read the question visually while also analysing the image, which removes the separate text channel and substantially reduces text-only shortcuts (though, as the authors note, some subtle statistical shortcuts may remain).

In total, MMMU-Pro comprises 3,460 items — the same 1,730 questions in two formats (standard and vision-input) — covering the six disciplines and 30 subjects of MMMU. Increasing the options from four to ten lowers the accuracy achievable by guessing: nominally from about 25% to 10%, though the paper's reported random-choice baseline is around 12–13% (12.8% in the standard ten-option setting and 12.4% in the vision setting), since not every question carries a full set of ten options.

Evaluation and Scoring

Models are assessed in three settings: a standard four-option setting (reported only for comparison with the original MMMU), a standard ten-option setting, and the vision-only setting. The overall MMMU-Pro score is computed as the average of the ten-option standard setting and the vision-only setting; the four-option setting and the original MMMU validation performance are included solely to highlight the increased difficulty.

Models are evaluated with both direct-answer and chain-of-thought (CoT) prompts, with the higher score reported. The authors find that CoT prompting generally improves results, though the benefit varies by model and discipline (large gains in reasoning-intensive fields such as Tech & Engineering, little or none in Art & Design). Explicit OCR prompts have little effect, since capable models already extract embedded text without them; the authors stress that accurate text recognition alone is not sufficient, because the vision setting also requires understanding the text in its visual context.

Results and Development

At the benchmark's release in late 2024, model performance on MMMU-Pro was substantially lower than on the original MMMU, with reported decreases of 16.8% to 26.9% across models. The best model at release, GPT-4o, fell from 69.1% on MMMU to about 52% overall on MMMU-Pro; other strong systems such as Claude 3.5 Sonnet and Gemini 1.5 Pro dropped comparably, and open-source models further still.

Because rigorous new human evaluation on MMMU-Pro would be costly, the authors approximated human-expert performance by reusing the original MMMU expert-evaluation data (in which experts recorded their solutions), adapting it to the harder ten-option and vision formats. The estimated overall accuracy is about 73.0% (low), 80.8% (medium), and 85.4% (high), which the authors emphasise is an approximation rather than a direct measurement.

Frontier models improved rapidly thereafter. As of mid-2026, on independent leaderboards that evaluate the official 3,460-item configuration (such as Artificial Analysis), the strongest models scored in the low-to-mid 80s — for example, Gemini 3.5 Flash at roughly 84% — approaching, but not yet clearly exceeding, the estimated best-human baseline of about 85.4%; most of the top entries are reasoning ("thinking") models. Google reported that Gemini 3 Pro reached 81.0% on MMMU-Pro. Reported figures vary noticeably across sources — some leaderboards place the top models nearer 81%, others nearer 84% — because harnesses, prompting protocols, and dataset versions differ. In particular, some leaderboards labelled "MMMU-Pro" evaluate a reduced four-option format of about 1,700 questions rather than the official ten-option and vision configuration, and report correspondingly higher scores; self-administered vendor figures may also differ from independently run evaluations.

Limitations

The MMMU-Pro authors note several remaining limitations. The dataset may still contain subtle statistical shortcuts that models can exploit, even after filtering and option augmentation, and its scope is limited to predefined disciplines and question formats. The vision-only setting increases difficulty but does not fully capture the complexity of human perception. Finally, because the reported human-expert performance is approximated from the original MMMU rather than measured directly, comparisons of models against a "human-expert level" carry potential bias. As a practical matter, the dataset has also been updated since its release; combined with differing harnesses and prompting protocols, this means MMMU-Pro scores are comparable only when the dataset version, option format, prompting protocol, and scoring harness are specified.

External links

• Official MMMU / MMMU-Pro repository on GitHub
• MMMU project page and leaderboard
• MMMU-Pro dataset on Hugging Face

Literature

• Yue, X. et al. (2025). MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark. ACL 2025. arXiv:2409.02813.
• Yue, X. et al. (2024). MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. CVPR 2024. arXiv:2311.16502.
• Hu, K. et al. (2025). Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos. arXiv:2501.13826.
• Liang, P. et al. (2022). Holistic Evaluation of Language Models (HELM). arXiv:2211.09110.