MMMU Benchmark

From Systems Analysis Wiki
Jump to navigation Jump to search

MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI) is a benchmark test designed to evaluate the capabilities of multimodal models — large multimodal models (LMMs) that jointly process images and text — across a wide range of academic subjects. It was introduced in the 2023 paper MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI by a team led by Xiang Yue, and was published as an oral presentation at the CVPR conference in 2024[1].

MMMU is designed to measure three essential skills of multimodal models — perception, knowledge, and reasoning — at the college level. It is positioned as a multimodal counterpart to text-only knowledge benchmarks such as MMLU, and as a more demanding alternative to earlier visual question-answering benchmarks (such as ScienceQA), on which state-of-the-art models had begun to reach high scores. Unlike those benchmarks, MMMU requires deliberate reasoning combined with domain-specific expert knowledge rather than commonsense or simple perceptual understanding[1]. Despite the phrase "Expert AGI" in the title, the authors explicitly caution that strong performance on MMMU should be regarded as a necessary but not a sufficient criterion for expert-level artificial general intelligence[1].

Description and Content

MMMU consists of 11,550 multimodal questions covering six core disciplines, 30 subjects, and 183 subfields. The questions were manually collected by a team of about 50 college students (including the coauthors) from college exams, quizzes, textbooks, and lecture materials, and were then deduplicated and filtered for format, options, and overly easy items. The dataset is divided into a few-shot development set (150 questions, 5 per subject), a validation set (900 questions), and a test set (10,500 questions)[1]. The six disciplines are:

  • Art & Design (art, art theory, design, music).
  • Business (accounting, economics, finance, management, marketing).
  • Science (biology, chemistry, physics, mathematics, geography).
  • Health & Medicine (clinical medicine, pharmacy, public health, basic and diagnostic medical science).
  • Humanities & Social Science (history, literature, sociology, psychology).
  • Tech & Engineering (computer science, electronics, mechanical and materials engineering, energy)[1].

The defining feature of MMMU is the diversity of its visual material: the questions incorporate 30 highly heterogeneous image types, including charts, diagrams, maps, tables, sheet music, chemical structures, medical images, paintings, and geometric figures[1][2]. Roughly 97.5% of questions embed at least one image in the question stem, and about 7.4% include multiple images[1].

Most questions are multiple-choice — 10,861 (94.03%), each with four answer options — while the remaining 689 (5.97%) are open-ended. On the multiple-choice subset, random guessing yields roughly 25% accuracy. The authors label questions by difficulty as easy (28%), medium (45%), and hard (27%), and evaluate models primarily in a zero-shot setting without fine-tuning, using micro-averaged accuracy with rule-based answer extraction (invalid multiple-choice responses are treated as random selections)[1].

Results and Development

When MMMU was released in late 2023, even the strongest proprietary models scored far below human experts. On the validation set, GPT-4V(ision) reached about 56% and Google's Gemini Ultra about 59% (the rounded figures reported in the paper's abstract), while most open-source LMMs performed substantially lower[1]. As a reference, human-expert performance was later added to the official leaderboard in January 2024 — it was not part of the original paper — and is reported for three expert settings: 76.2% (worst), 82.6% (medium), and 88.6% (best)[3].

Progress on the original (four-option) MMMU was rapid. According to third-party aggregators, by early-to-mid 2026 the strongest models had reached roughly 85–87% on the validation set, approaching the best-expert mark of 88.6%; most of the top entries are reasoning ("thinking") models[4]. Such figures are not directly comparable across sources, because evaluation harnesses, prompting protocols (zero-shot versus chain-of-thought), and even the exact question set differ. For example, the Vals AI leaderboard reports scores near 88–89%, but its evaluation uses a four-option format of about 1,700 questions drawn from the MMMU-Pro dataset and compares them against the original-MMMU human baseline — a configuration that differs from both the original MMMU test set and the official MMMU-Pro benchmark[5].

This clustering of leading models near the best-expert mark led toward a gradual "saturation" of the original four-option format, which reduced its ability to differentiate frontier systems — the same trajectory previously seen with MMLU — and motivated the release of harder variants (see below). On the substantially harder official MMMU-Pro, the gap is larger: at release (late 2024) the models tested showed accuracy decreases of 16.8% to 26.9% relative to MMMU, but more recent frontier models have closed much of that gap. Google reported that Gemini 3 Pro scored 81.0% on MMMU-Pro (and 87.6% on the video-based Video-MMMU)[6], approaching the estimated best-human baseline for MMMU-Pro of about 85.4% — a figure that is itself an approximation rather than a direct measurement (see below). Some public leaderboards describe progress as approaching an asymptotic limit, but this is a leaderboard-specific interpretation rather than an established result, and it depends on which dataset variant and protocol are used[5].

A further caveat applies to recent results: the official MMMU test-set answers were released in February 2026, so results reported after that date should be interpreted with additional care regarding possible data contamination, especially for models trained or post-trained after the release[7].

Limitations and Criticism

Despite its widespread adoption as a standard multimodal evaluation, MMMU has several notable limitations.

Multiple-choice Format and Shortcuts

With only four answer options, a model can sometimes reach the correct answer through elimination or statistical shortcuts rather than genuine multimodal understanding, which can overestimate its true ability. A related issue is that some questions are answerable by text-only models without consulting the image at all, meaning that a portion of the benchmark does not strictly require visual perception. Both problems were among the primary motivations for the MMMU-Pro variant, which filters out text-only answerable questions and expands the option set[8].

Evaluation Methodology

Scores depend heavily on the evaluation protocol — zero-shot versus chain-of-thought prompting, the specific prompt wording, the scoring method, and the evaluation harness — as well as on which split (validation or test) is used. Results are therefore not directly comparable unless the protocol is fully specified. The MMMU-Pro authors report that chain-of-thought reasoning generally improves scores, whereas explicit OCR prompts have little effect, since capable models already extract embedded text without them[8].

Data Quality and Human Baseline

In the original paper, the authors manually analysed 150 sampled GPT-4V errors and found that perceptual mistakes dominated, but also identified a small share of annotation errors (about 2%), indicating some label noise in the set[1]. In addition, human-expert performance for the harder MMMU-Pro is not measured directly but approximated from the original MMMU evaluation data (with an estimated high-end accuracy of about 85.4%), which the authors acknowledge introduces potential bias into the reported human ceiling and complicates claims that models have reached "human-expert level"[8].

To address the limitations of the original MMMU and to extend it to new settings, several variants and related benchmarks have been created.

  • MMMU-Pro. A more robust and difficult version, introduced in September 2024 and published at ACL 2025. It is built through a three-step process: filtering out questions answerable by text-only models, augmenting the candidate options from four to ten, and adding a vision-only input setting in which the question text is embedded inside the image with no accompanying text prompt, forcing the model to read and reason simultaneously. The consolidated set contains 3,460 questions (1,730 standard-format and 1,730 vision-input), and the models tested at release showed accuracy decreases of 16.8% to 26.9% relative to the original MMMU[8].
  • Video-MMMU. A video-based benchmark led by Kairui Hu and colleagues (with several MMMU authors), introduced in January 2025, that evaluates knowledge acquisition from educational videos. It comprises 300 expert-level videos across the same six disciplines and assesses three cognitive stages — perception, comprehension, and adaptation. Using a knowledge-gain metric (Δknowledge, the improvement after watching a video rather than absolute accuracy), it revealed a large human–model gap: human experts improved by 33.1%, whereas the best model at the time (GPT-4o) improved by only 15.6%[9].
  • Uni-MMMU. A related benchmark introduced in October 2025 that jointly evaluates multimodal understanding and image generation across eight reasoning-centric domains, using verifiable intermediate reasoning steps and a reproducible scoring protocol for both textual and visual outputs[10].

Literature

  • Yue, X. et al. (2024). MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. CVPR 2024. arXiv:2311.16502.
  • Yue, X. et al. (2025). MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark. ACL 2025. arXiv:2409.02813.
  • Hu, K. et al. (2025). Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos. arXiv:2501.13826.
  • Zou, K. et al. (2025). Uni-MMMU: A Massive Multi-discipline Multimodal Unified Benchmark. arXiv:2510.13759.
  • Liang, P. et al. (2022). Holistic Evaluation of Language Models (HELM). arXiv:2211.09110.

References

  1. ↑ 1.00 1.01 1.02 1.03 1.04 1.05 1.06 1.07 1.08 1.09 Yue, X. et al. "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI". arXiv:2311.16502, 2023 (CVPR 2024, pp. 9556–9567). [1]
  2. ↑ The arXiv abstract and several dataset mirrors state 30 image types, while the official GitHub repository and the CVPR poster list 32, apparently reflecting a later revision of the taxonomy. [2]
  3. ↑ Human Expert performance was added to the MMMU leaderboard on 31 January 2024; the reported values are 76.2% (worst), 82.6% (medium), and 88.6% (best expert). MMMU project page. [3]
  4. ↑ "MMMU Leaderboard". LLM-Stats, 2026. [4]
  5. ↑ 5.0 5.1 Vals AI, "MMMU Pro" leaderboard, 2026: the page evaluates a four-option format of approximately 1,700 questions and benchmarks against the original MMMU human baseline (76.2–88.6%). [5]
  6. ↑ "A new era of intelligence with Gemini 3". Google, 2025. [6]
  7. ↑ MMMU-Benchmark, GitHub: "[2026-02-12] We have released the answers for the MMMU test set." [7]
  8. ↑ 8.0 8.1 8.2 8.3 Yue, X. et al. "MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark". arXiv:2409.02813, 2024 (ACL 2025). [8]
  9. ↑ Hu, K. et al. "Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos". arXiv:2501.13826, 2025. [9]
  10. ↑ Zou, K. et al. "Uni-MMMU: A Massive Multi-discipline Multimodal Unified Benchmark". arXiv:2510.13759, 2025. [10]
Retrieved from "https://systems-analysis.ru/eng/index.php?title=MMMU_Benchmark&oldid=2938"