MMLU Benchmark

MMLU (Massive Multitask Language Understanding) is a benchmark test designed to evaluate the capabilities of large language models (LLMs) across a wide range of subject areas. It was introduced in the 2020 paper Measuring Massive Multitask Language Understanding by a team of researchers led by Dan Hendrycks of UC Berkeley, and was published at the ICLR conference in 2021.

The goal of MMLU is to test how well a model has absorbed the diverse knowledge and skills acquired during its pre-training phase by evaluating it in a zero-shot or few-shot setting without additional fine-tuning. MMLU was created as a more challenging alternative to earlier tests (such as GLUE and SuperGLUE), on which state-of-the-art models had rapidly approached or surpassed human baselines.

Description and Content

MMLU consists of 15,908 multiple-choice questions covering 57 different disciplines. The dataset is divided into a few-shot development set (5 questions per subject), a validation set of 1,540 questions, and a test set of 14,079 questions; models are evaluated on the test split. The subjects include:

• STEM subjects (mathematics, physics, biology, computer science, engineering).
• Humanities and social sciences (history, philosophy, law, economics).
• Applied and professional fields (medicine, business, accounting).

The difficulty ranges from elementary school level to advanced professional level. The questions are drawn from real exam materials from schools, universities, and professional tests, such as the GRE and USMLE. Each question has four possible answers, so random guessing yields an accuracy of 25%. To achieve a high score, a model must possess extensive encyclopedic knowledge and reasoning abilities.

Results and Development

When MMLU was released in 2020, most of the evaluated models performed at or near random-chance levels. Among them, GPT-3 (175B parameters) achieved the best result for a purely few-shot model, at ~43.9% (and 37.7% in the zero-shot setting); the highest overall score in the original paper belonged to UnifiedQA — a T5-based model fine-tuned on question-answering datasets and evaluated by transfer — at ~48.9%. Both remained far below the authors' estimate of expert-level human performance, which they placed at ~89.8% (based on the 95th percentile of human test-takers); non-expert human annotators scored only about 34.5%. This gap confirmed the difficulty and high standard of the new benchmark.

Progress on the benchmark was rapid. By March 2023, GPT-4 reached 86.4% in the standard 5-shot setting. In late 2023, Google's Gemini Ultra reported 90.04%, becoming the first model to exceed the authors' 89.8% human-expert estimate — although this figure used a chain-of-thought prompting protocol with self-consistency (CoT@32) rather than the standard 5-shot setting, in which it scored 83.7%. By mid-2024, several leading models — including GPT-4o, Anthropic's Claude 3.5 Sonnet, and Llama 3.1 405B — were clustered in the high 80s (roughly 86–88%), though such figures are not always directly comparable, since evaluation harnesses and prompting protocols differ.

This clustering near the top of the scale led to a gradual "saturation" of the benchmark: leading models began to achieve near-maximum scores, which diminished MMLU's ability to differentiate between their capabilities. The authors of MMLU-Pro noted that scores had reached a plateau, with little significant progress after GPT-4's 86.4%. This spurred the community to develop new, more difficult tests.

Limitations and Criticism

Despite its widespread use, MMLU has several significant limitations.

Data Quality and Correctness

In 2024, researchers conducted a manual analysis of a sample of 5,700 MMLU questions and found a significant number of errors.

• Approximately 6.5% of the analysed questions contain errors in their labeling or wording.
• In certain categories, the proportion is very high. For example, in the "Virology" section, 57% of the analysed questions contained errors (multiple correct answers, unclear wording, or the wrong reference answer).

This means that even a perfect model cannot score 100% on the original dataset, and some of the apparent improvements in metrics may reflect models memorizing systematic errors in the set.

Evaluation Methodology and Data Leakage

• Sensitivity to the evaluation protocol. Scores depend heavily on how a model is evaluated — zero-shot, five-shot, or chain-of-thought — as well as on the specific prompt wording, scoring method, and evaluation harness. Results are therefore not directly comparable unless the protocol is specified, and the original MMLU is known to be particularly sensitive to prompt formatting.
• Data contamination. There is a risk that questions and answers from public benchmarks are included in the training datasets of LLMs. In such cases, a model effectively "knows" the correct answers, making the evaluation unfair.
• Multiple-choice format. With only four options, a model can sometimes reach the correct answer through elimination or shortcuts rather than genuine understanding, which can overestimate its true ability.

Derivative Versions and Extensions

To address the problems of the original MMLU, several variants have been created.

• MMLU-Redux. A corrected and re-annotated subset of the dataset, introduced in June 2024. The initial version contained 3,000 relabeled questions across 30 subjects; it was later expanded in MMLU-Redux 2.0 to 5,700 questions covering all 57 subjects. It is designed for more reliable evaluation, free of the distortions caused by data errors.
• MMLU-Pro. An expanded and more difficult version of the test, introduced in June 2024 and published at NeurIPS 2024. It contains over 12,000 questions consolidated into 14 broader disciplines, each with 10 answer choices instead of four. This reduces the probability of random guessing to 10% and lowers the benchmark's sensitivity to prompt formatting. The questions were expert-reviewed and include additional, more challenging tasks.
• MMMLU (Multilingual MMLU). A multilingual version released by OpenAI in September 2024. The MMLU test set was professionally translated by human translators into 14 languages, including widely spoken ones (Spanish, Chinese) and low-resource ones (e.g., Yoruba), allowing model capabilities to be evaluated and compared across languages; such multilingual evaluations typically reveal a marked drop in accuracy on non-English and especially low-resource languages. The release does not include Russian, and the original English test set is not part of the multilingual package.

External links

• Official MMLU repository on GitHub
• MMLU page on Papers with Code with model results
• MMLU — Wikipedia

See also

• Large language model
• GLUE Benchmark
• SuperGLUE (benchmark)
• Few-Shot and Zero-Shot Learning
• Fine-tuning (deep learning)
• Pre-training of large language models

Literature

• Liang, P. et al. (2022). Holistic Evaluation of Language Models (HELM). arXiv:2211.09110.
• Chang, Y. et al. (2023). A Survey on Evaluation of Large Language Models. arXiv:2307.03109.
• Ni, S. et al. (2025). A Survey on Large Language Model Benchmarks. arXiv:2508.15361.
• Biderman, S. et al. (2024). The Language Model Evaluation Harness (lm-eval): Guidance and Lessons Learned. arXiv:2405.14782.
• Kiela, D. et al. (2021). Dynabench: Rethinking Benchmarking in NLP. arXiv:2104.14337.
• Xu, C. et al. (2024). Benchmark Data Contamination of Large Language Models: A Survey. arXiv:2406.04244.
• Chiang, W.-L. et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv:2403.04132.