MATH Benchmark
MATH (an acronym for Mathematics Aptitude Test of Heuristics) is a large dataset and benchmark for evaluating the mathematical reasoning and problem-solving skills of large language models (LLMs). The dataset was introduced in 2021 by a group of researchers led by Dan Hendrycks and contains 12,500 problems sourced from American high school mathematics competitions, such as the AMC 10, AMC 12, and AIME[1].
The problems cover a wide range of subjects (algebra, geometry, number theory, combinatorics, etc.) and are graded by difficulty level. Unlike standard textbook problems, they often require creative approaches and heuristic methods rather than the direct application of formulas. Each problem is accompanied by a complete step-by-step solution and a final answer, making MATH a valuable resource for both training and testing models[2].
Structure and Features of the Dataset
The MATH benchmark has several key features that make it a challenging and reliable evaluation tool.
Problem Format
All problems and solutions are presented in LaTeX format, and the Asymptote language is used to describe geometric diagrams. This allows all conditions, including images, to be represented in a text-based format that can be processed by a language model. Each problem is tagged with one of seven mathematical subjects and one of five difficulty levels[1].
Automated Evaluation
The final answers in the dataset are enclosed in the special `\boxed{...}` format and adhere to a strict standard (e.g., fractions are in their simplest form). This enables automated evaluation of models using the exact match metric, which eliminates subjectivity and ambiguity when checking results. A model must produce the strictly correct answer for the problem to be considered solved[1].
Problem Difficulty and Human Performance
MATH is one of the most challenging mathematical tests for AI. The problems are difficult even for individuals with strong mathematical backgrounds.
- In the study accompanying the dataset, a group of university students was tested, with scores ranging from ~40% to ~90% for Olympiad winners.
- Even a three-time gold medalist of the International Mathematical Olympiad could not solve all the problems without errors[1].
This demonstrates that successfully solving MATH problems requires not only knowledge but also high precision and mathematical intuition.
Model Results and Progress in Problem Solving
Initial Results (2021)
When the benchmark was launched in 2021, even the largest models achieved extremely low scores.
- The GPT-3 model (175 billion parameters) was only able to solve about 5% of the problems correctly.
- Fine-tuned versions of GPT-2 showed an accuracy of 6–7%[1].
The authors concluded that simply scaling up models had little effect on performance and that new algorithmic approaches were needed for progress[3].
Breakthroughs with Minerva and GPT-4 (2022–2023)
A breakthrough occurred with the advent of models specifically trained on scientific texts and new problem-solving methods.
- In 2022, the Google Minerva model achieved an accuracy of about 50%, demonstrating that a combination of scale and specialized training could drastically improve solution quality[3].
- In 2023, OpenAI's GPT-4 showed another leap forward. By using tools, the model was able to significantly improve its results:
- With Code Interpreter (executing code to verify calculations), its accuracy reached nearly 70%.
- Using a code-based self-verification method (self-checking and correcting errors with code), it set a record of 84.3% of problems solved[4].
This result is comparable to the performance of strong human competitors and approaches an expert threshold.
Significance and Impact
The MATH benchmark has played a key role in the development of LLMs' mathematical abilities. It clearly demonstrated that solving complex problems requires more than simple scaling, necessitating new approaches such as:
- Training on complete step-by-step solutions.
- Specialized training on scientific data.
- Using external tools for calculation and verification.
Despite significant progress, MATH remains an important and difficult challenge. It continues to serve as an indicator of the level of mathematical reasoning in LLMs and stimulates research into robustly solving problems that require multi-step reasoning[1].
Links
Literature
- Liang, P. et al. (2022). Holistic Evaluation of Language Models (HELM). arXiv:2211.09110.
- Chang, Y. et al. (2023). A Survey on Evaluation of Large Language Models. arXiv:2307.03109.
- Ni, S. et al. (2025). A Survey on Large Language Model Benchmarks. arXiv:2508.15361.
- Biderman, S. et al. (2024). The Language Model Evaluation Harness (lm-eval): Guidance and Lessons Learned. arXiv:2405.14782.
- Kiela, D. et al. (2021). Dynabench: Rethinking Benchmarking in NLP. arXiv:2104.14337.
- Ma, Z. et al. (2021). Dynaboard: An Evaluation‑As‑A‑Service Platform for Holistic Next‑Generation Benchmarking. arXiv:2106.06052.
- Goel, K. et al. (2021). Robustness Gym: Unifying the NLP Evaluation Landscape. arXiv:2101.04840.
- Xu, C. et al. (2024). Benchmark Data Contamination of Large Language Models: A Survey. arXiv:2406.04244.
- Liu, S. et al. (2025). A Comprehensive Survey on Safety Evaluation of LLMs. arXiv:2506.11094.
- Chiang, W.-L. et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv:2403.04132.
- Boubdir, M. et al. (2023). Elo Uncovered: Robustness and Best Practices in Language Model Evaluation. arXiv:2311.17295.
- Huang, L. et al. (2023). A Survey on Hallucination in Large Language Models. arXiv:2311.05232.
Notes
- ↑ 1.0 1.1 1.2 1.3 1.4 1.5 Hendrycks, D., et al. "Measuring Mathematical Problem Solving With the MATH Dataset." arXiv:2103.03874. [1]
- ↑ "AI Benchmarks and Datasets for LLM Evaluation." arXiv:2412.01020. [2]
- ↑ 3.0 3.1 "Language models surprised us." Planned-Obsolescence.org. [3]
- ↑ "GPT-4 Code Interpreter smashes maths benchmarks, hits new SOTA." The Decoder. [4]