MATH-500

MATH-500 is a compact benchmark designed to evaluate the mathematical problem-solving and multi-step reasoning capabilities of large language models (LLMs). It is a fixed subset of 500 free-response problems selected from the test split of the MATH dataset. The subset was introduced in OpenAI's 2023 research paper Let's Verify Step by Step by Hunter Lightman and colleagues, which was later published at ICLR 2024.

The original paper described the collection as a representative subset of the MATH test set rather than giving it a separate formal benchmark name. It subsequently became widely known as MATH-500 and was released as a standalone dataset through the OpenAI PRM800K repository and public dataset mirrors.

Unlike multiple-choice benchmarks, MATH-500 requires a model to produce a numerical, algebraic, or symbolic answer, usually after generating a step-by-step solution. It is commonly used in research on chain-of-thought reasoning, process reward models, reinforcement learning with verifiable rewards, model distillation, and test-time computation.

Origin and Construction

MATH-500 is derived from the original MATH dataset, formally named the Mathematics Aptitude Test of Heuristics. Introduced by Dan Hendrycks and colleagues in 2021, MATH contains 12,500 competition mathematics problems, divided into 7,500 training problems and 5,000 test problems. The problems were collected from competitions such as AMC 10, AMC 12, AIME, and related sources, and each item includes a full reference solution and a final answer.

During the research that produced the PRM800K process-supervision dataset, the OpenAI researchers needed a larger and more diverse set of training problems. Repeatedly collecting labels for solutions to only the original 7,500 training problems risked overfitting. The researchers therefore adopted a nonstandard split:

• The original 7,500 MATH training problems were retained.
• 4,500 problems from the original MATH test set were added to the training pool.
• The remaining 500 test problems were held out for evaluation.

The 500 evaluation problems were selected uniformly at random. The authors compared their subject and difficulty distributions with those of the complete 5,000-problem MATH test set and concluded that the smaller subset was broadly representative of the original test distribution.

Despite its name, MATH-500 is therefore not a collection of the 500 hardest MATH problems. It contains problems from all five MATH difficulty levels. It should not be confused with MATH-Hard or MATH Level 5, which are difficulty-filtered versions that retain only the hardest category of problems.

Content and Format

MATH-500 contains problems written in English and mathematical notation. Problems and reference solutions use LaTeX, while some geometry items inherited from MATH may include diagrams encoded in the Asymptote vector-graphics language.

The benchmark covers the same seven subject areas as the full MATH dataset:

• Prealgebra.
• Algebra.
• Number Theory.
• Counting and Probability.
• Geometry.
• Intermediate Algebra.
• Precalculus.

Each problem also has a difficulty label from Level 1 to Level 5. The levels are relative to the subject: Level 1 represents the easiest problems within a subject, while Level 5 represents the hardest.

The commonly used Hugging Face distribution exposes a single 500-row test split with the following fields:

Field	Description
problem	The natural-language and LaTeX problem statement.
solution	A complete reference solution containing intermediate reasoning.
answer	The normalized final answer extracted from the reference solution.
subject	One of the seven mathematical subject categories.
level	A difficulty rating from 1 to 5.
unique_id	A path-like identifier inherited from the original MATH dataset.

Because the benchmark contains exactly 500 problems, each correctly or incorrectly graded item changes the aggregate score by 0.2 percentage points.

Evaluation Methodology

Final-Answer Evaluation

The standard task is to present a model with a problem and ask it to generate a solution followed by a final answer. Prompts frequently instruct the model to place the final result inside a LaTeX expression such as \boxed{...}.

A typical evaluation pipeline performs the following operations:

1. Generate one or more model responses for each problem.
2. Extract the final answer, commonly from the last \boxed{...} expression.
3. Normalize differences in whitespace, LaTeX commands, fractions, units, signs, and equivalent formatting.
4. Compare the extracted answer with the reference answer.
5. Report the proportion of correctly answered problems.

The primary metric is usually called accuracy or pass@1. In a deterministic evaluation, it is the percentage of problems solved by one response per problem. In sampling-based evaluations, pass@1 may instead be estimated from several stochastic responses generated for every problem.

The original MATH evaluation used normalized exact matching. Later implementations added symbolic checks, including expression comparison with SymPy. OpenAI's released grader is intentionally conservative and may reject some mathematically correct answers; the repository also acknowledges that an imperfect parser can occasionally accept an incorrect answer.

Test-Time Search and Verification

MATH-500 is also frequently used to evaluate methods that generate multiple candidate solutions and spend additional computation selecting an answer. Common protocols include:

• Majority voting or consensus, in which the most frequent final answer among several generations is selected.
• Best-of-N selection, in which a reward model or verifier ranks N candidate solutions.
• Process-reward selection, in which individual reasoning steps are scored and combined into a solution-level score.
• Pass@k, which measures whether at least one of k generated solutions is correct.
• Tool-assisted evaluation using a Python interpreter or computer algebra system.

In Let's Verify Step by Step, the generator produced up to 1,860 candidate solutions per problem. A process reward model ranked these candidates, and the highest-ranked solution was automatically graded. The resulting system solved 78% of the 500 problems. This result was a best-of-1,860 search score and is not directly comparable with single-response accuracy.

Evaluation reports should therefore specify the prompt, model version, sampling temperature, number of generations, token budget, answer parser, tool access, and candidate-selection method. Scores obtained under different protocols may measure substantially different combinations of base-model capability and test-time computation.

Research Use and Development

MATH-500 was initially used to compare process supervision with outcome supervision. Outcome-supervised reward models receive a label for an entire solution based primarily on its final result. Process-supervised reward models instead receive correctness labels for individual reasoning steps.

The associated PRM800K dataset contains approximately 800,000 step-level labels across 75,000 model-generated solutions to about 12,000 MATH problems. Human annotators marked steps as positive, negative, or neutral. The study found that the process-supervised reward model was substantially more effective than the outcome-supervised alternative at selecting correct solutions from large candidate pools.

The benchmark later became a standard evaluation for reasoning-focused models, mathematical model distillation, reinforcement learning, and inference-time search. Selected published results illustrate both rapid progress and the difficulty of comparing protocols:

Selected published results on the MATH-500 split

Year	System	Reported score	Evaluation note
2023	OpenAI generator with a process reward model	78.0%	Best-of-1,860 candidate selection; not a single-sample result.
2024	GPT-4o	60.3%	Pass@1 in OpenAI's zero-shot chain-of-thought evaluation.
2024	OpenAI o1	94.8%	Pass@1 with the model's high test-time-compute setting; OpenAI used the same 500-problem split.
2025	DeepSeek-R1	97.3%	Reported pass@1 estimate; 64 sampled responses per problem were used to estimate the metric.

These figures are not a strict leaderboard. They were produced with different model versions, sampling procedures, prompts, token budgets, and verification systems.

By September 2024, OpenAI stated that recent frontier models were performing so well on MATH and GSM8K that the benchmarks were becoming ineffective for differentiating leading systems. The MATH result in that report used the same 500-problem split later commonly labelled MATH-500. This saturation encouraged greater use of more difficult or newly released competition problems, including AIME, Olympiad-level benchmarks, and dynamically refreshed evaluations.

Limitations and Criticism

Public Test Set and Data Contamination

MATH-500 is a public and static test set derived from competition problems that have circulated online for many years. The authors of Let's Verify Step by Step acknowledged that some MATH problems probably appeared in the pre-training data of their models. Although they attempted to remove MATH problems from their additional MathMix training corpus using string matching, they could not rule out paraphrases or other less obvious forms of overlap.

A 2025 study reported that some Qwen2.5-family models could reconstruct substantial portions of MATH-500 questions when given only partial problem statements. The authors interpreted this behaviour as evidence that results on public mathematics benchmarks can be inflated by memorization or training-data contamination. Such findings are model-specific and do not establish the amount of contamination in every evaluated system, but they demonstrate why benchmark provenance must be considered when interpreting high scores.

Final Answers Do Not Verify Reasoning

Standard MATH-500 accuracy checks the final answer rather than the logical validity of the complete solution. A model may receive credit after using incorrect reasoning that happens to reach the correct result. Conversely, a correct derivation may be marked wrong if its final expression is not parsed or normalized successfully.

This distinction was central to the original process-supervision research: final-answer checking can produce false-positive labels for solutions containing reasoning errors, whereas step-level evaluation can identify where an otherwise plausible solution first becomes invalid.

Protocol Dependence

Reported performance can vary according to:

• Whether generation is greedy or sampled.
• Whether the prompt requests chain-of-thought reasoning.
• The number of candidate solutions generated.
• The use of majority voting, reward models, or symbolic verifiers.
• The maximum reasoning-token budget.
• Access to Python, calculators, search, or computer algebra systems.
• The answer-extraction and normalization implementation.
• Whether embedded Asymptote diagrams are preserved as source code, removed, or rendered.

Consequently, two evaluations described only as “MATH-500 accuracy” may not be directly comparable.

Size, Scope, and Saturation

With only 500 problems, the benchmark has limited statistical resolution. A difference of one solved problem changes the score by 0.2 percentage points, and small score differences can fall within ordinary sampling variation.

The benchmark also measures a restricted domain: English-language, high-school competition mathematics across seven subject categories. It does not comprehensively test formal proof construction, advanced university mathematics, research mathematics, real-world quantitative modelling, or interactive theorem proving.

Finally, near-ceiling scores reduce the benchmark's ability to distinguish the strongest reasoning models. A model scoring 97% fails only about 15 problems, leaving relatively little room for measuring further improvement or diagnosing capabilities by subject and difficulty.

External links

• Official PRM800K repository and MATH-500 split
• MATH-500 dataset on Hugging Face
• Let's Verify Step by Step
• Official repository for the original MATH dataset

See also

• GSM8K

Literature

• Hendrycks, D. et al. (2021). Measuring Mathematical Problem Solving With the MATH Dataset. NeurIPS. arXiv:2103.03874.
• Lightman, H. et al. (2024). Let's Verify Step by Step. ICLR 2024. arXiv:2305.20050.
• Uesato, J. et al. (2022). Solving Math Word Problems with Process- and Outcome-Based Feedback. arXiv:2211.14275.
• Snell, C. et al. (2024). Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters. arXiv:2408.03314.
• DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948.
• Wu, M. et al. (2025). Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination. arXiv:2507.10532.