FrontierMath

FrontierMath is a benchmark program designed to evaluate advanced mathematical reasoning and research-level problem-solving capabilities in artificial intelligence systems. It was developed by the nonprofit research organization Epoch AI in collaboration with professional mathematicians and was introduced in the 2024 paper FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI by Elliot Glazer and colleagues.

Unlike mathematics benchmarks based primarily on school exercises, standardized tests, or previously published competition problems, FrontierMath was constructed from original problems written specifically for evaluating frontier AI models. Its questions range from advanced undergraduate and Olympiad-style mathematics to graduate-level material and contemporary research mathematics. Most problems have objectively checkable final answers but require substantial theoretical insight, extended symbolic reasoning, or specialized computation.

The name originally referred to a closed-answer benchmark containing problems with known solutions. By 2026, Epoch AI used FrontierMath as the name of a broader program with two principal components:

• FrontierMath: Tiers 1–4, a predominantly private benchmark of difficult but solved mathematical problems.
• FrontierMath: Open Problems, a dynamic collection of genuinely unsolved research questions whose solutions are evaluated by specialist mathematicians and custom verifiers.

On 12 June 2026, Epoch AI released version 2 of the Tiers 1–4 benchmark. The revised collection contains 338 problems: 295 in Tiers 1–3 and 43 in Tier 4. Twelve problems are public, comprising ten from Tiers 1–3 and two from Tier 4; the principal benchmark scores are calculated on private problems.

Background and Motivation

FrontierMath was created in response to the rapid saturation of earlier mathematical benchmarks. Datasets such as GSM8K, MATH, and standardized examination collections were useful for measuring arithmetic, algebra, and competition-style reasoning, but frontier models increasingly achieved high or near-ceiling results on them. Many such datasets were also publicly accessible for years, creating a risk that their questions or solutions had entered model-training corpora.

Epoch AI identified several limitations in existing mathematical evaluations:

• Many benchmarks concentrate on primary-school, secondary-school, or early undergraduate mathematics.
• Publicly available questions may be memorized or indirectly learned during pre-training.
• Multiple-choice questions can sometimes be answered through elimination or statistical shortcuts.
• Short exercises may not require the sustained reasoning characteristic of advanced mathematics.
• A high aggregate score may provide little information once most frontier models approach the top of the scale.

FrontierMath was therefore designed around new problems whose statements and solutions had not previously circulated online. Most of the benchmark was kept private to reduce data contamination and to preserve its usefulness as a holdout evaluation.

The project also sought to test mathematical abilities that are difficult to capture with ordinary competition problems. These include identifying an appropriate advanced theorem, combining techniques from several fields, deriving a previously unstated formula, constructing and testing conjectures computationally, and carrying out long chains of exact symbolic calculation.

Development and Version History

The benchmark was developed incrementally. The technical paper analyzed an early 119-problem version, while additional questions and the separate Tier 4 expansion were released later.

Principal FrontierMath releases

Date or identifier	Size	Description
FrontierMath_10-22-24	119 problems	Early collection analyzed in the original technical paper.
6 November 2024	147 problems	Expanded prerelease evaluation set.
26 November 2024	180 problems	Version later used for OpenAI's reported o3 evaluation.
4 December 2024	197 problems	Included five public examples; one duplicate was subsequently identified.
28 February 2025	300 problems	Completion of the original Tiers 1–3 collection.
30 June 2025	50 Tier 4 problems	Completion of the original exceptionally difficult Tier 4 expansion.
12 June 2026	338 problems	Version 2: 295 Tiers 1–3 problems and 43 Tier 4 problems after corrections and removals.

The original completed benchmark therefore contained 350 problems: 300 in the base collection and 50 in Tier 4. During preparation of version 2, Epoch AI corrected 123 Tiers 1–3 problems and 12 Tier 4 problems. It also removed five problems from Tiers 1–3 and seven from Tier 4. The organization described the update as addressing errors in 42% of the original 350 problems.

Because version 2 modifies both problem statements and reference evaluation procedures, scores obtained on earlier releases are not necessarily directly comparable with scores on the revised collection.

Problem Authorship and Construction

More than 60 mathematicians from over a dozen countries contributed to the original benchmark. Contributors included graduate students, postdoctoral researchers, faculty members, mathematical competition specialists, and research mathematicians. Collectively, they had won 14 International Mathematical Olympiad gold medals, and one contributor was a Fields Medal recipient.

Authors were asked to create problems satisfying several design requirements:

Requirement	Purpose
Originality	The problem should be newly created rather than copied or lightly modified from a published source.
Definite answer	The problem must have a well-defined answer that can be evaluated consistently.
Automated verifiability	Correctness should be checkable through exact matching, symbolic algebra, or a custom verification program.
Resistance to guessing	The answer space should be sufficiently large or structured so that a model cannot obtain substantial credit through random guessing.
Computational tractability	Verification should be possible using ordinary computational resources rather than an impractically large search.
Substantive difficulty	Solving the problem should require advanced mathematical knowledge, creativity, sustained reasoning, or nontrivial computation.

A heuristic used during the original construction process was that a solver who had not performed most of the required mathematical work should have less than approximately a 1% chance of guessing the correct answer. Authors were also generally expected to provide verification scripts that could run in less than one minute on standard hardware.

Each submission included:

• A formal problem statement.
• The expected final answer or class of acceptable answers.
• A detailed mathematical solution.
• A verification program where appropriate.
• Subject and technique metadata.
• Estimates of the background knowledge, creative insight, and execution time required.
• Information about possible ambiguities and alternative solution methods.

Problems underwent blind peer review by another mathematician. Reviewers checked correctness, clarity, originality, answer uniqueness, computational feasibility, and the validity of the reference verifier. Authors then revised their submissions in response to review comments.

A second-review study described in the technical report nevertheless found quality issues in a sample of 35 problems, including two incorrect reference answers, six cases with missing assumptions, and two problems considered excessively guessable. These findings, together with the extensive 2026 corrections, illustrate the difficulty of quality assurance for expert-level mathematical benchmarks.

Mathematical Scope

FrontierMath covers a broad selection of pure and applied mathematical fields, including:

• Number theory.
• Combinatorics and graph theory.
• Group theory and representation theory.
• Linear and abstract algebra.
• Algebraic geometry.
• Differential geometry and topology.
• Real, complex, and functional analysis.
• Ordinary and partial differential equations.
• Probability theory.
• Mathematical logic.
• Category theory.
• Special functions.
• Numerical and computational mathematics.
• Theoretical computer science.

The version analyzed in the technical report included material from approximately 70% of the top-level categories of the MSC2020 classification, excluding general, historical, and educational categories. The authors identified more than 200 distinct mathematical techniques across the collection.

Number theory and combinatorics were disproportionately represented, together accounting for approximately 34% of the MSC subject tags in the analyzed version. These areas are particularly suitable for automatically verifiable questions because many difficult problems have exact integer, finite, algebraic, or combinatorial answers.

The benchmark nevertheless includes substantially more specialized subject matter than most earlier LLM mathematics evaluations. Problems may involve topics such as arithmetic geometry, spectral sequences, stochastic processes, algebraic topology, analytic number theory, or representation-theoretic calculations.

Difficulty Tiers

The original Tiers 1–3 collection was designed around three approximate difficulty levels. Tier 4 was subsequently added as a separate expansion containing more research-intensive problems.

Tier	Approximate level	Intended human difficulty	Typical characteristics
Tier 1	Advanced undergraduate or difficult mathematical competition level	Several hours for a strong mathematician	May require advanced coursework, extensive calculation, computer experimentation, or access to reference material. The original target was approximately 20% of the Tiers 1–3 collection.
Tier 2	Graduate level	Approximately one day or longer for a relevant specialist	Usually requires specialized knowledge, a non-obvious insight, multistage derivation, or substantial exact computation. The original target was approximately 40%.
Tier 3	Early research level	Several days for a specialist or advanced doctoral student	Resembles a small exploratory research project and may require developing a new argument or computational method. The original target was approximately 40%.
Tier 4	Exceptional research level	Several days to several weeks for professional researchers	Problems were generally created by professors and postdoctoral researchers through research projects specifically organized for the benchmark.

The proportions for Tiers 1–3 describe the historical design target rather than the precise composition of version 2, which is publicly reported as a combined 295-problem set.

The 50-problem Tier 4 collection was completed in June 2025. Its development included a FrontierMath symposium attended by approximately 30 mathematicians, with problem-writing activity in analysis, algebraic geometry, combinatorics, number theory, and topology. Version 2 retains 43 of these problems after corrections and removals.

Difficulty metadata originally included three partly independent dimensions:

• Background, scored from 1 to 5, ranging from high-school mathematics to active research-level knowledge.
• Creativity, estimating the time a suitable expert would require to discover the central idea.
• Execution, estimating the additional work required to complete the proof or computation after the main idea was identified.

These ratings are expert judgments rather than experimentally calibrated measurements. A problem may require extensive specialist background but little creativity once an appropriate theorem is known, or it may use elementary concepts while requiring a highly original construction.

Human Performance

FrontierMath is broad enough that no individual mathematician is expected to be equally familiar with every represented field. A specialist in algebraic geometry may be poorly positioned to solve a difficult probability problem, while a combinatorialist may lack the background required for a question in functional analysis.

Epoch AI organized a human evaluation involving approximately 40 strong mathematics students and experts at the Massachusetts Institute of Technology. Participants were divided into eight teams of four or five and were allowed Internet access. Over 4.5 hours, the teams attempted a selection of 23 Tiers 1–3 problems.

The average team solved approximately 19% of the assigned problems. When the successful answers of all teams were combined, at least one team solved approximately 35% of the set. This experiment was not intended as a universal human baseline: it used a small problem sample, teamwork, limited time, and participants whose specializations did not cover the entire benchmark.

Mathematicians including Terence Tao, Timothy Gowers, and Richard Borcherds also reviewed examples from the benchmark and described the upper-tier questions as unusually challenging. Such qualitative assessments provide evidence about difficulty but are not directly comparable with model accuracy.

Answer Types and Verification

FrontierMath uses free-response questions rather than multiple-choice answers. Depending on the problem, the expected output may be:

• An integer or rational number.
• A real or complex algebraic expression.
• A polynomial.
• A tuple or finite sequence.
• A matrix.
• A finite set.
• A combinatorial object.
• A symbolic object represented in SymPy.
• Any object satisfying a problem-specific mathematical predicate.

Unique integer answers can be graded through exact comparison. Symbolic expressions may be simplified and compared using a computer algebra system. Problems with multiple valid answers require custom verifier programs that test whether the submitted object satisfies the stated constraints.

For example, a construction problem might ask for a graph with specified properties. Rather than comparing the graph with a single reference object, the verifier can check its number of vertices, adjacency structure, forbidden subgraphs, and required invariant.

Automated verification distinguishes FrontierMath from evaluations requiring a human to read every generated proof. It enables repeated testing of many model configurations, but it generally checks the submitted mathematical object rather than the validity of every intermediate reasoning step.

Evaluation Methodology

Current Tiers 1–4 Protocol

Under the version 2 evaluation protocol, a model receives a problem statement together with the expected answer type. It may generate an extended reasoning trace and use a stateless Python execution tool. When ready, the model must submit the source code of a zero-argument function named answer:

def answer():
    return expected_mathematical_object

The function must return the requested Python object without printing output. The grading system runs the function and passes its return value to the problem-specific verifier.

The current execution environment includes the Python standard library and mathematical packages such as:

• galois.
• gmpy2.
• mpmath.
• networkx.
• numpy.
• pyadic.
• scipy.
• sympy.

Python calls are stateless, so values created in one call are not automatically available in the next. Individual Python executions and submitted answer functions are limited to 30 seconds on the reference infrastructure.

The published v2 protocol permits a maximum context of one million tokens. If a run reaches 660,000 tokens without an answer, the evaluator requires the model's next action to be an attempted submission. This large allowance is intended to support long-form reasoning, repeated experimentation, and substantial test-time computation rather than ordinary short-form question answering.

Scoring

Each problem receives a binary score:

$\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}(q)=\{\begin{array}{ll}1,& \text{if the submitted answer passes the verifier},\\ 0,& \text{otherwise}.\end{array}$

Aggregate accuracy is calculated as:

$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}=\frac{\text{number of correctly solved problems}}{\text{number of evaluated problems}}.$

Scores for Tiers 1–3 and Tier 4 are normally reported separately because the two collections have different difficulty profiles. Epoch AI's benchmark hub generally reports results on private problems unless a public-only evaluation is explicitly identified.

Because version 2 contains 295 Tiers 1–3 problems, one item changes the full-set accuracy by approximately 0.34 percentage points. In the 43-problem Tier 4 set, one item changes accuracy by approximately 2.33 percentage points, resulting in much wider statistical uncertainty.

Protocol Dependence

FrontierMath performance depends not only on the underlying model but also on the evaluation configuration. Relevant variables include:

• Model version and reasoning-effort setting.
• Maximum token allowance.
• Sampling temperature and number of attempts.
• Access to Python and installed libraries.
• Whether multiple candidate answers are generated.
• Use of majority voting, reward models, or external verifiers.
• Time limits for model reasoning and code execution.
• Handling of malformed submissions and tool failures.
• Benchmark version and the exact private subset used.

A result should therefore specify whether it was obtained on version 1 or version 2, which tier collection was used, and what tools and inference-time resources were available.

Model Results and Progress

The original technical report evaluated six contemporary systems, including OpenAI o1-preview, o1-mini, GPT-4o, Claude 3.5 Sonnet, Grok 2 Beta, and Gemini 1.5 Pro 002. Every evaluated system solved less than 2% of the early 119-problem collection.

Performance increased rapidly after the benchmark's introduction. In December 2024, OpenAI reported that its o3 system solved 25.2% of a 180-problem FrontierMath version. The result was substantially higher than the initial evaluations, although it used a different dataset version and inference configuration.

By 2026, reasoning-focused systems were solving a large majority of the revised benchmark. Selected published results are shown below.

Selected FrontierMath results

Date	System	Benchmark version	Tiers 1–3	Tier 4	Notes
2024	Six systems evaluated in the technical report	Early 119-problem version	All below 2%		Included o1-preview, o1-mini, GPT-4o, Claude 3.5 Sonnet, Grok 2 Beta, and Gemini 1.5 Pro 002.
December 2024	OpenAI o3	180-problem prerelease version	25.2% overall		Result reported by OpenAI; the later tier split was not applicable to this aggregate figure.
July 2026	GPT-5.6 Sol	Version 2	89.0%	83.0%	Results reported by OpenAI under a high-compute reasoning configuration.
July 2026	Claude Fable 5	Version 2	87.0%	87.8%	Best result across settings reported by Epoch AI; the Tier 4 estimate had an uncertainty interval of approximately ±5.2 percentage points.

The table is not a strictly controlled leaderboard. The 2024 and 2026 figures use different problem sets, model interfaces, token limits, and inference budgets. Even two results on version 2 may differ in reasoning effort, sampling procedure, or tool use.

Nevertheless, the increase from less than 2% on the early collection to scores in the high 80s on version 2 represents unusually rapid progress. It also means that Tiers 1–4 may provide decreasing resolution for differentiating the strongest systems, although substantial uncertainty remains on individual problems and on performance outside the automatically verifiable answer format.

FrontierMath: Open Problems

FrontierMath: Open Problems is a separate, dynamic benchmark based on mathematical questions that were unsolved when admitted to the collection. Epoch AI announced the pilot program on 27 January 2026 with support from Schmidt Sciences.

This component differs fundamentally from Tiers 1–4:

Feature	Tiers 1–4	Open Problems
Status at inclusion	The author and reviewers know a solution.	No accepted solution is known.
Reference answer	A hidden answer or custom verifier is available.	A verifier tests a proposed construction or consequence, but no reference solution is assumed.
Stability	Versioned static collection.	Dynamic collection from which solved or unsuitable problems may be removed.
Intended difficulty	Hours, days, or occasionally weeks of expert work.	Potentially publishable research contributions or major mathematical advances.
Evaluation	Mostly automatic binary grading.	Automated verification followed by expert mathematical review where required.

Open Problems accepts questions from active researchers. A candidate problem must be sufficiently precise for evaluation and must admit a proposed answer that can be checked by a custom verifier. This excludes many important open questions asking for unrestricted proofs, conceptual classifications, or existence results without a finite certificate.

Each problem is assigned an approximate notability level:

• Moderately interesting: a useful result that could contribute to a specialized paper.
• Solid result: a result likely to form a substantial part of a publishable research work.
• Major advance: a significant development within an established research area.
• Breakthrough: a result with broad or potentially transformative mathematical importance.

As of 10 July 2026, the official benchmark page listed 15 active problems.

Active Open Problems by notability level, 10 July 2026

Notability level	Listed problems	Marked as solved by AI
Moderately interesting	4	1
Solid result	5	0
Major advance	3	0
Breakthrough	3	0
Total	15	1

Topics represented on the page included Hadamard matrices, Ramsey theory, arithmetic Kakeya problems, Boolean-function complexity, del Pezzo surfaces in positive characteristic, Steiner systems, absolute Galois groups, the inverse Galois problem, Littlewood–Richardson coefficients, symplectic packing, Apéry-style irrationality proofs, integer factorization, and knot theory.

The collection changes over time. Two problems were added in February 2026, and one was removed in March after a proposed AI-generated solution did not meet the project's publishability standard. Epoch AI also changed the public interface in June 2026 so that unsuccessful model attempts were no longer displayed by default.

Custom verifiers for Open Problems are not generally released without restriction. Epoch AI offers access to them for a fee intended to support benchmark maintenance and expansion. As of 10 July 2026, the organization stated that OpenAI was the only entity that had purchased verifier access. Epoch AI described Open Problems as independently developed and owned by Epoch AI rather than by OpenAI.

Funding, Access, and Governance

The original Tiers 1–3 benchmark was financed through a partnership between Epoch AI and OpenAI. In January 2025, Epoch AI clarified that OpenAI had commissioned the organization to produce 300 advanced mathematics problems and had ownership of and access to the associated statements and solutions, except for a holdout set of 50 questions.

For the original 50-problem Tier 4 expansion, OpenAI received access to 30 problems, while 20 were retained as a holdout. Epoch AI retained the right to conduct evaluations and publish aggregate benchmark results. The version 2 benchmark pages state more generally that OpenAI funded FrontierMath and has exclusive access to a subset of the collection.

Epoch AI issued the January 2025 clarification after criticism about disclosure of the commercial relationship. The organization acknowledged that its communication had not been sufficiently clear and that some problem contributors had not been systematically informed of all sponsorship and access arrangements before participating.

The arrangement creates two competing considerations:

• Private sponsor access can support the substantial cost of authoring, reviewing, and maintaining research-level evaluations.
• Exclusive or early access can give the sponsor information unavailable to other model developers and independent researchers.

Holdout problems reduce the risk that a sponsor directly trains on the complete benchmark, but they do not remove the broader reproducibility and governance questions associated with private evaluations.

FrontierMath: Open Problems uses a different structure. It was launched with support from Schmidt Sciences, is described as independently owned by Epoch AI, and finances part of its operation through paid verifier access.

Limitations and Criticism

Private Data and Reproducibility

Keeping most questions private reduces direct benchmark contamination, but it prevents independent researchers from fully inspecting the data, reproducing grading decisions, testing alternative evaluation harnesses, or assessing whether a model has encountered particular problems during development.

The public subset contains only 12 of the 338 version 2 problems. Public examples illustrate the general style and evaluation interface but may not be statistically representative of the complete private benchmark.

The sponsor-access arrangement creates an additional asymmetry. A model developer with access to some private problems can use them for internal analysis, prompt development, error investigation, or evaluation-system optimization, even when the problems are not explicitly used as training examples.

Data Quality and Version Instability

Creating correct research-level questions is difficult even for professional mathematicians. A problem can contain an unstated assumption, ambiguous notation, an incorrect reference answer, a verifier bug, an unexpectedly simple shortcut, or more than one mathematically valid interpretation.

The June 2026 revision addressed issues in 42% of the original collection, correcting 135 problems and removing 12. The figure does not mean that every affected item had an incorrect mathematical answer; revisions may also concern wording, assumptions, formatting, verifier behavior, or grading robustness. It nevertheless shows that the first version should not be treated as a permanently fixed gold standard.

Substantial revisions also complicate historical comparisons. A model's version 1 score cannot be converted reliably into a version 2 score without rerunning the evaluation.

Final-Answer Verification

The standard benchmark evaluates the submitted object rather than the complete reasoning trace. A model may receive full credit after producing flawed, circular, or unverifiable reasoning if its final object passes the checker. Conversely, a mathematically valid derivation may receive no credit because of a formatting error, unsupported Python type, timeout, or mismatch with the verifier's expected representation.

Custom verifiers can also contain implementation errors or accept unintended constructions. A verifier establishes compliance with its programmed conditions, which may not perfectly capture the intended mathematical claim.

FrontierMath therefore primarily measures the ability to produce a verifiable final answer under a specified computational interface. It does not directly measure proof quality, explanatory clarity, originality, or the reliability of every intermediate statement.

Restricted Form of Research Mathematics

Many important mathematical problems do not have short, automatically verifiable outputs. Research may require proving a general theorem, developing a conceptual framework, classifying a family of structures, resolving ambiguities in definitions, or writing a long argument whose correctness depends on human judgment.

The closed-answer requirement favors areas in which difficult questions can be reduced to integers, explicit symbolic expressions, finite constructions, or computational certificates. This partly explains the comparatively high representation of number theory and combinatorics.

Even Tier 4 problems are shorter and more bounded than many real research projects, which can require months or years of literature review, failed approaches, collaboration, and conceptual development.

Approximate Difficulty Labels

Tier assignments and estimates of human work time are based on expert judgment. They depend on the solver's specialization, knowledge of relevant literature, programming skill, and familiarity with a hidden shortcut.

A problem that takes one specialist several days may be immediate to another specialist who recognizes a particular theorem. Conversely, a problem classified as advanced undergraduate mathematics may become extremely difficult when it requires an unusual construction or extensive computation.

The MIT team experiment provides useful evidence but does not establish a universal human baseline because of its small sample, short time limit, collaboration format, and uneven match between participant expertise and problem subjects.

Statistical Resolution and Saturation

The current Tier 4 set contains only 43 problems. A single changed result moves the score by approximately 2.33 percentage points, and a difference of several points may not represent a statistically robust capability difference.

By July 2026, selected models were reported above 80% on both benchmark components. Such performance suggests that Tiers 1–4 may be approaching saturation much faster than anticipated when the benchmark was introduced in 2024.

Open Problems is intended to extend evaluation beyond this ceiling, but it introduces new difficulties: the set is small and dynamic, some questions may be impossible as formulated, and expert judgment may be needed to decide whether a technically valid construction constitutes a meaningful research solution.

Comparability of Computational Budgets

The one-million-token evaluation allowance permits models to perform extensive inference-time search, repeated calculations, and code-assisted exploration. A high score may therefore reflect a combination of model knowledge, reasoning quality, tool use, computational budget, and answer-selection strategy.

Comparisons are misleading when one system receives a short single attempt while another is allowed hundreds of thousands of reasoning tokens or multiple candidate solutions. FrontierMath reports should distinguish base-model capability from the contribution of test-time computation.

External links

• Official FrontierMath hub
• FrontierMath: Tiers 1–4 overview
• FrontierMath Tiers 1–3 v2 benchmark page
• FrontierMath Tier 4 v2 benchmark page
• FrontierMath: Open Problems
• FrontierMath technical paper

See also

• Large language model

Literature

• Glazer, E. et al. (2024). FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI. arXiv:2411.04872.
• Hendrycks, D. et al. (2021). Measuring Mathematical Problem Solving With the MATH Dataset. NeurIPS. arXiv:2103.03874.
• Lightman, H. et al. (2024). Let's Verify Step by Step. ICLR. arXiv:2305.20050.
• Trinh, T. H. et al. (2024). Solving Olympiad Geometry without Human Demonstrations. Nature, 625, pp. 476–482. [11].
• Azerbayev, Z. et al. (2024). Llemma: An Open Language Model for Mathematics. ICLR. arXiv:2310.10631.
• Xu, C. et al. (2024). Benchmark Data Contamination of Large Language Models: A Survey. arXiv:2406.04244.