SciCode Benchmark

SciCode is a benchmark designed to evaluate the ability of large language models (LLMs) to write code that solves realistic scientific research problems. Unlike exam-style question–answer suites, its tasks are drawn from real research workflows across the natural sciences and require a combination of domain-knowledge recall, reasoning, and code synthesis. The benchmark was curated by scientists and AI researchers and first released as an arXiv preprint in July 2024; a peer-reviewed version was presented at the NeurIPS 2024 Datasets and Benchmarks Track.

In total, SciCode contains 338 subproblems decomposed from 80 challenging main problems, spanning 16 subfields of the natural sciences, and it ships with optional scientist-written background descriptions, gold-standard reference solutions, and executable test cases used for automated grading. The benchmark was intentionally built to remain hard as more general suites saturated: at release the strongest model tested could solve only a single-digit percentage of the main problems in the most realistic setting.

Background

By the mid-2020s, popular coding and knowledge benchmarks such as HumanEval and MMLU had approached saturation, with frontier language models scoring near the ceiling and thus ceasing to be a reliable measure of further progress. The authors of SciCode argued that, because models now outperform average humans on many challenging tasks, it had become difficult to construct evaluations that are simultaneously high-quality, realistic, and hard.

SciCode addresses this by targeting a capability that general-purpose suites do not isolate: writing code to solve genuine scientific research problems. Where SWE-bench measures software-engineering ability on real GitHub issues and GPQA or Humanity's Last Exam measure closed-ended academic knowledge, SciCode sits at the intersection of scientific knowledge and programming, requiring a model to translate a research task into working numerical code. The problems focus on three broad needs of computational science: numerical methods, simulation of physical systems, and scientific calculation.

Creation of the benchmark

SciCode was assembled by a team of roughly thirty authors combining domain scientists and AI researchers. The paper lists Minyang Tian and Luyu Gao as equal-contribution lead authors, with Minyang Tian, Hao Peng, and Eliu Huerta as corresponding authors. Contributing institutions span the University of Illinois Urbana-Champaign, Argonne National Laboratory, Carnegie Mellon University, the University of North Carolina at Chapel Hill, MIT, Harvard University, the University of Chicago, UT Austin, Stanford University, Princeton University, and the NSF AI Institute for Artificial Intelligence and Fundamental Interactions (IAIFI).

The construction process proceeded from real research problems rather than invented puzzles:

• Sourcing. Scientists in 16 natural-science subfields contributed problems from their own research areas; many of the underlying methods have appeared in published work, which helps establish their correctness and relevance.
• Decomposition. Each main problem was factorized into a sequence of subproblems. A subproblem corresponds to a single function to implement, supplied with a docstring, a gold reference solution, and a set of test cases; later subproblems typically reuse the code written for earlier ones.
• Test-case design. Evaluation cases were extracted from real scientific workflows—reproducing results published in papers or matching analytical solutions derived from theory—alongside generic numerical checks.
• Quality control. Every problem passed through three rounds of validation: review by in-domain scientists, by out-of-domain scientists, and by a frontier model (GPT-4).
• Optional background. Annotators wrote optional scientific-background notes that can be prepended to the prompt; supplying them measurably improves model performance.

The public dataset is released on Hugging Face and the evaluation harness is open source. The released set is split into a test set of 288 subproblems (from 65 main problems) and a smaller validation/development set of 50 subproblems (from 15 main problems)—338 subproblems from 80 main problems in total.

Structure and content of the benchmark

SciCode comprises 16 subfields. The paper and the project's GitHub group them under five named domains—physics, mathematics, chemistry, biology, and materials science—while the project site refers to six natural-science disciplines; computational mechanics is treated as part of mathematics in the paper's statistics. All solutions are written in Python, drawing on the standard scientific stack (e.g. NumPy and SciPy). Representative tasks give a sense of the difficulty and breadth: computing the electrostatic energy of a periodic system by Ewald summation, integrating a molecular-dynamics simulation with a Berendsen thermostat and barostat via the velocity-Verlet algorithm, or deriving the entanglement rate of a generalized amplitude-damping quantum channel.

Two structural features distinguish SciCode from earlier coding benchmarks:

• Compositionality. Because problems are solved subproblem-by-subproblem and later steps build on earlier code, a main problem counts as solved only if all of its subproblems pass their tests. Main-problem accuracy is therefore an all-or-nothing measure and is much lower than per-subproblem accuracy.
• Execution-based grading. Answers are graded by running the generated code against executable numerical and domain-specific test cases that are not shown in the model's prompt, making evaluation deterministic and reproducible rather than dependent on a model judge.

Evaluation is reported under several settings that materially affect scores: with or without the scientist-written background; and at the subproblem level versus the main-problem level. The "most realistic" setting is defined as no background provided, graded at the main-problem level.

All tasks in SciCode share common properties:

• Research-grounded: problems come from real scientific workflows, and many derive from methods used in published research.
• Verifiable: each subproblem has executable test cases—reproducing published or analytical results—so grading is automated and objective.
• Multi-step and knowledge-intensive: solving a problem requires recalling domain knowledge, reasoning about it, and synthesizing correct code, often across many dependent steps.

Model performance results

SciCode proved extremely difficult at release, and, as with other frontier benchmarks, scores depend heavily on the evaluation date, the grading setting, and whether the metric is per-subproblem or per-main-problem. The figures below should therefore be read as a time series under stated conditions rather than as a single fixed ranking.

• Initial release (2024, most realistic setting). Measured on main problems with no background, leading models scored in the single digits. In the first preprint, Claude 3.5 Sonnet was the best model at 4.6% main-problem accuracy (about 26% at the subproblem level); the NeurIPS camera-ready version reported OpenAI's o1-preview as best at 7.7%. Claude 3 Opus and GPT-4o solved about 1.5% of main problems (GPT-4o roughly 25% of subproblems), and DeepSeek-Coder-v2—the best open model tested—about 3.1% (21% of subproblems); several open models (e.g. Llama-3-70B-Instruct, Mixtral-8×22B-Instruct) failed to complete any main problem despite passing some subproblems.
• Effect of background knowledge. Providing the scientist-written background roughly doubled the ceiling at launch, lifting the best model (Claude 3.5 Sonnet) to about 12.3% of main problems and roughly 35% of subproblems—evidence that a large share of the difficulty lies in recalling the right domain knowledge.
• Current state (2026, subproblem level). On modern independent leaderboards that grade the 288-item test set at the subproblem level using SciCode's scientist-annotated background prompt, frontier models reached the mid-50s to about 60%. On the Artificial Analysis board, Anthropic's Claude Fable 5 led at about 60.2%, followed by Google's Gemini 3.1 Pro Preview at 58.9% and GPT-5.4 (xhigh) at 56.6%; the public availability of Claude Fable 5 was briefly suspended in June 2026 under a U.S. export-control directive before the controls were lifted at the end of that month. Open-weight models also advanced markedly, with Moonshot AI's Kimi K2.6 reported in the low-to-mid 50s on secondary aggregators, though such aggregator figures can rest on different methodology than the Artificial Analysis or original SciCode setups and are best treated with caution.

Because the launch figures are all-or-nothing main-problem scores and the 2026 figures are per-subproblem scores on the test set, the two are not directly comparable; both nonetheless show rapid improvement on scientific coding over roughly two years.

Criticism and limitations

Despite its adoption, SciCode has recognized limitations, most of them inherent to its design.

• All-or-nothing main-problem metric. Requiring every subproblem to pass makes the headline main-problem score conservative and can understate partial competence; a model that gets nearly all of a long problem right still scores zero on it.
• Background-dependence. Because supplying scientist-written background roughly doubled accuracy at launch, measured "capability" is entangled with how much scaffolding the prompt provides, complicating cross-study comparison.
• Narrow scope. SciCode targets Python numerical and scientific computing in the natural sciences; it does not evaluate large-scale software engineering, open-ended research, or the ability to pose new scientific questions.
• Contamination risk. The test set is public, so, as with any released benchmark, scores may drift upward over time through training-data leakage—though research-grounded problems are less templated than exam-style items.
• Verification heterogeneity. Public leaderboards mix independently run and self-reported results and use different harnesses, metrics, and model settings; as a result they can disagree on both the numbers and the current leader.

Significance and outlook

SciCode occupies a distinct niche in the evaluation landscape and has become a standard reference for scientific coding.

• A distinct capability axis. By grounding tasks in real research and grading them by execution, SciCode measures scientific/research coding as a capability separate from general software engineering (SWE-bench) and closed-ended academic knowledge (Humanity's Last Exam, GPQA).
• Adoption into composite suites. SciCode is used as a component of aggregate capability indices, most notably the Artificial Analysis Intelligence Index, giving it a role in tracking frontier progress.
• Toward scientific assistants. The steep rise in scores between 2024 and 2026 suggests real progress toward models that can act as helpful scientific collaborators, while the persistent gap between subproblem and end-to-end main-problem performance marks the harder frontier that remains.

Model results by source

SciCode scores are not absolute: they depend on the evaluation date, whether scientist-written background is supplied, whether the metric is per-subproblem or per-main-problem, and which harness and model settings are used. Numbers are therefore best compared within a single source rather than across sources. The two tables below are point-in-time snapshots under different conditions—the original paper (main-problem accuracy, no background) and an independent evaluator (subproblem accuracy with background)—and each is dated.

Original benchmark paper (2024) — main-problem accuracy, most realistic setting

Results reported in the SciCode paper, graded on main problems with no background provided (the most realistic setting). Under this all-or-nothing metric, a problem counts as solved only if every one of its subproblems passes; per-subproblem accuracy (discussed above) is substantially higher. The first preprint reported Claude 3.5 Sonnet as the strongest model; the NeurIPS camera-ready version added and led with o1-preview.

Model	Main-problem accuracy, %
OpenAI o1-preview (camera-ready)	7.7
Claude 3.5 Sonnet (preprint best)	4.6
DeepSeek-Coder-v2 (best open model)	3.1
GPT-4o	1.5
Claude 3 Opus	1.5
Llama-3-70B-Instruct	0.0
Mixtral-8×22B-Instruct	0.0
With gold scientist-written background, the best model (Claude 3.5 Sonnet) reached about 12.3%.

Artificial Analysis — independent leaderboard (2026)

An independent evaluator that runs its own harness on the 288-subproblem test set, grading at the subproblem level and using SciCode's scientist-annotated background prompt (the with-background condition). Its figures are therefore not comparable to the paper's no-background main-problem scores above; the appropriate 2024 baseline is the with-background subproblem setting, in which the best model of that time scored in the mid-30s. The entries below are the top of a live board read on 1 July 2026 and change frequently. The public availability of Claude Fable 5 was briefly suspended in June 2026 under a U.S. export-control directive; the controls were lifted at the end of that month.

#	Model	Accuracy, %
1	Claude Fable 5 (Adaptive Reasoning, Opus 4.8 fallback)	60.2
2	Gemini 3.1 Pro Preview	58.9
3	GPT-5.4 (xhigh)	56.6

External links

• Official SciCode project site
• Research paper presenting the benchmark (arXiv)
• Peer-reviewed paper (NeurIPS 2024)
• SciCode — GitHub
• SciCode dataset — Hugging Face
• SciCode Leaderboard — Artificial Analysis

Literature

• Chen, M. et al. (2021). Evaluating Large Language Models Trained on Code (HumanEval). arXiv:2107.03374.
• Austin, J. et al. (2021). Program Synthesis with Large Language Models (MBPP). arXiv:2108.07732.
• Jimenez, C. E. et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv:2310.06770.
• Hendrycks, D. et al. (2021). Measuring Massive Multitask Language Understanding (MMLU). arXiv:2009.03300.
• Rein, D. et al. (2023). GPQA: A Graduate-Level Google-Proof Q&A Benchmark. arXiv:2311.12022.
• Liang, P. et al. (2022). Holistic Evaluation of Language Models (HELM). arXiv:2211.09110.
• Xu, C. et al. (2024). Benchmark Data Contamination of Large Language Models: A Survey. arXiv:2406.04244.