SWE-bench Pro

SWE-Bench Pro (also written SWE-bench Pro) is a large-scale benchmark for evaluating the ability of large language models (LLMs) and agentic systems to solve complex, long-horizon software engineering tasks. It was introduced in September 2025 by Scale AI as a substantially more challenging successor to the original SWE-bench, explicitly designed around enterprise-level problems that lie beyond the scope of earlier code benchmarks; a revised version of the accompanying preprint followed in November 2025. The benchmark contains 1,865 tasks sourced from 41 actively maintained repositories spanning business applications, B2B services, and developer tools. As in SWE-bench, each task provides the model with a repository snapshot and an issue description and requires it to generate a code patch that resolves the problem; validation is automated by running the repository's own tests.

A central design goal of SWE-Bench Pro is resistance to training-data contamination. The public tasks are drawn exclusively from repositories under strong copyleft licenses such as GPL, while a separate commercial subset is built from proprietary startup codebases; both choices raise legal and access barriers intended to reduce the likelihood that the evaluation code was present in model training corpora. The benchmark is markedly harder than its predecessor: on the public set, the strongest models at launch resolved only about 23% of tasks under Scale's standardized scaffold, compared with the 70%+ the same models typically reach on SWE-bench Verified.

Motivation and Design Goals

SWE-Bench Pro was created to address four limitations that its authors identified in existing code benchmarks:

• Data contamination — because popular open-source code is likely present in training data, it is hard to tell whether a model is genuinely solving a problem or recalling a memorized solution.
• Limited task diversity — many benchmarks focus on simple utility libraries in a single language (the original SWE-bench consists entirely of Python tasks) and fail to represent the full range of real-world software.
• Oversimplified problems — ambiguous or under-specified issues are often discarded, which does not reflect a real developer's workflow.
• Unreliable and irreproducible testing — inconsistent environment setups make it hard to know whether a solution truly works or the environment is merely misconfigured.

To counter these, the benchmark sources tasks from diverse and complex codebases (consumer-facing applications, B2B platforms, and developer tools), enforces contamination resistance through licensing and private codebases, and preserves under-specified issues by having human experts refine them rather than removing them. Its tasks are deliberately long-horizon: a professional engineer may need hours to days to complete one, and reference solutions typically involve edits across multiple files and substantial code changes.

Methodology and Evaluation

Each problem is constructed through a four-stage workflow:

1. Sourcing — repositories are selected from a curated set of public and private projects.
2. Environment creation — professional engineers build reproducible Docker-based environments with all dependencies and build tools, so the codebase and its tests run out of the box.
3. Harvesting — problems are extracted by scraping consecutive commit pairs, which are kept only if they fix a bug or add a feature, exhibit a fail-to-pass transition for new tests, and include pass-to-pass tests confirming that unrelated functionality still works.
4. Augmentation — human experts turn unstructured commit and issue metadata into a problem statement and a requirements brief (with an optional interface), giving enough context to reproduce the reference patch without prescribing an implementation.

The process relies on three human-in-the-loop checkpoints: manual environment construction, human augmentation of the issue description and requirements, and human verification of test relevance and flakiness.

The primary metric is the resolve rate — the share of tasks an agent completes successfully under a single end-to-end attempt per task (Pass@1). A task counts as resolved only if a submitted patch satisfies two strict conditions in the evaluation environment: it must fix the target issue (the new fail-to-pass tests, which fail on the original code, now pass) and it must introduce no regressions (all pre-existing pass-to-pass tests continue to pass). To characterize model behavior, the authors also cluster the failure modes observed in agent trajectories.

Dataset Composition

SWE-Bench Pro's 1,865 tasks are divided into three subsets:

• Public set — 731 instances from 11 open-source repositories under copyleft licenses. This is the main public-facing benchmark, and results are tracked on the public leaderboard.
• Held-out set — 858 instances from a separate group of 12 copyleft repositories. Its results are not published on the public leaderboards; keeping it private lets it act as a control against overfitting, since a model that scores well on the public set but poorly on the structurally similar held-out set suggests that public tasks may have leaked into training.
• Commercial (private) set — 276 instances from 18 proprietary codebases contributed by early-stage startups under partnership agreements. These are not publicly accessible, but aggregate results are reported on a separate leaderboard.

The public and held-out repositories each contribute on the order of 50–100+ problems, whereas the commercial set is spread far more thinly — 276 tasks across 18 repositories, roughly 15 per repository. Reference solutions require medium-to-large modifications, averaging 107.4 lines of code changed across 4.1 files per task. The tasks span several programming languages — primarily Python, Go, JavaScript, and TypeScript — which distinguishes SWE-Bench Pro from the Python-only original SWE-bench. All tasks are human-verified and augmented with sufficient context to ensure they are solvable.

Results and Model Performance

Scale evaluated frontier models on the public set using the SWE-agent scaffold. The headline finding was a sharp drop in performance relative to SWE-bench Verified: whereas most leading models exceed 70% on Verified, the best performers on SWE-Bench Pro — OpenAI GPT-5 and Claude Opus 4.1 — resolved only about 23% of the public tasks (GPT-5 at 23.3%; Claude Opus 4.1 at roughly 23%, reported as 22.7–23.1% across sources). On the harder commercial set, scores fell further — GPT-5 to 14.9% and Claude Opus 4.1 to 17.8% — underscoring that evaluation on previously unseen, proprietary code is a stricter test of generalization.

The benchmark also exposed a wide spread between models. Frontier systems substantially outperformed older models such as GPT-4o (about 4.9%) and Qwen3-32B (about 3.4%). Performance varied by programming language — Go and Python tasks generally saw higher resolve rates (some models exceeding 30%), while JavaScript and TypeScript results were lower and more erratic — and by repository, with some codebases yielding resolve rates below 10% for every model. The strongest models were not only the most accurate but also the most consistent across languages and repositories, and every model degraded as solutions required more lines and more files to be changed.

In the months after release, newer frontier models improved substantially on Scale's standardized public leaderboard, where the top standardized result reached roughly 59% by mid-2026, still well short of the same models' SWE-bench Verified scores. Reported figures diverge sharply by harness: numbers produced under vendors' own scaffolds run considerably higher — into the 70–80% range — than Scale's standardized results, so any SWE-Bench Pro score must be read together with the scaffold and evaluation conditions that produced it.

Limitations and Significance

As a static, test-based benchmark, SWE-Bench Pro shares limitations of the broader SWE-bench family: correctness is judged by a finite set of unit tests, and passing them does not fully guarantee that a patch matches the developer's intended behavior. Its designers mitigate this with mandatory pass-to-pass regression checks and human review of test relevance, but the general caveat that benchmark scores can overstate true capability still applies. The benchmark is also narrow in scope: it covers only a limited set of languages and only the issue-resolution and patch-writing slice of software engineering — not design, code review, or architecture — and the human-written requirements briefs can remove some of the ambiguity that characterizes real-world issues. Results are further sensitive to the scaffold and compute budget used, so figures from different harnesses are not directly comparable.

A distinct and empirically documented weakness is runtime contamination, separate from the training-data contamination the benchmark was built to resist. In 2026, independent investigators reported that the public open-source evaluation containers could leak reference solutions at evaluation time: although each repository is checked out at the task's base commit, the images retained reachable future git objects — later commits on the main branch, feature branches such as origin/dev, tags, and reflogs — so an exploring agent could recover the fixing commit with ordinary commands such as git log or git show. An audit of agent trajectories by Cursor found that 63% of one frontier model's successful public-set resolutions retrieved an existing fix rather than deriving one — 57% via upstream web lookup and 9% via git-history mining — and that sealing the git history and blocking network access lowered that model's SWE-Bench Pro score from 87.1% to 73.0%. These findings do not overturn the benchmark's design, which does reduce training-time leakage, but they show that its contamination resistance is relative rather than absolute, and that scores obtained without a sealed, network-isolated harness can overstate genuine problem-solving ability.

Despite these caveats, SWE-Bench Pro has quickly become an influential frontier signal for coding agents. By combining licensing and private codebases aimed at reducing training-time contamination, industrially relevant and long-horizon tasks, and reproducible Docker-based evaluation, it offers a harder and more realistic measure of autonomous software engineering than earlier benchmarks, and it complements SWE-bench Verified in tracking progress toward professional-level AI developers.

External links

• SWE-Bench Pro on GitHub
• Official SWE-Bench Pro Leaderboard (Public Dataset)
• SWE-Bench Pro dataset on Hugging Face
• Cursor — Reward hacking and runtime contamination on SWE-Bench Pro

Literature

• Deng, X. et al. (2025). SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?. arXiv:2509.16941.
• Jimenez, C. E. et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. arXiv:2310.06770.