Terminal-Bench

From Systems Analysis Wiki
Jump to navigation Jump to search

Terminal-Bench is a benchmark designed to evaluate the capabilities of autonomous artificial intelligence (AI) agents on realistic, end-to-end tasks performed through a command-line terminal. Unlike knowledge tests that grade a single textual answer, Terminal-Bench places an agent inside a sandboxed operating-system environment and scores whether it can actually accomplish an objective—compiling software, configuring servers, training models, debugging systems, or recovering data—by verifying the final state of that environment[1]. The project is a joint effort of Stanford University and the Laude Institute, developed openly with contributions from a large community of researchers and engineers[2].

Terminal-Bench is conceived as a measure of how far agents have progressed toward autonomously completing valuable, long-horizon work in the environment that has quietly become the backbone of modern coding agents—the terminal[1]. The benchmark is distributed together with its evaluation harness, so that any language model paired with any agent scaffold can be run against the same tasks and reported on a public leaderboard at tbench.ai[3].

Background

By the mid-2020s, agentic AI systems such as Cursor, OpenAI's Codex CLI, Claude Code, and Gemini CLI had made the command line a primary interface through which models perform complex work[1][4]. Existing agent benchmarks, however, tended to fall into one of two traps: either they measured narrow, artificial tasks that did not reflect real workflows, or they were not difficult enough to separate genuinely capable frontier agents from those that could handle only routine operations[1].

A direct methodological ancestor was SWE-bench, which asks a model to resolve real GitHub issues and grades it against the unit tests contributed by the pull request that originally closed them. Members of the Laude Institute had previously built the K-Prize (the Konwinski Prize), a $1 million competition around a contamination-free, continually updated version of SWE-bench. Working on that problem made clear that SWE-bench's design—tasks constrained to GitHub issues in open-source Python repositories—was a special case of a broader abstraction: an agent operating in an arbitrary computing environment, verified by tests over the resulting state[5]. Terminal-Bench generalized that idea to the terminal itself.

Creation of the benchmark

Original release

Terminal-Bench was publicly introduced on 19 May 2025, alongside Terminus, a research-preview agent that serves as a neutral scaffold for evaluating language models as autonomous terminal agents[6]. The effort was led at Stanford by Mike A. Merrill and at the Laude Institute by Alexander G. Shaw (a former Google engineer), within the institute's programme of "impact research"—work intended to produce usable artifacts and not only papers. The Laude Institute was founded by Andy Konwinski, a co-founder of Databricks and Perplexity, whose emphasis on shipping early and gathering feedback shaped the project's development[5]. The initial task set, Terminal-Bench-Core, contained 80 tasks and shipped with both a dataset and an evaluation harness connecting a language model to a Docker-based terminal sandbox[3].

Reception was strong. By late 2025 the repository had passed 1,000 GitHub stars and drawn contributions from nearly 100 developers worldwide, and Terminal-Bench scores had begun appearing on the model cards of newly released systems such as DeepSeek-V3.1-Terminus and Qwen3-Coder, as well as in Anthropic's Claude 4 and Claude Sonnet 4.5 announcements[4][7]. One prolific external contributor, security researcher Nicholas Carlini, authored a large fraction of the task corpus, ranging from cryptanalysis exercises to compiler and interpreter construction[2].

Terminal-Bench 2.0

From the outset Terminal-Bench was intended as a "living" benchmark that would be tightened as models improved. Through 2025, frontier performance on the original set climbed past 50%, which raised the prospect that the benchmark would saturate before it had accumulated enough evaluations to serve as a durable signal[7]. Post-launch experience had also surfaced reliability problems in the original corpus: some tasks were unsolvable for artificial reasons, others set arbitrary success thresholds, and a few were environmentally unstable—most notoriously a "download YouTube video" task that broke as YouTube changed its anti-bot protections[7].

Terminal-Bench 2.0 was released on 7 November 2025 to address these issues, in collaboration with Snorkel AI as one of the top external contributors. It comprises 89 rigorously curated tasks, each reviewed for reproducibility, specification quality, and solvability, with easier and unreproducible items removed and specifications tightened so that near-100% performance is attainable in principle by a sufficiently capable agent. The stated design target was a "50% performance ceiling": a difficulty level low enough to yield a meaningful signal, but high enough that clear room for improvement remains[7]. The release also introduced a new, more scalable evaluation harness called the Harbor framework—now the official harness for running the benchmark—and updated the default scaffold to Terminus 2[8][2]. The accompanying paper, "Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces," was submitted to arXiv on 17 January 2026 (arXiv:2601.11868), with Mike A. Merrill as lead author and 84 co-authors, reflecting the community-built nature of the task corpus[1][9].

Terminal-Bench 2.1

A subsequent verified refresh, Terminal-Bench 2.1, was released on 6 May 2026[6]. It keeps the same 89 tasks but repairs 28 of them: 9 whose external dependencies had drifted, 8 whose resource limits were too tight, and the remainder affected by mismatches between a task's instructions and its tests. The refresh adds continuous validation, which re-checks task environments over time, and after the fixes no task is left unsolvable[10]. Because the changes remove artificial failures rather than add difficulty, measured scores generally rise relative to 2.0—by as much as roughly 12 percentage points on comparable model-and-scaffold pairs. Version 2.1 is evaluated under controlled conditions (submissions may not alter timeouts or resources) and is the more reliable basis for cross-model comparison, while the larger 2.0 leaderboard remains as a historical record on the original task set[10][11].

Structure and content of the benchmark

Each Terminal-Bench task is a self-contained unit consisting of a unique Docker container (a base image with the relevant packages and files pre-installed), a natural-language instruction describing the objective, a human-written oracle solution, and a set of automated tests that verify the outcome. Grading inspects only the final state of the container—files produced, services running, values computed—not the specific commands the agent issued or its intermediate steps[2]. Many tasks assign per-test weights, allowing partial credit when an agent satisfies some but not all of the verification conditions[3].

Submissions pair a backbone model with an agent scaffold—Codex CLI, Terminus 2, Mini-SWE-Agent, Claude Code, and others—and each Model + Scaffold combination is reported as its own leaderboard row[2]. Evaluation is designed to be reproducible: each task specifies fixed resource and time limits—CPU, memory, storage, GPU allocation, and separate agent and verifier timeouts—that submissions may not alter, so that scores are not confounded by hardware differences[12][13].

The 89 tasks of the version 2 set span 16 categories and three difficulty tiers, as recorded in the task registry[12]:

Category Tasks
Software engineering 26
System administration 9
Security 8
Scientific computing 8
Data science 8
File operations 5
Debugging 5
Model training 4
Mathematics 4
Data processing 4
Machine learning 3
Games, personal assistant, optimization, data querying, video processing 1 each

By difficulty, the set is weighted toward the middle tier: 4 easy, 55 medium, and 30 hard tasks[12]. The tasks are deliberately varied and resistant to pattern-matching on training data. Representative examples from the version 2 set include cross-compiling the game Doom to run on the MIPS architecture, recovering a deliberately truncated SQLite database, performing differential cryptanalysis of the FEAL cipher, cracking a password-protected 7-Zip archive with John the Ripper, and installing Windows 3.11 under emulation[12].

All tasks share common properties:

  • Realistic and end-to-end: each task mirrors a problem from a genuine engineering or research workflow rather than an isolated code snippet[1].
  • Programmatically verifiable: success is determined by tests run against the container state, suitable for automated grading[2].
  • Difficult and contamination-resistant: tasks are chosen so that they are unlikely to have been memorized, and so that clearing them requires genuine multi-step execution rather than recall[14].

The primary metric is the task resolution rate (equivalently, pass@1 accuracy), reported per submission with a 95% confidence interval[2].

Model performance results

Terminal-Bench was designed to be hard, and it has remained so as models have improved. A reported score depends heavily on the benchmark version (2.0 vs the cleaned 2.1), the agent scaffold, the task subset, and the evaluation date, so the figures below should be read as a moving picture rather than a fixed ranking.

  • Original benchmark (2025). Through 2025, the strongest systems pushed accuracy on the original 80-task set past the 50% mark, which was itself the trigger for a harder second version[7].
  • At the paper's release (January 2026). Under the benchmark's own harness at the time of the paper, frontier models and agents scored below 65%. The authors noted a steep trajectory: in the roughly eight months between the releases of Gemini 2.5 Pro and GPT-5.2, state-of-the-art performance on the benchmark nearly doubled[1].
  • Scaffold matters more than it looks. On the official leaderboard, a submission's score reflects the agent scaffold wrapped around the model as much as the model itself. The top of the 2.0 board is held by heavily engineered, third-party custom scaffolds; the same frontier model under the neutral Terminus 2 harness scores substantially lower—for example, Terminus 2 with Claude Opus 4.6 reaches about 62.9%, versus a custom-scaffold ceiling near 84.7%[13].
  • Current state (mid-2026). On the historical 2.0 board (142 submissions), the best model-and-scaffold combinations reach the mid-80s, with seven entries above 80%, though the median sits near 44%[13]. The cleaner 2.1 refresh places its top verified entries in the low-80s (Codex CLI with GPT-5.5 at about 83.4%)[10]. Independent evaluations that use a harder subset land lower still: on Artificial Analysis's "Terminal-Bench Hard," leaders sat in the low-60s[14].

The authors acknowledge that, if the trend continues, Terminal-Bench may itself approach saturation within about a year—at which point models would be able to reliably and autonomously complete most well-defined terminal tasks, and a further revision would likely be needed[1].

Criticism and limitations

Despite its rapid adoption as a de-facto standard for agentic coding evaluation, Terminal-Bench has recognized limitations, several of which its own maintainers have documented.

  • Reliability of the tasks. The first release contained tasks that were unsolvable, arbitrarily threshold-gated, or dependent on unstable external services (such as the YouTube-download task affected by changing anti-bot measures). These problems motivated the version 2.0 redesign and the further 2.1 refresh, which repaired 28 more tasks and added continuous validation[7][10].
  • Leaderboard integrity. Because submissions are self-run and the large, open 2.0 board rewards heavily engineered custom scaffolds, the maintainers introduced explicit policies against cheating and reward hacking in April 2026 following integrity concerns; the team-verified 2.1 board and its controlled-condition runs are in part a response. Top entries on the open board should therefore be read with caution[6].
  • Scaffold confound. Because a leaderboard entry is a model plus an agent scaffold, a high score can reflect a well-engineered harness as much as a capable underlying model. This makes cross-model comparison delicate: the same model can differ by many points depending on the scaffold, and the neutral Terminus 2 harness is provided precisely to enable more like-for-like comparison of bare models[13][2].
  • External validity. Strong Terminal-Bench performance may not transfer to messier real-world terminal work. The authors of TerminalWorld, a benchmark of 200 verified real-world terminal tasks, report that even the strongest systems solve at most about 62.5% and that scores correlate only weakly with curated benchmarks such as Terminal-Bench (Pearson r = 0.20), suggesting the two capture different skills[15].
  • Narrow task scope. By construction, Terminal-Bench measures closed-ended, verifiable tasks that suit terminal execution, chiefly technical workflows. It does not assess creativity, open-ended research, or the ability to pose new problems, so strong performance does not by itself indicate general capability[1]. Broader efforts such as TUA-Bench extend terminal-use evaluation beyond technical work to 120 tasks across five families—including document editing, email management, and live-web information seeking—where the strongest agent (Claude Code with Claude Opus 4.8) reaches 65.8%[16].
  • Score sensitivity and saturation. Reported numbers vary with the version, scaffold, task subset, and date; different leaderboards therefore disagree on both magnitudes and the current leader. And because the benchmark targets a 50% ceiling for frontier systems, its useful lifetime is bounded—rapid progress that is welcome for the field also shortens the window in which the benchmark discriminates[7][1].

Significance and outlook

Terminal-Bench arrived as agentic coding tools were becoming mainstream, and it filled a gap between artificial micro-benchmarks and tasks too easy to challenge frontier agents.

  • A common reference for agents. It offers researchers and developers a standardized—if scaffold- and harness-sensitive—way to track how well agents execute real, multi-step work over time, and its scores are now cited on the model cards of major releases[4].
  • An open, extensible platform. Distributed with its harness (Terminus/Terminus 2) and, from version 2.0, the Harbor framework, the project is built to absorb community-contributed tasks and to port other benchmarks into a common evaluation format[3][8].
  • A moving target by design. The explicit 50% ceiling means the benchmark is meant to be re-hardened and re-verified as capabilities advance. Beyond the original set, version 2.0, and the 2.1 refresh, the maintainers opened a call for contributions to a harder Terminal-Bench 3.0 in March 2026, and in June 2026 announced Terminal-Bench Challenges—long-horizon, token-intensive, single-task benchmarks aimed at evaluating agents on large autonomous projects[6].

Model results by source

Terminal-Bench scores are not absolute: they depend on the benchmark version, the agent scaffold and harness used for grading, the task subset, and the date. Four distinct public boards illustrate this and are not directly comparable: the project's own leaderboards for version 2.0 (large and historical) and version 2.1 (a smaller, verified refresh of the same 89 tasks), and two boards run by the independent evaluator Artificial Analysis—"Terminal-Bench Hard" (a hard subset) and "Terminal-Bench v2.1" (its own run of the refreshed set). Figures are best compared within a single source. All tables below are point-in-time snapshots that change frequently; each is dated, and for current standings the live leaderboards should be consulted.

tbench.ai — official leaderboards

Both official boards score each Model + Scaffold submission by task resolution rate over the 89-task set, reported with a 95% confidence interval; verification checks the final container state only. The 2.1 board is the cleaned, verified refresh and is the sounder basis for comparison; the 2.0 board is larger (142 submissions, median about 44%) but runs on the original task set, and its top places are held by third-party custom scaffolds rather than reference harnesses. The snapshots below are from mid-2026[11][13].

Terminal-Bench 2.1 (verified refresh; 13 submissions, top entries shown):

# Agent (scaffold) Model Accuracy, % (±95% CI)
1 Codex CLI GPT-5.5 83.4 ±2.2
2 Claude Code Claude Fable 5 83.1 ±2.0
3 Terminus 2 Claude Fable 5 80.4 ±2.3

Terminal-Bench 2.0 (historical; 142 submissions, top entries shown):

Rank Agent (scaffold) Model Accuracy, % (±95% CI)
1 NexAU-AHE GPT-5.5 84.7 ±2.1
2 LemonHarness Multiple 84.5 ±2.6
3 Capy GPT-5.5 83.1 ±2.1
4 Codex CLI GPT-5.5 82.2 ±2.2
5 Polaris Multiple 82.2 ±2.8
6 WOZCODE Claude Opus 4.7 80.2 ±2.1
7 TongAgents Gemini 3.1 Pro 80.2 ±2.6
8 LemonHarness Multiple 79.9 ±3.0
For reference — recognizable reference-harness entries (lower on the board):
~36 Terminus 2 (neutral harness) Claude Opus 4.6 62.9
~50 Claude Code Claude Opus 4.6 58.0

Artificial Analysis — independent leaderboards

An independent evaluator that runs its own harness. Its "Terminal-Bench Hard" board scores a challenging subset (28 of the 423 models it tracks had been evaluated at the time of writing) and its leaders sit far lower than the official boards. Its separate "Terminal-Bench v2.1" board runs the refreshed 89-task set with the Terminus 2 harness in an e2b sandbox (pass@1 averaged over three repeats) and reports top scores around 84–85%—Claude Opus 4.8 and Claude Fable 5 tied at 84.6%, with GPT-5.5 (xhigh) at 84.3%—in line with the official 2.1 figures. The two boards are not comparable to each other. The "Terminal-Bench Hard" leaders below are read from the Artificial Analysis leaderboard as of 1 July 2026[14][17].

# Model Accuracy, % (Terminal-Bench Hard)
1 Claude Fable 5 (adaptive reasoning, max effort, Opus 4.8 fallback) 62.9
2 GPT-5.5 (xhigh) 60.6
3 GPT-5.5 (high) 59.8

See also

Literature

  • Merrill, M. A. et al. (2026). Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces. arXiv:2601.11868.
  • Chu, Z. et al. (2026). TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks. arXiv:2605.22535.
  • TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents (2026). arXiv:2606.28480.
  • Jimenez, C. E. et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. arXiv:2310.06770.
  • Wang, X. et al. (2025). The OpenHands Software Agent SDK: A Composable and Extensible Foundation for Production Agents. arXiv:2511.03690.
  • Mialon, G. et al. (2023). GAIA: A Benchmark for General AI Assistants. arXiv:2311.12983.

References

  1. ↑ 1.00 1.01 1.02 1.03 1.04 1.05 1.06 1.07 1.08 1.09 Merrill, M. A., Shaw, A. G., Carlini, N. et al. "Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces". arXiv:2601.11868, 2026. [1]
  2. ↑ 2.0 2.1 2.2 2.3 2.4 2.5 2.6 2.7 "Terminal-Bench 2.0". Snorkel AI — Terminal-Bench 2.0 page. [2]
  3. ↑ 3.0 3.1 3.2 3.3 "Terminal-Bench". GitHub (laude-institute / harbor-framework). [3]
  4. ↑ 4.0 4.1 4.2 "Evaluating Coding Agents with Terminal-Bench". Snorkel AI. [4]
  5. ↑ 5.0 5.1 "Chat With the Terminal-Bench Team". Snorkel AI. [5]
  6. ↑ 6.0 6.1 6.2 6.3 "Terminal-Bench — News". tbench.ai. [6]
  7. ↑ 7.0 7.1 7.2 7.3 7.4 7.5 7.6 "Terminal-Bench 2.0: Raising the bar for AI agent evaluation". Snorkel AI, 7 November 2025. [7]
  8. ↑ 8.0 8.1 "Running Terminal-Bench". Harbor framework documentation. [8]
  9. ↑ "Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces". OpenReview. [9]
  10. ↑ 10.0 10.1 10.2 10.3 "Terminal-Bench 2.1". Snorkel AI (official leaderboard). [10]
  11. ↑ 11.0 11.1 "Terminal-Bench — Leaderboard (v2.1)". tbench.ai. [11]
  12. ↑ 12.0 12.1 12.2 12.3 "terminal-bench-2 — task registry". GitHub (laude-institute/terminal-bench-2). [12]
  13. ↑ 13.0 13.1 13.2 13.3 13.4 "Terminal-Bench — Leaderboard (v2.0)". tbench.ai. [13]
  14. ↑ 14.0 14.1 14.2 "Terminal-Bench Hard Benchmark Leaderboard". Artificial Analysis. [14]
  15. ↑ Chu, Z. et al. "TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks". arXiv:2605.22535, 2026. [15]
  16. ↑ "TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents". arXiv:2606.28480, 2026. [16]
  17. ↑ "Terminal-Bench v2.1 Benchmark Leaderboard". Artificial Analysis. [17]
Retrieved from "https://systems-analysis.ru/eng/index.php?title=Terminal-Bench&oldid=2823"