GDPval-AA v2

GDPval-AA v2 is an independent benchmark for evaluating advanced artificial intelligence (AI) models on real-world, economically valuable knowledge-work tasks. It is the second-generation evaluation harness and leaderboard built by the research company Artificial Analysis on top of the GDPval dataset, which was released by OpenAI in 2025. Rather than asking closed-ended exam questions, the evaluation requires a model to act as an agent—reading reference files, running code, browsing the web, and producing finished deliverables such as documents, spreadsheets, slide decks, and diagrams—which are then scored by comparing competing models against one another, with human professional work used to anchor the rating scale.

GDPval-AA v2 runs the 220-task public "gold" subset of GDPval inside a sandboxed agent built on Artificial Analysiss open-source Stirrup harness, and ranks models by an Elo rating derived from blind pairwise comparisons of model submissions. In the v2 revision, the Elo scale is re-anchored so that human-expert deliverables sit at approximately 1000 points. GDPval-AA v2 is the single highest-weighted component of the Artificial Analysis Intelligence Index (version 4.1), carrying a 20% weight.

Background

By 2025, many popular AI evaluations had lost their power to distinguish frontier models. Classic academic suites such as MMLU and even harder tests like Humanity's Last Exam were criticised for measuring exam-style recall and abstract reasoning rather than the kind of work people are actually paid to do. To close this gap, evaluation efforts moved from academic exams toward increasingly applied, market-relevant tasks—software-engineering fixes, machine-learning engineering, and freelance-style jobs—culminating in evaluations that grade complete professional deliverables.

GDPval was OpenAI's contribution to this shift. Introduced in September 2025 and described in a research paper led by Tejal Patwardhan and colleagues, GDPval measures model performance on real-world economically valuable tasks across 44 occupations drawn from the 9 industries that contribute most to U.S. GDP. The nine sectors were selected because each contributes over 5% of U.S. GDP, and within each sector the five highest-earning, predominantly knowledge-work occupations were chosen using U.S. Bureau of Labor Statistics and O*NET data. The full dataset contains 1,320 tasks, of which a 220-task gold subset (five per occupation) was open-sourced together with reference files and an experimental automated grader.

Each GDPval task was built from the representative work of industry professionals with an average of 14 years of experience and passed through multiple rounds of expert review. Unlike ordinary text prompts, tasks come with context and supporting files, and the expected outputs span documents, slides, diagrams, spreadsheets, and multimedia. In OpenAI's own study—grading model outputs against human deliverables through blind expert comparison—Claude Opus 4.1 was the strongest model, rated as good as or better than industry experts on roughly 47–48% of gold-subset tasks and excelling at aesthetics such as document formatting and slide layout, while GPT‑5 (high) came second and led on accuracy and instruction-following; by contrast GPT‑4o scored around 12%. OpenAI also reported that frontier models could complete such tasks roughly 100 times faster and 100 times cheaper than human experts, while cautioning that these figures cover only model inference and API pricing, not the human oversight and iteration required in practice.

From GDPval to GDPval-AA

OpenAI's primary grading method relied on paid human experts, which is accurate but slow and costly and does not scale to continuous, independent model comparison. Artificial Analysis—an independent AI-evaluation firm—therefore built GDPval-AA, its own evaluation framework that runs the open GDPval gold subset on any language model and grades the results automatically. GDPval-AA was the centrepiece of a broad overhaul of the Artificial Analysis Intelligence Index (version 4.0), announced in late 2025 and widely reported in January 2026, which replaced several saturated benchmarks with tasks meant to reflect how models are actually deployed in enterprises: as agents that call tools, navigate systems, and assemble composite outputs rather than answering a single prompt.

To run GDPval tasks on arbitrary models, Artificial Analysis released an open-source agentic framework called Stirrup. Within Stirrup, a model is placed in a code-execution sandbox (an E2B environment) and given a set of tools it may call at its own discretion, including web search (via the Brave Search API, returning the top results) and a web-fetch tool that extracts page content as text, alongside shell and file-system access. The model works through a task over multiple turns until it produces the required deliverable. In the first generation of GDPval-AA, agents were allowed up to 100 turns and outputs were graded by a single frontier model acting as judge (Gemini 3 Pro), which Artificial Analysis reported testing for self-preference bias. Because all matches are anonymised and scored head-to-head between competing models, the outcome for each model is an Elo rating rather than a raw percentage.

The v2 upgrade

On 15 June 2026, Artificial Analysis upgraded the evaluation to GDPval-AA v2 as part of the Intelligence Index version 4.1. The revision made several changes to methodology:

• Re-baselined Elo scale. Scores were re-anchored so that human-expert deliverables correspond to roughly 1000 Elo, giving the numbers a direct, interpretable reference point—models above 1000 are, on balance, judged to beat human experts on these tasks.
• Panel of judges. The single LLM judge of the first version was replaced by a rotating panel of three frontier-model judges, reducing the influence of any one grader's idiosyncrasies.
• Longer horizons. The per-task turn limit was raised from 100 to 250 turns, with the ability to exit early, allowing agents to attempt longer, more complex trajectories.
• Upgraded sandbox. The execution environment was updated with new and expanded software dependencies so that models can handle a wider range of file types and workflows.

For its contribution to the composite Intelligence Index, a model's GDPval-AA v2 Elo is frozen at the time the model is added and normalised with the formula clamp((Elo − 500) / 2000); the fixed range keeps each model's index contribution stable over time even as the leaderboard evolves. GDPval-AA v2 carries a 20% weight—more than any other single evaluation in the index—ahead of Terminal-Bench 2.1 (16%) and τ³-Bench Banking (14%).

Methodology summary

The core properties of the evaluation can be summarised as follows:

• Task set. The 220-task public gold subset of GDPval, spanning documents, spreadsheets, slides, and diagrams across finance, healthcare, legal, and other professional domains.
• Agentic execution. Each model runs inside the Stirrup harness with shell access, a code-execution sandbox, web search, and web fetch, over as many as 250 turns.
• Grading. Deliverables from competing models are compared head-to-head on the same task; for each matchup the two outputs are anonymised and a panel of LLM judges selects a winner, and these blind pairwise results are aggregated into an Elo rating per model.
• Reference point. The Elo scale is anchored to human-expert deliverables at approximately 1000, so the leaderboard measures models relative to skilled human professionals rather than on an absolute percentage scale.

Because it grades complete file outputs produced through tool use rather than short text answers, GDPval-AA v2 differs sharply from traditional multiple-choice or exact-match benchmarks and is intended to track practical, workplace-relevant capability.

Results

Under the v2 methodology, the leading systems are judged to clear the human-expert baseline (≈1000 Elo) comfortably. At the v2 launch on 15 June 2026, Artificial Analysis reported Claude Fable 5 (run with a fallback to Claude Opus 4.8) leading at 1818 Elo, followed by Claude Opus 4.8 at 1638, with GPT‑5.5 (xhigh) at 1531; Claude Fable 5 is a model that is not currently available for use. At the level of the full Intelligence Index v4.1, the same picture held: Claude Fable 5 (with fallback) led overall but was unavailable, leaving Claude Opus 4.8 as the most intelligent generally available model, narrowly ahead of GPT‑5.5.

Because it is a live, relative ranking, the v2 board shifts as new models are added. In a July 2026 snapshot, Claude Fable 5 (with fallback) stood at roughly 1756 Elo and Claude Sonnet 5 had risen to second place; for current standings the live leaderboard should be consulted.

Two coexisting Elo scales. A recurring source of confusion is that Artificial Analysis publishes GDPval-AA results on more than one Elo scale. The v2 figures above re-anchor the scale to human-expert deliverables at ≈1000; on this scale Claude Opus 4.8 is reported at about 1638. The broader standalone GDPval-AA leaderboard, however, uses the original (non-re-anchored) scale, on which Claude Opus 4.8 leads at about 1890 Elo—an implied win rate of roughly 67% against the second-placed GPT‑5.5 (xhigh)—having reached that score using about 15% fewer turns and 35% fewer output tokens per task than its predecessor Claude Opus 4.7, while still using roughly 30% more turns than GPT‑5.5. The two scales are not comparable, so any GDPval-AA number is meaningful only together with the scale and date it refers to.

Leaderboard snapshots

The two tables below come from Artificial Analysis but use different Elo scales; they are point-in-time snapshots that change frequently, and the live leaderboard should be consulted for current standings.

GDPval-AA v2 — re-anchored scale (human expert ≈ 1000), launch snapshot of 15 June 2026.

Model	Elo (v2)
Claude Fable 5 (with Opus 4.8 fallback) — not currently available for use	1818
Claude Opus 4.8 (Adaptive Reasoning, Max Effort)	1638
GPT-5.5 (xhigh)	1531
Human-expert deliverables — baseline anchor   ≈ 1000

In a later July 2026 snapshot on the same v2 scale, Claude Fable 5 stood at roughly 1756 Elo and Claude Sonnet 5 (Adaptive Reasoning, Max Effort) had risen to second place.

Standalone GDPval-AA leaderboard — original Elo scale, live snapshot of 1 July 2026. This is the broader public leaderboard on Artificial Analysiss original scale, which uses a different anchor from the v2 human-anchored figures above and is therefore not comparable to them; only the top entries are shown, and each Elo carries a 95% confidence interval of roughly ±25–40 points.

#	Creator	Model	Elo
1	Anthropic	Claude Opus 4.8 (Adaptive Reasoning, Max Effort)	1890
2	OpenAI	GPT-5.5 (xhigh)	1769
3	OpenAI	GPT-5.5 (high)	1753
4	Anthropic	Claude Opus 4.7 (Adaptive Reasoning, Max Effort)	1753
5	Anthropic	Claude Opus 4.7 (Non-reasoning, High Effort)	1677
6	Anthropic	Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)	1676
7	OpenAI	GPT-5.4 (xhigh)	1674
8	MiniMax	MiniMax-M3	1668
9	Google	Gemini 3.5 Flash (medium)	1658
10	Google	Gemini 3.5 Flash (high)	1656
11	OpenAI	GPT-5.5 (medium)	1649
12	Anthropic	Claude Opus 4.6 (Adaptive Reasoning, Max Effort)	1619
For reference — lower down the same scale:
33	Anthropic	Claude Opus 4.5 (Reasoning)	1452
67	OpenAI	GPT-5 (high)	1294
140	OpenAI	GPT-5 (medium)	1000
312	OpenAI	GPT-4o (Nov '24)	348

Note: at the time of capture the standalone board did not list the newest restricted or preview models (for example Claude Fable 5 and Claude Sonnet 5), which appear on the v2 board used in the Intelligence Index—further indication that the two leaderboards are maintained separately.

Criticism and limitations

As an evaluation, GDPval-AA v2 inherits both the strengths and the constraints of the underlying GDPval dataset, and adds some of its own.

• One-shot deliverables, not whole jobs. Each item is a single, well-specified deliverable produced in one pass; the benchmark does not capture the iteration, context-building, and multi-draft revision that characterise real professional work. OpenAI itself frames GDPval as an early step that does not reflect the full nuance of many economic tasks, and notes that its speed and cost advantages omit the human oversight and iteration needed in practice. Strong scores therefore indicate where models can assist with routine, writable tasks, not that an occupation can be automated.
• U.S.- and knowledge-work-centric by design. The task set is built from U.S. GDP sectors, U.S. occupational data, and English-language deliverables, and it deliberately excludes physical or manual labour. It therefore measures a specific slice of digital knowledge work rather than the global economy.
• Model-based grading. Where OpenAI used paid human experts, GDPval-AA relies on LLM judges, which raises questions about judge bias and self-preference. Artificial Analysis mitigates this with anonymised pairwise comparison, and in v2 with a panel of three judges rather than one; even so, OpenAI's own experimental automated grader agreed with human experts about 66% of the time, against roughly 71% agreement among humans, indicating that automated grading is imperfect.
• Possible presentation bias. In OpenAI's study, part of Claude Opus 4.1's lead was attributed to more polished formatting rather than superior substance. Pairwise judging—human or model—can reward aesthetics, a factor that carries over to GDPval-AA.
• Relative, scale- and snapshot-dependent scores. Elo is a relative measure; ratings drift as the field of models and the number of matchups grow, and Artificial Analysis publishes results on two non-comparable scales (the original standalone scale and the v2 human-anchored scale), on which the same model can appear at very different numbers. Comparisons across scales, judges, or dates are therefore unreliable.
• Limited public coverage and restricted leaders. The evaluation runs on a 220-task subset of the full 1,320-task benchmark, and the top of the v2 board has been held by a model (Claude Fable 5) that is not currently available for use, complicating any reading of it as the practical state of the art.

More broadly, GDPval-AA v2 measures task-level economic productivity, not scientific reasoning, originality, or general intelligence; Artificial Analysis pairs it with other evaluations precisely because a high GDPval-AA score does not imply frontier performance on hard reasoning.

Significance and outlook

GDPval-AA v2 reflects a wider move in AI evaluation away from saturated, closed-ended exams and toward agentic, deliverable-based tasks that resemble paid work.

• A work-grounded reference point. By anchoring its Elo scale to human-expert deliverables at ≈1000, the benchmark offers an interpretable way to ask how close models are to skilled professionals on concrete outputs, and to track that gap over time.
• Influence on aggregate rankings. As the highest-weighted component (20%) of the Artificial Analysis Intelligence Index, GDPval-AA v2 materially shapes how frontier models are ranked and compared in the wider ecosystem.
• Grounding the labour-impact debate. OpenAI's stated aim for GDPval was to ground discussion of AI's economic effects in evidence rather than speculation; an independent, continuously updated implementation extends that goal beyond a single lab.
• Open tooling. Because Stirrup is released as open source, the harness that runs the evaluation can be inspected, reused, and adapted by other researchers, supporting reproducibility.

The results to date suggest that the strongest systems are at or beyond human-expert quality on a narrow, well-specified band of digital deliverables, while leaving open-ended judgment, collaboration, and novel problem-solving as separate and largely unmeasured frontiers.

External links

• GDPval-AA leaderboard — Artificial Analysis
• Intelligence Index v4.1 announcement — Artificial Analysis
• Intelligence Benchmarking Methodology — Artificial Analysis
• Stirrup agentic harness — GitHub
• GDPval announcement — OpenAI
• GDPval research paper (arXiv:2510.04374)
• GDPval gold subset dataset — Hugging Face
• GDPval Explorer

Literature

• Patwardhan, T. et al. (2025). GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks. arXiv:2510.04374.
• Liang, P. et al. (2022). Holistic Evaluation of Language Models (HELM). arXiv:2211.09110.
• Chang, Y. et al. (2023). A Survey on Evaluation of Large Language Models. arXiv:2307.03109.
• Chiang, W.-L. et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv:2403.04132.
• Boubdir, M. et al. (2023). Elo Uncovered: Robustness and Best Practices in Language Model Evaluation. arXiv:2311.17295.
• Xu, C. et al. (2024). Benchmark Data Contamination of Large Language Models: A Survey. arXiv:2406.04244.