OSWorld

OSWorld is an open-source environment and benchmark for evaluating artificial intelligence agents that perform open-ended tasks in real computer operating systems. It was introduced by Tianbao Xie and colleagues in the 2024 paper OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments, published in the Datasets and Benchmarks Track of NeurIPS 2024.

OSWorld is both a virtualized computer environment and a collection of tasks built on that environment. An evaluated agent observes a desktop, controls the mouse and keyboard, interacts with real applications, and attempts to satisfy a natural-language instruction. After the agent finishes, task-specific programs inspect the resulting files, application state, browser data, operating-system settings, or other artifacts to determine whether the requested outcome was achieved.

The original benchmark contains 369 Ubuntu tasks, including 268 single-application tasks and 101 workflows involving multiple applications. It also provides 43 supplementary Windows tasks for analysis. The collection includes 302 distinct initial states and 134 execution-based evaluation functions.

The environment supports Ubuntu, Windows, and macOS, although the principal public benchmark was developed and standardized on Ubuntu. Its applications include Chromium or Google Chrome, LibreOffice Writer, Calc and Impress, Thunderbird, VLC media player, Visual Studio Code, GIMP, a terminal, a file manager, and document viewers.

At the benchmark's introduction, individual human participants completed 72.36% of the evaluated tasks, while the strongest model-agent configuration achieved only 12.24%. The authors attributed the largest failures to inaccurate GUI grounding, insufficient operational knowledge, weak long-horizon planning, and difficulty recovering from errors.

In July 2025, the project released OSWorld-Verified, an in-place revision of the original benchmark with corrected tasks, more robust evaluators, updated applications and websites, and a scalable AWS evaluation platform. On 26 June 2026, the maintainers released OSWorld 2.0, a separate successor benchmark containing 108 substantially longer workflows that take skilled human users a median of approximately 1.6 hours to complete.

Background and Motivation

A computer-use agent is an AI system that translates a user's high-level request into a sequence of operations performed through a computer interface. Unlike an ordinary chatbot, the agent must repeatedly observe a changing environment, decide what to do, execute an action, verify the result, and revise its plan when the interface behaves unexpectedly.

Before OSWorld, many agent benchmarks concentrated on restricted environments:

• Web-navigation benchmarks evaluated tasks inside one browser or a small collection of websites.
• Mobile benchmarks exposed a simplified device interface or recorded demonstrations.
• GUI-grounding datasets asked a model to identify one interface element from a short description.
• Tool-use benchmarks represented applications as structured APIs rather than visible graphical interfaces.
• Static trajectory datasets evaluated next-action prediction without allowing the agent to explore the environment.
• Application-specific benchmarks covered only spreadsheets, presentations, command-line programs, or another narrow domain.

These approaches support controlled experimentation but do not reproduce the full complexity of general computer use. A real desktop task may involve several applications, modal dialogs, hidden menus, files in different formats, transient notifications, operating-system settings, network delays, and intermediate states that were created before the agent began working.

OSWorld was designed around several requirements:

Requirement	Implementation	Purpose
Real applications	Tasks run inside ordinary desktop and web software rather than simplified simulators.	Measures whether an agent can operate interfaces similar to those used by human users.
Open-ended task scope	New tasks can involve arbitrary applications installed in the virtual machine.	Avoids restricting evaluation to one website, API, or application family.
Multimodal observation	Agents can receive screenshots, accessibility information, or a combination of both.	Supports visual, textual, and hybrid computer-use systems.
General action space	Mouse and keyboard operations are represented through PyAutoGUI or a finite action vocabulary.	Allows interaction with almost any visible application.
Intermediate initial states	Configuration scripts can download files, open documents, arrange windows, and modify settings before an episode begins.	Reproduces tasks that start in the middle of a real workflow.
Cross-application workflows	More than one application may be required to complete a task.	Tests transfer of information and state across software boundaries.
Execution-based grading	Custom evaluators inspect the final computer state.	Avoids requiring the agent to reproduce one reference action trajectory.
Isolation and restoration	Tasks execute in virtual machines that can be reset.	Reduces persistent interference between tasks and limits damage to the host system.
Parallel execution	Several virtual machines can run simultaneously.	Makes large evaluations and agent training more practical.

The benchmark is intended to measure an entire computer-use system rather than only the underlying language or vision model. Results depend on the model, observation representation, prompting method, memory system, action parser, grounding module, error-recovery strategy, tool access, and execution budget.

Environment Architecture

Virtualized Computer Environment

OSWorld uses virtual machines to provide a controllable but realistic desktop. Applications execute normally inside the guest operating system, while a host-side controller manages initialization, observations, actions, artifact retrieval, and evaluation.

A typical episode proceeds through the following stages:

1. Start or restore a virtual-machine image.
2. Prepare task-specific files and external resources.
3. Execute initialization actions, such as opening a document or arranging application windows.
4. Provide the task instruction and initial observation to the agent.
5. Execute the agent's mouse, keyboard, or other supported actions.
6. Return an updated screenshot, accessibility tree, or combined observation.
7. Repeat the interaction until the agent declares completion, reports infeasibility, or reaches the step limit.
8. Retrieve files, browser information, application data, and other final-state evidence.
9. Run the task-specific evaluator.
10. Save the score, trajectory, screenshots, actions, logs, and optional video recording.

This architecture separates the evaluated agent from privileged environment operations. Initialization and grading can use APIs or internal application data that are not exposed to the agent, while the agent itself normally interacts through the same visible interface available to a user.

Virtualization also permits the environment to restore a reproducible starting point. Without restoration, one task could change application settings, browser history, files, or account data in a way that affects later tasks.

Initial-State Configuration

Many computer tasks do not begin from a clean desktop. A user may already have a spreadsheet open, a cursor positioned within a document, several browser tabs active, or a partially completed email draft.

OSWorld therefore uses a hybrid initialization process rather than storing a complete independent snapshot for every task. An initial state can be created by:

• Restoring a shared base virtual-machine image.
• Downloading task files.
• Copying assets into specified directories.
• Executing shell commands or application APIs.
• Opening a file in the required application.
• Loading browser pages.
• Resizing or repositioning windows.
• Moving the cursor or selecting an interface element.
• Changing application or operating-system settings.
• Waiting for applications and websites to finish loading.

The approach reduces storage requirements compared with maintaining a multi-gigabyte snapshot for every example. It also makes tasks easier to edit, but introduces dependencies on application startup time, network availability, changing websites, and the precise order of initialization commands.

Headless and Parallel Operation

OSWorld can operate without displaying each guest desktop directly on the host monitor. This headless mode supports automated experiments and parallel execution.

The original repository provides several infrastructure options:

Provider	Typical use	Characteristics
VMware	Desktop, laptop, or bare-metal workstation	Mature virtual-machine integration and support for local interactive debugging.
VirtualBox	Alternative local virtualization	Open-source-compatible option, although some parallel and Apple Silicon workflows may be less developed.
Docker with QEMU and KVM	Linux server with hardware virtualization	Runs virtual machines inside containers and permits several environments on one host.
AWS	Large-scale cloud evaluation	Uses a host-client architecture to launch many preconfigured worker instances in parallel.

OSWorld-Verified expanded the AWS implementation to support as many as approximately 50 simultaneous environments under the project's reference setup. The maintainers reported reducing a full evaluation from more than ten hours to minutes or, depending on configuration, within approximately one hour.

Operating Systems and Applications

The environment architecture was designed to support several operating systems, but the original benchmark emphasizes Ubuntu because its applications and interfaces can be distributed and configured with fewer licensing restrictions.

Platform	Status in the original release	Notes
Ubuntu	Principal 369-task benchmark	Uses open-source or freely available applications and standardized virtual-machine images.
Windows	43 supplementary tasks	Adapted primarily for cross-platform analysis; some software requires user activation because of copyright or licensing restrictions.
macOS	Supported by the environment architecture	Not the basis of the public 369-task benchmark and subject to additional virtualization and licensing constraints.

The core Ubuntu tasks are grouped into five broad categories:

Category	Applications or interfaces	Example capabilities
Operating system	File manager, terminal, settings, image viewer, PDF viewer	File operations, configuration, search, terminal commands, process and window management.
Office	LibreOffice Writer, Calc, and Impress	Document editing, spreadsheet manipulation, presentation formatting, formulas, tables, and file conversion.
Daily applications	Chromium or Chrome, Thunderbird, VLC media player	Web navigation, email operations, account settings, media playback, and information retrieval.
Professional applications	Visual Studio Code and GIMP	Code editing, configuration, text replacement, image processing, and specialized application controls.
Workflow	Combinations of several applications	Transferring information between files, browsers, email, office software, terminals, and other interfaces.

The phrase “eight representative applications” in the original paper refers to Chrome, VLC, Thunderbird, Visual Studio Code, GIMP, LibreOffice Writer, LibreOffice Calc, and LibreOffice Impress. Basic operating-system applications are included in addition to these programs.

Observation Space

An OSWorld agent can receive different representations of the current desktop. The selected representation strongly affects the measured result because each one exposes different information and creates different grounding problems.

Observation type	Repository identifier	Content	Advantages and limitations
Screenshot	screenshot	A complete image of the current desktop, including application windows, text, icons, menus, dialogs, and cursor state.	Closely resembles human visual input but requires accurate visual recognition and coordinate grounding.
Accessibility tree	a11y_tree	Structured information produced by operating-system accessibility APIs.	Provides textual labels, roles, values, and hierarchy but may be extremely long, incomplete, redundant, or inaccurate.
Screenshot and accessibility tree	screenshot_a11y_tree	Both the desktop image and structured accessibility representation.	Combines visual layout with textual metadata but increases context size and may introduce conflicting information.
Set-of-Mark	som	A screenshot annotated with numbered marks associated with accessibility-derived element metadata.	Simplifies element selection but depends on reliable element extraction and can visually obscure crowded interfaces.

On Ubuntu, accessibility information is obtained primarily through AT-SPI 2. On Windows, the original implementation uses PyWinAuto and operating-system accessibility interfaces.

Accessibility trees are not equivalent to a complete semantic description of an interface. Application developers may omit labels, expose implementation details, duplicate elements, or fail to follow accessibility conventions. Some visually obvious controls may be missing, while invisible or irrelevant elements may occupy much of the tree.

Screenshots have the opposite limitation: they preserve the visible interface but do not directly identify control boundaries, element names, or editable fields. A model may understand the requested action yet click a nearby icon or use an incorrect coordinate.

Action Space

The original benchmark provides two main action representations:

Action space	Description	Typical operations
pyautogui	The agent generates valid Python statements using the cross-platform PyAutoGUI library.	Mouse movement, clicking, dragging, scrolling, typing, key presses, keyboard shortcuts, and timed sequences.
computer_13	A finite parameterized action vocabulary wrapping common computer controls.	A smaller set of mouse, keyboard, scrolling, and control operations designed partly for reinforcement-learning research.

Examples of PyAutoGUI-style actions include:

pyautogui.moveTo(850, 420, duration=0.2)
pyautogui.click()
pyautogui.write("Quarterly report", interval=0.05)
pyautogui.hotkey("ctrl", "s")
pyautogui.scroll(-500)

PyAutoGUI actions can contain short Python control structures, allowing an agent to express several repeated operations in one environment step. This improves efficiency but also makes one “step” more expressive than a single physical mouse or keyboard action.

The original action space includes three special control outputs:

Control action	Meaning
WAIT	The agent expects the application or webpage to change and chooses not to perform an immediate input action.
FAIL	The agent concludes that the task is infeasible under the current conditions.
DONE	The agent declares that the requested task has been completed.

These signals are necessary because some interfaces load asynchronously and because the benchmark intentionally includes requests that cannot be completed.

Later OSWorld implementations also support model-provider-specific computer-use APIs and agent frameworks. Results obtained with native computer-use tools, direct coordinate actions, accessibility identifiers, terminal commands, or generated Python programs should not be treated as identical action conditions.

Benchmark Construction

Task Sources

The original tasks were based on real or realistically reconstructed computer-use requests. Sources included:

• Official software documentation.
• Tutorials and user guides.
• Video tutorials.
• How-to websites.
• Online courses.
• Question-and-answer forums.
• Reddit, Quora, Super User, and Stack Overflow discussions.
• Public blogs.
• Daily workflows designed by the authors.
• Tasks adapted from earlier benchmarks.

Examples were selected for popularity, practical usefulness, diversity, and suitability for reproducible evaluation. Multi-application workflows were more difficult to find in existing public resources, so the authors also combined related tasks or designed realistic scenarios.

A total of 84 examples were integrated or adapted from related datasets and environments, including NL2Bash, Mind2Web, SheetCopilot, PPTC, and GAIA.

Annotation Process

Every task required several components:

• A natural-language instruction.
• Relevant files or online resources.
• An initialization configuration.
• A target application or application set.
• One or more final-state retrieval functions.
• An execution-based evaluator.
• Human verification of feasibility and correctness.

The original paper reports that nine computer-science students spent more than three months constructing the benchmark. Approximately 1,800 person-hours were devoted to task annotation, workflows, and checking, with additional time spent collecting and selecting source tasks.

After annotation, two contributors who had not created a particular example attempted it manually. Tasks were revised when instructions were unclear, the setup was unstable, or the evaluator failed to accept a valid solution.

Dataset Statistics

Original OSWorld benchmark statistics

Property	Count	Share of the 369-task Ubuntu benchmark
Total Ubuntu tasks	369	100%
Single-application tasks	268	72.6%
Multi-application workflows	101	27.4%
Tasks integrated from related benchmarks	84	22.8%
Intentionally infeasible tasks	30	8.1%
Distinct initial states	302	—
Unique execution-based evaluation functions	134	—
Supplementary Windows tasks	43	Not included in the 369-task total

The benchmark's 30 infeasible examples represent requests involving nonexistent, removed, or otherwise unavailable features. An agent must recognize that the requested outcome cannot be achieved rather than performing arbitrary actions or falsely claiming success.

Task Representation

Original OSWorld tasks are generally stored as structured configuration records. Exact fields vary by task and repository version, but a conceptual record can be represented as follows:

{
  "id": "task-identifier",
  "snapshot": "base-environment-name",
  "instruction": "Natural-language request presented to the agent",
  "config": [
    {
      "type": "download",
      "parameters": {
        "url": "task asset",
        "path": "/home/oai/share/input.xlsx"
      }
    },
    {
      "type": "open",
      "parameters": {
        "path": "/home/oai/share/input.xlsx"
      }
    }
  ],
  "related_apps": [
    "libreoffice_calc"
  ],
  "evaluator": {
    "func": "task-specific comparison function",
    "result": "getter for the final artifact",
    "expected": "reference artifact or rule set",
    "options": {
      "task-specific": "comparison parameters"
    }
  }
}

Common task components include:

Component	Function
id	Identifies the example and its stored results.
snapshot	Selects the base virtual-machine image or environment state.
instruction	Contains the request shown to the agent.
config	Defines initialization actions such as downloading files, opening applications, or arranging windows.
related_apps	Records applications involved in the task.
evaluator	Specifies how to retrieve and compare the final state.
trajectory	May provide annotation, provenance, or a reference solution path for development and analysis.

The evaluated agent is not required to reproduce a stored reference trajectory. It may reach the accepted state through a different menu, keyboard shortcut, command-line operation, or application workflow, provided that the evaluator recognizes the result.

Execution-Based Evaluation

Getter and Evaluator Functions

OSWorld grades tasks by inspecting the resulting environment. A getter retrieves relevant evidence, while an evaluator compares that evidence with the task requirements.

Getter functions may retrieve:

• A file from the virtual machine.
• Spreadsheet values, formulas, sheet names, or formatting.
• Document text and style information.
• Presentation slides and objects.
• Image pixels or metadata.
• Browser cookies, history, tabs, URLs, or page content.
• Email recipients or message fields.
• Application configuration files.
• Operating-system settings.
• Accessibility-tree elements.
• A directory listing.
• Runtime data from a website or external service.

Evaluation functions may use:

• Exact string or numerical comparison.
• Structured file parsing.
• Spreadsheet-aware comparison.
• Document- or presentation-aware rules.
• Image similarity and metadata checks.
• Browser-state checks.
• Accessibility-tree selectors.
• File existence and path rules.
• Logical conjunctions or disjunctions of several conditions.
• Dynamic retrieval of an up-to-date reference value.

The reward function returns a value between 0 and 1. Some tasks are effectively binary, while others can expose graded or component-level completion. The headline score is normally the mean reward across the selected task set:

overall score = sum of task rewards / number of evaluated tasks

An execution-based evaluator has two advantages over trajectory matching:

• It can accept different valid action sequences.
• It measures whether the requested state was actually created rather than whether the agent merely described an intention.

It also has limitations. A program can reject a valid result because of formatting, application-version differences, an unexpected but equivalent solution, or a bug in the comparison logic.

Completion, Failure, and Step Limits

An episode normally ends when:

• The agent emits its completion signal.
• The agent reports that the task is infeasible.
• The maximum number of steps is reached.
• The environment or model encounters an unrecoverable error.

A frequently used reference configuration permits 15 agent steps, although later systems often use 50, 100, or more. Since one PyAutoGUI program can contain several low-level operations, an OSWorld step should not be interpreted as exactly one human action.

Increasing the step limit can improve results by giving the agent more opportunities to search, recover, and verify. It can also increase cost and permit inefficient systems to outperform concise agents. Scores should therefore always be reported together with the step budget.

Google Drive Tasks

Eight original tasks depend on Google Drive or associated account configuration. Network restrictions, OAuth state, IP changes, or provider defenses can prevent their initialization.

The official project recognizes two valid evaluation configurations:

• Configure the eight tasks manually and evaluate all 369 tasks.
• Exclude them and evaluate the remaining 361 tasks.

Reports must state which denominator was used. One task changes a 369-task score by approximately 0.27 percentage points and a 361-task score by approximately 0.28 percentage points.

Human Evaluation

The original study asked human participants to operate the same environment. Their average success rate was 72.36%, compared with 88% on the WebArena sample used in the study. The median human completion time was approximately 112 seconds, compared with approximately 35 seconds for the sampled WebArena tasks.

The 72.36% figure should not be interpreted as the maximum possible performance. Human participants could make mistakes, misunderstand unfamiliar applications, encounter unstable tasks, or run out of time. Expert users with repeated access to the tasks may achieve a different score.

Original Model Evaluation

The initial study evaluated systems based on GPT, Gemini, Claude, Qwen, Mixtral, Llama, and CogAgent-family models. The experiments compared screenshot input, accessibility trees, combined observations, and Set-of-Mark prompting.

Selected results reported in the original OSWorld paper

Observation condition	Strongest reported configuration in that condition	Overall success rate
Accessibility tree	GPT-4	12.24%
Screenshot	Gemini-Pro Vision	5.80%
Screenshot and accessibility tree	GPT-4V	12.17%
Set-of-Mark	GPT-4V	11.77%
Human performance	Individual human users	72.36%

These are historical results from the paper's models, prompts, application versions, and evaluators. They should not be compared directly with later OSWorld-Verified systems.

Findings from the Original Study

The authors identified several recurring patterns.

GUI Grounding

Screenshot-based models often chose the correct general operation but clicked the wrong coordinate, selected a nearby control, or failed to identify a small interface element. Higher screenshot resolution generally improved performance, but also increased image-processing cost.

Set-of-Mark annotations helped some models by assigning identifiers to visible elements. The method could nevertheless fail when the underlying accessibility tree omitted an element or produced an inaccurate bounding box.

Accessibility-Tree Quality

Accessibility trees improved some results by exposing labels and control roles. However, raw trees could contain millions of tokens, repeated nodes, invisible elements, or uninformative implementation details. Aggressive filtering reduced context length but risked removing the control needed for the task.

Operational Knowledge

Models frequently lacked knowledge of application-specific menus, shortcuts, terminology, and workflow conventions. They could recognize that a spreadsheet needed modification but fail to find the relevant LibreOffice dialog, formula syntax, or formatting control.

Professional applications such as GIMP and LibreOffice Calc were especially difficult because successful operation required specialized procedural knowledge rather than only general visual understanding.

Cross-Application Workflows

Tasks spanning several programs generally produced low scores. The agent had to remember information found in one application, open another program, preserve relevant values, and verify that the final artifact reflected the complete request.

Long Interaction Histories

Textual histories of earlier observations and actions were more useful than retaining only previous screenshots. Full histories nevertheless consumed context-window capacity and could distract the model with obsolete information.

Error Recovery

Many systems repeated an unsuccessful click, continued from an incorrect state, or declared completion without checking the result. A single early grounding error could propagate through the remainder of the trajectory.

OSWorld-Verified

OSWorld-Verified was announced on 28 July 2025 as a comprehensive revision of the original benchmark rather than a separate task family.

The maintainers collected and addressed more than 300 community-reported issues over approximately two months with a team of about ten contributors. The revision followed more than a year of public use by academic groups, model providers, and computer-agent developers.

Sources of Instability

The project identified several classes of benchmark degradation:

• Websites changing their HTML, CSS, URLs, or visible content.
• Anti-bot systems, CAPTCHAs, geolocation restrictions, and IP blocking.
• Removed downloads or discontinued website features.
• Applications loading more slowly than initialization scripts expected.
• Windows opening in an unexpected size or position.
• Ambiguous color, range, formatting, or scope instructions.
• Multiple valid solutions not recognized by the evaluator.
• Exact formatting checks rejecting equivalent files.
• Spreadsheet formulas being compared incorrectly with displayed values.
• Image, PDF, document, and presentation differences that were visually irrelevant.
• Missing fonts, codecs, audio devices, or video support.
• Task assets becoming unavailable.

These problems demonstrate that a virtual-machine snapshot alone does not make an online computer-use benchmark permanently reproducible. Tasks remain dependent on external websites, accounts, software versions, network conditions, and evaluator assumptions.

Main Improvements

OSWorld-Verified introduced or expanded:

Area	Improvement
Task instructions	Clarified ambiguous requirements and invalid assumptions.
Evaluators	Added tolerance for equivalent formatting, formulas, colors, units, capitalization, and multiple valid solution paths.
Initialization	Improved synchronization so that actions wait for applications and files to load.
Web tasks	Updated URLs, selectors, page structures, proxies, and alternative sites.
Assets	Migrated or mirrored task files to more stable storage.
Virtual machines	Added missing fonts, media support, and application fixes.
Infrastructure	Expanded AWS-based host-client execution and large-scale parallelization.
Verification	Established an official process for rerunning submitted agents under common settings.
Transparency	Published trajectories and implementation details for verified systems where permitted.

The data remain in the original OSWorld repository. A result obtained from the updated files should be identified as OSWorld-Verified rather than assumed to reproduce the 2024 paper configuration.

July 2025 Result Snapshot

The maintainers reported the following selected results when introducing OSWorld-Verified:

Selected OSWorld-Verified results reported in July 2025

System	Reported success rate	System class
CoACT-1	60.76%	Agentic framework
Agent S2.5 with o3	56.0%	Agentic framework
GTA1 with o3	53.1%	Agentic framework
Claude 4 Sonnet	43.9%	General-purpose model-based agent
UI-TARS	40.0%	Specialized computer-use model

These results were a historical snapshot rather than a permanent leaderboard. The systems used different orchestration layers and, in some cases, extensive human trajectory data or specialized grounding components.

The same report found that an o3-based implementation varied from approximately 9.1% to 23.0% when its step budget changed. This illustrates why a model name alone is insufficient to define an OSWorld result.

Verified Leaderboard Policy

For an entry to appear in the verified section of the official leaderboard, the maintainers generally require one of two procedures:

• The project team runs the submitted agent code on its evaluation platform.
• A trusted institution provides monitoring data and complete trajectories.

The agent implementation must be available for inspection, although a proprietary model API or model weights do not necessarily have to be disclosed. This policy attempts to separate reproducible results from self-reported scores produced with unknown task versions or settings.

Software and Reproducibility

The OSWorld software is written primarily in Python and released under the Apache License 2.0. The original repository requires Python 3.10 or later.

A local installation can be started with:

git clone https://github.com/xlang-ai/OSWorld
cd OSWorld
pip install -r requirements.txt
python quickstart.py

A complete benchmark run additionally requires:

• A compatible virtualization provider.
• An OSWorld virtual-machine image.
• Model API credentials or a local model server.
• Application and account configuration required by the selected tasks.
• Network access and optional proxy settings.
• Sufficient disk space for images, logs, and trajectories.
• A result directory.
• An explicit task list and maximum step count.

A reproducible report should identify:

• Repository commit or release date.
• Whether the tasks are original OSWorld or OSWorld-Verified.
• The 369-task or 361-task configuration.
• Virtualization provider and image version.
• Model and exact endpoint or checkpoint.
• Agent framework and grounding model.
• Observation type.
• Action space.
• Screenshot resolution.
• Prompt and system instructions.
• History or memory policy.
• Maximum step count.
• Sampling parameters.
• Access to the terminal, Python, accessibility trees, or external tools.
• Handling of timeouts, retries, and environment failures.
• Number of repeated runs.
• Any post-processing or manual intervention.

OSWorld scores are therefore scores of complete agent systems under specified operating conditions. They are not pure measurements of an underlying foundation model.

OSWorld 2.0

OSWorld 2.0 is a successor benchmark released on 26 June 2026. It retains the real-computer and execution-based principles of the original project but shifts the focus from relatively short desktop tasks to professional workflows lasting tens of minutes or several hours.

The release contains 108 long-horizon workflows and uses 31 self-hosted websites. It spans seven professional domains and 21 subcategories:

• Research.
• Creative production.
• Engineering.
• Personal services.
• Business and finance.
• Administration and compliance.
• Healthcare.

Tasks use realistic input files, stateful user profiles, messages that may arrive during execution, and application states that must be inferred from several independent sources.

Comparison with OSWorld 1.0

Property	OSWorld 1.0 / Verified	OSWorld 2.0
Main purpose	Broad evaluation of general desktop operation	Long-horizon professional and everyday workflows
Number of tasks	369, or 361 without Google Drive tasks	108
Typical human duration	Median approximately 112 seconds in the original study	Median approximately 1.6 hours
Tasks taking more than one hour	Not a defining property	69.6%
Typical agent horizon	Approximately 30 actions in the original characterization	More than 250 average steps; 318 average tool calls for one reference configuration
Applications and services	Real desktop applications and public websites	Desktop applications plus 31 controlled or self-hosted websites
Evaluation density	Primarily final-state execution checks	Final completion plus an average of 27.25 scoring checkpoints per task
Primary score	Mean execution-based task reward	Binary completion at 500 steps, accompanied by a partial score
Task distribution	Operating system, office, daily, professional, and cross-app tasks	Seven professional domains and 21 subcategories

The two benchmarks are not directly score-compatible. OSWorld 2.0 changes the tasks, infrastructure, time horizon, evaluation checkpoints, website environment, user-state representation, and model budget.

Challenge Phenomena

OSWorld 2.0 annotates each task with non-exclusive challenge categories:

Challenge	Tasks	Share	Description
Cross-source reasoning	46	42.6%	Combining relevant information from emails, documents, websites, records, or previous messages.
Visual-spatial precision	45	41.7%	Performing accurate placement, alignment, geometry, timing, or layout-sensitive operations.
Implicit-state inference	43	39.8%	Recovering information that is not stated directly and is not available from one obvious source.
Multi-item state tracking	43	39.8%	Maintaining consistent state across many rows, records, events, candidates, or document edits.
Conflict disambiguation	39	36.1%	Determining which of several stale, noisy, or contradictory sources is authoritative.
Multimodal editing	30	27.8%	Producing or modifying images, video, audio, CAD models, or other non-text artifacts.
Tutorial following	22	20.4%	Extracting procedures from PDFs, webpages, videos, or examples and adapting them to the current task.
Dynamic environment	10	9.3%	Revising the plan after new task-relevant information arrives during execution.
Streaming interaction	6	5.6%	Acting while the visible environment changes between observations and actions.
Proactive interaction	6	5.6%	Recognizing incomplete or conflicting requirements and asking the simulated user for clarification.

Percentages overlap because one workflow may contain several challenges.

Evaluation Metrics

OSWorld 2.0 uses two principal task-completion measures:

Metric	Meaning
Binary completion	Receives full credit only if the workflow reaches the required complete final state.
Partial score	Averages task-specific checkpoint results to measure how much of the workflow was completed correctly.

A task contains an average of 27.25 scoring checkpoints. Checkpoints may evaluate intermediate records, files, submitted forms, application state, generated artifacts, or required pieces of information.

The primary reported evaluation permits up to 500 steps. Separate safety reports audit potentially sensitive actions rather than combining safety automatically with the completion score.

Initial OSWorld 2.0 Results

Selected 500-step results from the OSWorld 2.0 paper are shown below:

Agent configuration	Binary completion	Partial score
Claude Opus 4.8, maximum thinking and batched actions	20.6%	54.8%
Claude Opus 4.7, maximum thinking and batched actions	18.2%	48.9%
GPT-5.5	13.0%	49.5%

These values describe the paper's specific agents, prompts, tools, budgets, and model endpoints rather than an automatically updated leaderboard.

The study found a strong relationship between human task duration and agent failure:

• On tasks taking skilled humans less than 45 minutes, selected frontier agents achieved approximately 20–24% binary completion.
• For the 137–163 minute duration group, no evaluated model exceeded 10%.
• Above 163 minutes, every model retained in the reported comparison had zero binary completions.

Longer workflows produced cumulative failures in state tracking, source reconciliation, prioritization, verification, and time management. Agents often collected relevant evidence but began the final form, report, or submission process too late to finish within the step budget.

Data Access and Version Pinning

The OSWorld 2.0 environment is published in a separate repository under the Apache License 2.0.

The official Python task classes are distributed through a gated Hugging Face dataset rather than directly in the public repository. Access requests are automatically approved, but the gate is intended to reduce accidental training contamination and prevent an executing agent from retrieving evaluator details or solutions from an indexed public source.

Large assets are distributed separately, and official release manifests pin:

• Task version.
• Website version.
• Evaluation code.
• Provider image.
• Supporting assets.

An example release identifier is osworld-v2-2026.06.24. Reports should use an explicit manifest rather than only the label “OSWorld 2.0”.

Some workflows rely on mocked websites hosted by the project or deployed locally. GitLab-backed tasks require a self-hosted installation because exposing one shared private access token would create a security risk.

Related Benchmarks and Extensions

Several projects extend OSWorld or reuse its environment for more specialized evaluation.

Benchmark or dataset	Year	Focus	Main characteristics
OSWorld-G	2025	GUI grounding	Contains 564 fine-grained samples covering text matching, element recognition, layout understanding, precise manipulation, and infeasibility.
OSWorld-Human	2025; revised 2026	Agent efficiency	Provides a human-determined trajectory for each original OSWorld task and compares the number and latency of agent steps with human workflows.
OS-Harm	2025	Safety	Adds 150 tasks involving deliberate misuse, prompt injection, and unintended model behavior across several harm categories.
OSWorld-MCP	2025	GUI and structured tool use	Adds 158 Model Context Protocol tools across seven applications and identifies 250 OSWorld tasks for which the tools are applicable.
OSWorld 2.0	2026	Long-horizon professional workflows	Contains 108 workflows with stateful profiles, authentic artifacts, controlled websites, dense checkpoints, and up to 500 evaluation steps.

OSWorld-G

OSWorld-G isolates the grounding component of computer use. Instead of evaluating an entire task trajectory, it tests whether a model can identify or manipulate the correct interface element under realistic instructions.

Its 564 samples cover:

• Text matching.
• Element recognition.
• Layout understanding.
• Fine-grained manipulation.
• Infeasible grounding requests.

The accompanying Jedi training collection contains approximately four million synthetic grounding examples. In the reported experiments, improved grounding increased one agent configuration's OSWorld performance from approximately 5% to 27%.

This result supports the original OSWorld finding that coordinate grounding is a major bottleneck, while also showing that grounding quality alone does not solve planning, operational knowledge, or error recovery.

OSWorld-Human

OSWorld-Human provides a verified human action trajectory for each original task and evaluates temporal efficiency in addition to final success.

The revised study reports that leading agents still require approximately 2.7 to 4.3 times as many steps as the corresponding human trajectories. Large-model calls for planning and reflection account for much of the latency, and later steps in a long trajectory can take up to three times longer than early steps because the model must process a larger history.

The dataset highlights a limitation of success-rate leaderboards: an agent can eventually finish a task while taking too long or spending too much inference compute to be practical.

OSWorld-MCP

OSWorld-MCP evaluates hybrid systems that can choose between direct GUI actions and structured MCP tools.

The benchmark provides:

• 158 curated tools.
• Coverage of seven common applications.
• 250 tool-applicable OSWorld tasks.
• 25 distractor tools.
• Tasks requiring several successive tool calls.
• Tool Invocation Rate and Average Completion Steps in addition to task accuracy.

In one reported configuration, MCP access increased the success rate of an o3-based agent from 8.3% to 20.4% under a 15-step budget. Models nevertheless invoked useful tools infrequently and struggled to compose several tools correctly.

The extension tests whether agents can decide not only what operation is required, but also whether a GUI action, command, or structured application tool is the most effective execution path.

OS-Harm

OS-Harm applies the OSWorld environment to computer-agent safety. Its 150 tasks are divided among:

• Deliberately harmful user requests.
• Prompt-injection attacks.
• Unsafe behavior initiated by the model.

The tasks cover harms such as harassment, copyright infringement, disinformation, and data exfiltration. The benchmark evaluates both whether the requested computer operation was completed and whether the agent performed unsafe actions.

OS-Harm addresses a gap in the original OSWorld metric, which concentrates on task outcome and generally does not penalize unnecessary harmful intermediate operations.

Limitations and Criticism

Dynamic Websites and Environment Drift

OSWorld uses real applications and websites, which increases realism but weakens long-term reproducibility.

External dependencies can change through:

• Website redesigns.
• New CAPTCHAs or bot-detection systems.
• Regional restrictions.
• Account authentication changes.
• Removed content or downloads.
• Application updates.
• Different fonts or codecs.
• Revised browser security behavior.
• Network outages.
• Loading-time variation.

OSWorld-Verified corrected more than 300 issues, but the same class of problems can recur. A result is reproducible only relative to a particular environment image, task version, website state, and date.

Task and Evaluator Correctness

A custom evaluator is itself software that can contain errors. It may:

• Expect an outcome inconsistent with the instruction.
• Reject an equivalent solution.
• Compare formatting too strictly.
• Ignore an important task component.
• Accept a shortcut that violates the intended workflow.
• Fail after an application update.
• Retrieve the wrong file or interface element.
• Depend on unstable external content.

The large OSWorld-Verified revision demonstrates the human effort required to maintain reliable task rewards in real applications.

Endpoint Scoring

Execution-based evaluation usually emphasizes the final state. It may not detect that an agent:

• Read or exposed private information unnecessarily.
• Modified an unrelated file.
• Performed a destructive intermediate action and later reversed it.
• Used an unsafe command.
• Violated a user preference.
• Repeated an operation excessively.
• Reached the correct result through an unauthorized shortcut.
• Gave the user a misleading explanation.

A successful task score therefore does not certify a safe, efficient, or trustworthy trajectory.

OS-Harm, OSWorld-Human, trajectory-level audits, and the separate safety reports in OSWorld 2.0 address parts of this limitation but do not provide a universal solution.

Public Benchmark Contamination

The original task instructions, assets, configurations, evaluator code, and many trajectories are public. They can enter:

• Pre-training corpora.
• Instruction-tuning datasets.
• Agent trajectory collections.
• Reinforcement-learning data.
• Prompt libraries.
• Application-specific grounding datasets.

A system can also be developed directly against the public benchmark. High performance may reflect genuine generalization, benchmark-specific scaffolding, memorized trajectories, or a mixture of these factors.

OSWorld 2.0 reduces direct exposure by placing official task classes behind a lightweight gate and using versioned release manifests. This does not guarantee that the tasks remain unseen by every model developer.

Agent-System Dependence

OSWorld evaluates a complete agent stack. Two implementations using the same foundation model can differ substantially because of:

• Grounding model.
• Planner.
• Reflection loop.
• Memory compression.
• Prompt wording.
• Screenshot resolution.
• Accessibility-tree filtering.
• Action parser.
• Tool access.
• Error-recovery procedure.
• Maximum steps.
• Repeated sampling.
• Human demonstrations.
• Terminal or scripting access.

A leaderboard entry should not be interpreted as an isolated measure of the named model.

GUI Skill Versus Scripting Skill

The environment contains a terminal and permits keyboard input. An agent may solve some nominally graphical tasks by opening the terminal and running shell, Python, or application commands.

This reflects a legitimate human computer-use strategy, but it complicates interpretation. A high score may represent general computer problem solving rather than direct mastery of the GUI being tested.

Comparisons should disclose whether terminal access, generated scripts, application APIs, or external tools were allowed.

Platform and Application Coverage

The core benchmark is centered on Ubuntu and open-source desktop software. Many organizations instead use:

• Windows.
• macOS.
• Microsoft Office.
• Adobe Creative Cloud.
• Enterprise resource-planning systems.
• Proprietary communication tools.
• Internal databases.
• Regulated or security-sensitive applications.

The supplementary Windows tasks do not provide coverage equivalent to the Ubuntu suite. Performance on OSWorld therefore cannot establish competence across all operating systems or workplace software.

Task Horizon

Most original tasks are much shorter than real professional projects. They may require several applications but usually represent one bounded request.

OSWorld 2.0 was created partly because improvements on short-horizon benchmarks did not demonstrate that agents could maintain state and priorities for an hour-long workflow. The successor shows that performance drops sharply as human task duration increases.

Human Baseline Interpretation

The original 72.36% human score is not a fixed upper bound. Participants differed in application familiarity and could encounter the same environment instabilities as agents.

Later agent systems may also receive resources not available in the original human study, such as:

• Large step budgets.
• Python scripting.
• Specialized grounding models.
• Stored demonstrations.
• Application-specific tools.
• Repeated attempts.
• Extensive inference-time computation.

Comparisons with the original human percentage should therefore describe the differing conditions.

Statistical Resolution

The 369-task benchmark has limited resolution. One task changes the full score by approximately 0.27 percentage points. Per-application subsets are smaller and consequently have wider uncertainty.

A model may also behave stochastically. One run per task does not show whether the same system succeeds reliably after a small visual, timing, or generation variation.

Repeated trials and confidence intervals provide stronger evidence but significantly increase evaluation cost.

Infrastructure Cost and Complexity

A complete evaluation requires virtual machines, storage, network access, model inference, account configuration, and substantial execution time.

Cloud parallelization shortens wall-clock time but may create considerable infrastructure and model-API costs. Local execution avoids some cloud expenses but requires compatible virtualization hardware and longer evaluation periods.

The complexity of the setup creates a risk that two groups using the same task list still differ in application versions, image configuration, network conditions, or timeout behavior.

Performance and Practicality

Task success alone does not measure whether an agent is useful in practice. A system can obtain a high score while taking many more actions and much longer than a human.

OSWorld-Human found that even leading agents use substantially more steps than necessary. OSWorld 2.0 additionally shows that higher completion rates can require disproportionately larger token budgets and costs.

Practical evaluation should therefore report:

• Success rate.
• Partial completion.
• Number of actions.
• Wall-clock time.
• Model tokens.
• Monetary cost.
• Recovery attempts.
• Safety violations.
• User interventions.

Cross-Version Comparability

At least three materially different configurations are commonly called OSWorld:

Name	Release	Interpretation
Original OSWorld	April 2024	The task and evaluator state associated with the NeurIPS paper.
OSWorld-Verified	July 2025	An in-place correction of the original task family with updated infrastructure and grading.
OSWorld 2.0	June 2026	A separate long-horizon successor with new tasks, websites, metrics, and release manifests.

Scores across these configurations should not be placed in one numerical ranking. A paper or model card should identify the exact task revision rather than reporting only “OSWorld accuracy”.

External links

• Official OSWorld 1.0 and OSWorld-Verified website
• Official OSWorld repository
• Original OSWorld paper
• OSWorld-Verified announcement
• Official OSWorld 2.0 website
• Official OSWorld 2.0 repository
• OSWorld 2.0 paper

See also

• AI agent
• Multimodal large language model

Literature

• Xie, T. et al. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. NeurIPS 2024. arXiv:2404.07972.
• Yuan, M. et al. (2026). OSWorld2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks. arXiv:2606.29537.
• Xie, T. et al. (2025). Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis. arXiv:2505.13227.
• Abhyankar, R.; Qi, Q.; Zhang, Y. (2025). OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents. arXiv:2506.16042.
• Jia, H. et al. (2025). Benchmarking MCP Tool Invocation in Computer-Use Agents. arXiv:2510.24563.
• Kuntz, T. et al. (2025). OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents. arXiv:2506.14866.
• Zhou, S. et al. (2024). WebArena: A Realistic Web Environment for Building Autonomous Agents. ICLR 2024. arXiv:2307.13854.