Τ-bench

τ-bench (pronounced tau-bench; short for Tool-Agent-User Interaction Benchmark) is a benchmark and simulation framework for evaluating how reliably large language model (LLM) agents interact with users, invoke software tools, and follow domain-specific policies during multi-turn conversations. It was introduced by Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan in June 2024 and was published at the International Conference on Learning Representations in 2025.

The original benchmark models customer-service interactions in two synthetic domains: retail and airline. An evaluated agent receives a policy document and access to domain-specific API tools, while a second language model simulates the customer from a hidden scenario description. The agent must gather missing information, explain restrictions, obtain confirmation where appropriate, and produce the required changes in a structured database.

The original release contains 165 tasks: 115 retail tasks and 50 airline tasks. It introduced state-based evaluation, which compares the environment at the end of an interaction with an annotated target state, and the reliability metric pass^k, which measures the probability that an agent succeeds in every one of k independent trials of the same task.

The name is also used for a broader benchmark family. The 2025 successor τ²-bench added a dual-control telecom environment in which both the agent and the simulated user can operate tools. In March 2026, the maintainers branded version 1.0.0 of the actively maintained framework as τ³-bench, adding corrected task sets, full-duplex voice evaluation, and a knowledge-intensive banking domain. The original tau-bench repository is retained as a historical release and is explicitly marked as outdated.

Background and Motivation

Earlier tool-use benchmarks commonly present an agent with a complete instruction and then allow it to act autonomously in a browser, terminal, code environment, or collection of APIs. Such setups test planning and tool selection, but they often omit two requirements of practical customer-facing systems:

• The information needed to complete a task may be distributed across several conversational turns.
• The agent may need to obey business rules that are not enforced directly by the tool implementation.

A real customer-service agent may need to identify a customer, inspect several records, ask clarifying questions, explain policy restrictions, request authorization before making a consequential change, and provide confirmation or financial information after the change. The correct action can depend on membership status, payment method, order state, cabin class, booking time, or preferences revealed only after the conversation begins.

The benchmark was designed around three deployment-oriented requirements:

Requirement	Meaning in τ-bench
Human interaction	The agent must communicate with a simulated customer over several turns and cannot assume that all required details are available in the opening message.
Tool-mediated action	The agent must read and modify structured databases through APIs rather than merely describe what should be done.
Policy compliance	The agent must interpret a domain policy containing procedures, permissions, exceptions, and restrictions that may not be enforced automatically by the tools.
Long-horizon state tracking	Information from earlier messages and tool outputs may remain relevant after many subsequent actions.
Consistency	Repeated runs of the same underlying scenario should reach the same correct outcome despite stochastic variation in the dialogue.

The final requirement motivated pass^k. A system that succeeds in 70% of isolated runs may still be unsuitable for a high-volume service if its behavior changes unpredictably across semantically equivalent conversations.

Benchmark Architecture

Interaction Model

The original paper formalizes each task as a partially observable Markov decision process. In practical terms, an episode contains four active components:

Component	Information available to it	Function
Agent	Domain policy, tool definitions, user messages, and tool results	Talks to the user and decides when and how to call APIs.
User simulator	A hidden task instruction and the user-agent conversation history	Produces natural-language customer messages and decides when the interaction is complete.
Environment	Structured databases and deterministic tool implementations	Executes valid API calls and returns observations or errors.
Evaluator	Hidden target actions, target state, required outputs, and final trajectory	Determines whether the completed episode satisfies the task criteria.

The agent does not receive the hidden user instruction or the annotated goal. The simulated user does not see the private history of tool calls between the agent and the environment. This separation forces the agent to communicate relevant findings instead of assuming that the customer has observed the backend operations.

At each agent turn, the model can either:

• Send a natural-language message to the user.
• Issue a structured tool call against the environment.

Tool execution is deterministic for a fixed database state and argument list. User responses are stochastic because they are generated by a language model. The episode ends when the user simulator returns the reserved marker ###STOP###, or when an execution limit is reached. The original experiments allowed at most 30 agent actions, where both user-facing messages and tool calls counted toward the limit.

Single-Control Setting

The original τ-bench is a single-control environment. Only the evaluated agent has API access. The user can provide information and authorization in natural language but cannot directly alter the simulated world.

This design fits airline booking and online retail, where a support representative normally performs the backend operation. It is less suitable for technical-support situations in which the customer must change settings on a personal device. That limitation later motivated τ²-bench, where the simulated user receives its own tools.

Domain Policies

Every domain includes a policy document supplied to the agent as part of its system context. The policy describes:

• Database concepts and identifiers.
• Valid business procedures.
• Conditions for cancellations, exchanges, modifications, refunds, and compensation.
• Information the agent must collect or communicate.
• Cases requiring customer confirmation.
• Actions that must be refused.

Some constraints are enforced by API code. For example, a tool can reject a payment identifier that does not belong to the customer. Other constraints are intentionally left to the agent. An airline booking tool may accept a baggage count even though the correct charge depends on membership tier and cabin class. This distinction tests whether the agent reasons over the policy rather than relying exclusively on backend validation.

Original Domains and Tasks

The first release contains two synthetic customer-service domains.

Statistics reported in the original τ-bench paper

Property	τ-retail	τ-airline
Users	500	500
Products or flights	50 products	300 flights
Orders or reservations	1,000 orders	2,000 reservations
Write tools	7	6
Non-write tools	8	7
Evaluation tasks	115	50

τ-retail

The retail domain represents an online store. The agent can inspect customers, products, item variants, payment methods, and orders. Typical tasks include:

• Cancelling or modifying a pending order.
• Returning one or more delivered items.
• Exchanging an item for a different option of the same product.
• Updating a customer address.
• Locating product options that satisfy several preferences.
• Calculating prices, savings, refunds, or other information requested by the customer.

Products have variants represented by distinct item identifiers, such as combinations of size, color, material, capacity, or power source. Policies limit how orders can be changed: a pending order can normally be modified or cancelled only once, and a delivered order can normally be returned or exchanged only once. The agent must therefore collect all relevant items before invoking a one-time write operation.

Retail tasks can combine several goals. A customer may request a return and an exchange in the same conversation, prefer one fallback if both are impossible, and ask for the financial consequence of each option. Such tasks test database search, numerical reasoning, preference tracking, and correct sequencing.

τ-airline

The airline domain represents a carrier serving 20 United States cities. Its database contains flights with schedules and prices, customer profiles, payment methods, membership tiers, and reservations. Typical tasks include:

• Booking a new one-way or round-trip itinerary.
• Finding direct or one-stop flights that satisfy timing and budget constraints.
• Changing flights, dates, cabins, or passenger information.
• Cancelling reservations.
• Processing eligible refunds or compensation.
• Calculating baggage allowances and fees.

The airline policy is more complex and contains exceptions based on booking time, cabin class, insurance, membership tier, payment method, and flight status. A request that sounds simple may require a different procedure from the one proposed by the customer. For example, a basic-economy itinerary may be non-modifiable but still cancellable within a limited period, requiring the agent to explain the restriction and offer cancellation followed by rebooking.

The original experiments found τ-airline substantially more difficult than τ-retail, reflecting its denser policy dependencies and more consequential multi-step decisions.

Data and Task Construction

The original domains were created in three broad stages.

Stage 1: Manual Domain Design

The authors manually co-designed simplified but internally connected database schemas, tool interfaces, and policies. The goal was not to reproduce every feature of a commercial airline or retailer, but to create domains rich enough to require realistic coordination among dialogue, database reasoning, and policy interpretation.

Stage 2: LM-Assisted Synthetic Data Generation

After defining each schema, the authors created example records and used GPT-4 to generate code for sampling larger collections of synthetic users, products, orders, flights, and reservations. The generated programs were manually corrected where necessary.

The released data are therefore synthetic. They model realistic relationships and identifiers without using actual customer records.

Stage 3: Manual Task Annotation and Validation

Each task was written as a hidden instruction for the user simulator together with ground-truth evaluation information. The authors iteratively ran a GPT-4-Turbo function-calling agent, inspected the resulting trajectories, and revised ambiguous instructions. Some retail tasks were exercised in more than 40 trials during this process.

The central design objective was a unique permissible outcome. A task instruction specifies enough preferences and fallback behavior that, when combined with the policy and initial database, only one final result should satisfy the scenario. This restriction makes objective state comparison possible, although later audits found that some tasks did not fully achieve the intended consistency.

Modular Representation

The original release separates domain materials into four types:

Material	Typical representation	Purpose
Databases	JSON	Store users, products, orders, flights, reservations, and payment data.
Tools	Python functions and schemas	Read or modify the databases through structured calls.
Policies	Markdown text	Define domain rules and procedures supplied to the agent.
Tasks	JSON records	Define the hidden user scenario and the expected actions or outputs.

A simplified conceptual task record is:

{
  "instruction": "You are a specified customer. Ask to return one item and exchange another. If both cannot be completed, choose the option that saves the most money.",
  "actions": [
    {
      "name": "return_delivered_order_items",
      "arguments": {
        "order_id": "ORDER_ID",
        "item_ids": ["ITEM_ID"],
        "payment_method_id": "PAYMENT_ID"
      }
    }
  ],
  "outputs": ["required amount", "required comparison"]
}

The exact schema evolved in later repositories, but the separation between user instruction, environment state, executable tools, and evaluation criteria remains central to the benchmark family.

User Simulation

The original benchmark used gpt-4-0613 as the simulated customer. Its system prompt describes:

• The customer's identity.
• The initial request.
• Information that should be disclosed only when relevant.
• Preferences among valid alternatives.
• Fallback behavior when the preferred action is impossible.
• Conversational characteristics such as brevity or emotional state.

The simulator receives the complete natural-language conversation but not the agent's private tool-call history. After each agent message, it samples a new user utterance. In the original experimental protocol, the user model used temperature 1.0, while evaluated agents used temperature 0.0. This choice intentionally introduces dialogue variation even when the task instruction and initial database remain unchanged.

Language-model simulation offers several advantages:

• A textual scenario can produce many natural phrasings without manually scripting every turn.
• The user can answer clarifying questions that were not anticipated in a fixed dialogue tree.
• Repeated trials expose whether the agent is robust to small changes in phrasing and conversation order.
• Evaluation can be automated without recruiting a new group of human participants for every model.

It also introduces a second source of error. The user model can forget a preference, perform incorrect arithmetic, accept a recommendation without checking it, reveal information too early, introduce a new intent, or terminate prematurely. These failures complicate attribution because an unsuccessful episode may be caused by the evaluated agent, the simulator, the task annotation, or the environment.

Evaluation Methodology

State-Based Reward

The original reward is binary. It combines two checks:

Component	Success condition
r_action	The final database state matches the unique target outcome implied by the annotated write actions.
r_output	The agent's user-facing messages contain every required informational output, where the task defines such outputs.
Overall reward	reward = r_action × r_output; the episode passes only when both components equal 1.

The agent may make any number of read-only calls and may express the correct answer in different conversational forms. This allows diverse trajectories to receive the same result when they reach the same final state and communicate the required information.

State-based grading is faster and more reproducible than asking a human or an LLM judge to assess every conversation. It is also incomplete. The original paper explicitly notes that a reward of 1 can be necessary without being sufficient for ideal behavior. An agent could make a change before obtaining explicit authorization, communicate poorly, or take an inappropriate intermediate action and still pass if the final database and required output strings match the target.

The pass^k Reliability Metric

Conventional pass@k, commonly used in code generation, measures whether at least one of k attempts succeeds. τ-bench instead introduced pass^k, called “pass-hat-k” by the authors, to measure whether all k attempts succeed.

Metric	Event counted as success	Interpretation
pass@k	At least one of k trials passes.	Benefits from repeated attempts and measures solution discovery.
pass^k	Every one of k trials passes.	Penalizes inconsistency and measures repeated reliability.
pass^1	The single selected trial passes.	Equal to ordinary mean task success and to pass@1.

Suppose a task is run n times and succeeds c times. The paper uses the following task-wise estimators, written here without mathematical rendering for parser compatibility:

• pass^k = mean over tasks of comb(c, k) / comb(n, k).
• pass@k = 1 - mean over tasks of comb(n - c, k) / comb(n, k).

Here comb(a, b) is the number of ways to choose b trials from a. An evaluation must run at least k trials per task to report pass^k at that value of k.

The metric is averaged by task, so it is not generally equal to raising the global pass^1 score to the power k. It gives equal weight to easy and difficult tasks and directly reveals how many scenarios are solved consistently rather than intermittently.

Original Experimental Protocol

The main paper used the following configuration:

• At least three independent trials per task for the main comparison.
• Agent sampling temperature of 0.0.
• User-simulator temperature of 1.0.
• A maximum of 30 agent actions per episode.
• The domain policy as the agent's system prompt.
• Native function calling where supported.
• GPT-4 user simulation for the reported experiments.

Results depend on both the agent model and the user model. Changing the simulator, prompt, tool schema, maximum turn count, or task revision defines a materially different protocol.

Baseline Agents

The original study compared three simple agent strategies:

Strategy	Operation	Characteristics
Function calling	The model chooses between a natural-language response and a provider-native structured function call.	Best-performing baseline in the original study and least dependent on text parsing.
ReAct	The model emits a textual reasoning trace followed by a JSON-like action.	Provides explicit intermediate reasoning but depends on reliable formatting and parsing.
Act-only	The model emits the textual action without an explicit reasoning trace.	Used as an ablation to measure the value of intermediate reasoning in text-formatted agents.

Native function calling consistently outperformed the text-formatted strategies for the strongest models. ReAct generally outperformed Act-only, suggesting that explicit reasoning helped models map unfamiliar observations to actions, but it did not close the gap with provider-native tool calling. Adding a separate “think” function to a function-calling agent did not improve the reported results.

Original Results

The following table reproduces the historical pass^1 results from the original paper. Scores are percentages. The “average” is the unweighted mean of the two domain scores, not an average over all 165 task instances.

Original τ-bench results

Model	Agent interface	Retail	Airline	Domain average
GPT-4o	Function calling	61.2	35.2	48.2
GPT-4 Turbo	Function calling	57.7	32.4	45.1
GPT-4 32K	Function calling	56.5	33.0	44.8
GPT-3.5 Turbo	Function calling	20.0	10.8	15.4
Claude 3 Opus	Function calling	44.2	34.7	39.5
Claude 3 Sonnet	Function calling	26.3	27.6	27.0
Claude 3 Haiku	Function calling	19.0	14.4	16.7
Gemini 1.5 Pro	Function calling	21.7	14.0	17.9
Gemini 1.5 Flash	Function calling	17.4	26.0	21.7
Mistral Large	Function calling	30.7	22.4	26.6
Mixtral 8×22B	Function calling	17.7	31.6	24.7
Llama 3 70B Instruct	Text ReAct	14.8	14.4	14.6

GPT-4o was the strongest evaluated function-calling model, but its average remained below 50%. The gap between retail and airline supported the authors' view that agents struggled especially with dense, ad hoc policy dependencies.

Reliability deteriorated rapidly with repeated trials. Although the GPT-4o function-calling agent exceeded 60% pass^1 on retail, its retail pass^8 fell below 25%. This means that many tasks were solved only intermittently, even when the underlying scenario and initial database were unchanged.

A policy-removal ablation also showed that the strongest agent used the airline policy more than the weaker model did. Removing the policy reduced GPT-4o's airline score by more than 20 percentage points in that experiment, while the effect on retail was much smaller. The authors interpreted this as evidence that retail operations were closer to common-sense tool use, whereas airline success required greater reliance on explicit rules.

These figures are historical and should not be mixed with results from corrected τ³ task files, different user simulators, newer models, or alternative orchestration frameworks.

Failure Analysis

The paper manually analyzed one GPT-4o function-calling trajectory for each of the 115 retail tasks. Forty trajectories initially failed. Four failures were attributed to a typo or ambiguity in the user instruction and were corrected; the remaining 36 were classified as agent failures.

Failure class	Share of the 36 analyzed agent failures	Typical example
Wrong tool argument	33.3%	Selecting the correct operation but supplying an incorrect item, order, customer, product, or payment identifier.
Wrong information	22.2%	Omitting a requested tracking identifier, calculating the wrong total, or giving information that changes the user's decision incorrectly.
Wrong decision	25.0%	Choosing an operation that conflicts with the policy or failing to recognize a required procedure.
Partial resolution	19.4%	Completing only one part of a compound request or stopping before all affected records have been checked.

The analysis identified three broader capability bottlenecks.

Database Reasoning

Agents often selected the right tool but failed to identify the unique record satisfying a customer's constraints. Weaker baselines also hallucinated identifiers that were absent from the database. These failures combine retrieval, state tracking, numerical reasoning, and argument construction.

Policy Interpretation

Agents sometimes treated an API's technical acceptance as proof that an action was allowed. They could ignore a rule requiring all exchange items to be collected before a one-time tool call, misapply cancellation conditions, or fail to distinguish a default procedure from an exception.

Compound Requests and Memory

Tasks requiring several database writes were generally more difficult. Agents could forget a request stated early in the dialogue, complete an explicit action but omit an implied supporting action, or terminate after updating only one of several relevant records.

The same categories later became important in research on agent safeguards, process supervision, workflow constraints, and reinforcement learning for tool use.

Software and Reproducibility

The original implementation was released under the MIT License and includes domain data, policies, tools, task definitions, runner code, historical trajectories, and an automatic error-analysis utility.

The archived implementation uses a script-based interface such as:

python run.py \
  --agent-strategy tool-calling \
  --env retail \
  --model MODEL_NAME \
  --model-provider PROVIDER \
  --user-model USER_MODEL \
  --user-model-provider USER_PROVIDER \
  --user-strategy llm

The current framework uses the tau2 command even when described as τ³-bench. A minimal text evaluation follows the general form:

git clone https://github.com/sierra-research/tau2-bench
cd tau2-bench
uv sync

tau2 run \
  --domain retail \
  --agent-llm AGENT_MODEL \
  --user-llm USER_MODEL \
  --num-trials 4 \
  --task-split base

For reproducibility, a published result should specify at least:

• Repository and commit or release tag.
• Domain and task split.
• Agent model and exact version.
• User-simulator model and exact version.
• Number of trials per task.
• Agent and user sampling parameters.
• Maximum number of turns or actions.
• Tool-calling interface and prompt policy.
• Reward components and any additional judge.
• Handling of simulator errors, infrastructure failures, and retries.

The agent and user frequently come from mutable commercial APIs. A model alias can change behavior after an evaluation, so preserving complete trajectories and dated model identifiers is particularly important.

Evolution of the Benchmark Family

Version Timeline

Name	Initial release	Main scope	Distinguishing feature
τ-bench	June 2024; ICLR 2025	Retail and airline customer service	Single-control interaction, state-based grading, and pass^k.
τ²-bench	June 2025	Adds telecom technical support	Dual-control environment in which both agent and user operate tools.
τ³-bench 1.0.0	March 2026	Maintained multi-domain framework	Corrected tasks, knowledge retrieval, banking, and full-duplex voice evaluation.

The superscripted names denote successive generations of the project rather than powers of a benchmark score. The maintained repository retains the name tau2-bench for software continuity even though its version 1.0.0 release is presented as τ³-bench.

τ²-bench and Dual Control

τ²-bench generalizes the original setting by allowing the simulated user to call tools in a shared environment. Its telecom domain includes a backend customer-management system for the agent and a simulated phone for the user. The agent may change account-side settings, while the user can inspect signal state, toggle airplane mode, enable mobile data, or perform other device-side actions under the agent's guidance.

The telecom task generator combines atomic fault and solution components. Fifteen subtask groups across service, mobile-data, and multimedia-message issues produced 2,285 valid combinations, from which 114 evaluation tasks were sampled to balance intent and complexity. The benchmark therefore tests not only reasoning and tool use but also whether an agent can communicate an executable troubleshooting procedure to another actor.

The τ²-bench study also introduced more structured simulator affordances and user personas. Ablations separated pure reasoning from communication and coordination by comparing the normal dual-control setting with configurations in which the agent received all information or controlled all tools.

τ³-bench Task Corrections

Community use revealed annotation and evaluation problems in the original retail and airline tasks. An independent verification effort associated with the SABER project documented policy conflicts, invalid identifiers, impossible scenarios, and ambiguous success conditions.

The official τ³ update integrated corrections developed with that team and other contributors. The maintainers reported fixes to 27 airline tasks and 26 retail tasks, including:

• Expected actions that contradicted the published policy.
• Ambiguous user preferences or missing cancellation reasons.
• Payment methods or other constraints absent from the customer record.
• Exchanges that required replacing an item with the identical item identifier.
• Missing fallback behavior when a search returned no valid option.
• Loopholes that allowed an agent to reach the target state through an unintended procedure.

On the models rerun by the maintainers, corrected airline pass^1 increased by approximately 14 to 20 percentage points, while retail changes ranged from a small decrease to an increase of about 5.5 points. The size of these changes demonstrates that task revision is not a minor implementation detail: results from different benchmark versions can represent different ground truth.

τ-Voice

τ-Voice extends the framework to full-duplex spoken interaction. The user and agent can speak simultaneously, interrupt, yield, or wait, and the evaluation can introduce background noise, telephony compression, accents, and other audio conditions. The benchmark combines voice-interaction quality with the same type of verifiable grounded task completion used by the text environments.

The τ-Voice paper reports evaluations across 278 tasks. In its setup, a text reasoning agent achieved 85% task completion, while audio-native agents achieved 31–51% in clean conditions and 26–38% under more realistic audio conditions. These results indicate that strong text-based tool use does not automatically transfer to real-time spoken interaction.

τ-Knowledge and τ-Banking

τ-Knowledge adds environments in which the policy cannot be placed completely in the agent's prompt. Its τ-Banking domain contains 97 tasks and 698 unstructured policy and procedure documents spanning 21 product categories. Agents must retrieve relevant documents, interpret them, discover some tools through documentation, and combine knowledge with transactional actions.

The framework supports dense retrieval, BM25, reranking, terminal-based document search, complete-context baselines, and gold-document ablations. The paper reports that the strongest evaluated configuration reached approximately 25.5% pass^1, showing that retrieval, policy reasoning, and correct action sequencing remained difficult even for frontier models with large reasoning budgets.

Limitations and Criticism

State Equivalence Is Not Full Behavioral Correctness

Final-state comparison verifies that the desired records were produced, but it does not prove that the trajectory was safe, authorized, efficient, or policy compliant. Potential false positives include:

• Performing a write before obtaining confirmation.
• Making a prohibited intermediate change and later reversing it.
• Revealing private information to an unauthenticated user.
• Giving misleading explanations while still producing the expected output substring.
• Calling unnecessary tools or repeatedly requesting the same information.

The original reward can therefore certify a narrow task outcome rather than the complete quality of the interaction. Later framework versions added action checks and optional LLM-based conversation review, but these introduce their own definitions and possible judge errors.

Unique-Outcome Assumption

The benchmark is easiest to grade when every task has one accepted final state. Real customer-service problems often have several equally valid resolutions, especially when users can negotiate trade-offs or change preferences.

Constraining a scenario to one outcome improves reproducibility but can penalize a reasonable alternative. It can also encourage tasks whose wording contains unusually specific fallback rules designed for evaluation rather than naturally occurring conversations.

User-Simulator Reliability

A simulated customer is not a neutral test fixture. Its model, prompt, sampling temperature, and hidden instruction affect the difficulty and even the solvability of an episode.

A later τ²-bench study manually reviewed 100 airline and 50 retail conversations generated with GPT-4.1 as both agent and user simulator. It reported at least one simulator error in 47% of the airline conversations and 40% of the retail conversations; 13% and 12%, respectively, contained an error judged critical to task completion.

These figures do not imply that every affected agent result was invalid, because many errors were benign. They do show that an observed failure cannot always be attributed entirely to the evaluated agent. Automatic retry or hallucination-detection systems can reduce simulator noise but may selectively remove difficult conversations and must be documented.

Annotation and Ground-Truth Errors

The original tasks were manually validated, yet later audits found numerous mismatches among user instructions, policies, databases, expected actions, and reward criteria. Some tasks penalized agents for correctly refusing an action that the policy prohibited, while others expected identifiers or payment methods that did not exist in the initial state.

The 2026 official corrections materially changed model scores. Historical results should therefore be tied to an exact data revision and should not be interpreted as scores on the current task set.

Simplified and Synthetic Domains

The airline and retail systems are deliberately smaller than production services. They omit many operational realities, including:

• Authentication and fraud-detection workflows.
• Partial outages and eventually consistent services.
• Human escalation and supervisor approval.
• Legal and regional policy differences.
• Accessibility requirements.
• Privacy, security, and audit logging.
• Real inventory, pricing, and scheduling changes during a conversation.
• Customer emotion and behavior grounded in actual human participants.

The simplification makes deterministic evaluation possible but limits direct claims about production readiness.

Public and Static Tasks

Policies, task files, databases, tools, and historical trajectories are publicly available. This supports reproducibility and error discovery, but it also permits benchmark-specific training, prompt optimization, and memorization.

Repeated release of corrected tasks does not eliminate this risk. A robust evaluation should use private or newly generated tasks, held-out domain variants, or explicit disclosure of benchmark exposure during model development.

Binary Scoring and Limited Diagnostics

An episode that completes four of five required operations receives the same task reward as one that fails immediately. Binary scoring matches the deployment requirement that the entire customer request be resolved, but it discards useful information about partial progress.

Trajectory analysis, action-level checks, subtask assertions, and error classification can provide richer diagnostics. However, aggregate leaderboard scores may omit those details.

Dependence on Evaluation Protocol

Scores can change with:

• The user-simulator model.
• The order and wording of policy instructions.
• Native function calling versus text-formatted actions.
• Sampling temperature.
• Maximum turns.
• Retry handling.
• Context-window limits.
• Tool-schema descriptions.
• Provider-side safety filters.
• Treatment of mixed text-and-tool messages.
• The selected task split and number of trials.

A result labelled only “τ-bench” is therefore ambiguous.

Cost and Statistical Resolution

pass^k requires several trials for every task. Reliable estimation at larger values of k can require substantial inference cost, especially when both the agent and simulator use commercial frontier models and every turn repeats a long policy and tool specification.

The original airline set contains only 50 tasks. One changed task affects pass^1 by two percentage points before averaging across domains, and pass^k estimates can have high variance when only a few trials are available.

Reliability Metric Includes More Than Agent Reliability

A falling pass^k curve can reflect variation in the agent, the user simulator, external model APIs, tool infrastructure, or task ambiguity. It is an end-to-end system-reliability metric, not a pure measurement of model weights.

This is valuable for deployment-oriented testing, but it complicates scientific attribution. Controlled simulator seeds, human validation, no-user ablations, and deterministic replay can help separate causes.

Original Single-Control Restriction

In the original benchmark, the user cannot manipulate the world. Many support tasks require the customer to inspect or change a device, upload a document, approve a payment, or perform physical steps. τ²-bench addresses this gap for telecom troubleshooting, but dual-control evaluation introduces additional simulator and coordination complexity.

External links

• τ-bench paper on OpenReview
• Original τ-bench repository
• Current τ³-bench framework repository
• Official τ-bench website and leaderboard
• τ²-bench paper
• τ-Voice paper
• τ-Knowledge paper

See also

• AI agent
• Function calling
• ReAct
• Large language model evaluation

Literature

• Yao, S. et al. (2025). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. ICLR 2025. arXiv:2406.12045.
• Barres, V. et al. (2025). τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment. arXiv:2506.07982.
• Cuadron, A. et al. (2025). SABER: Small Actions, Big Errors — Safeguarding Mutating Steps in LLM Agents. arXiv:2512.07850.
• Ray, S. et al. (2026). τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains. arXiv:2603.13686.
• Shi, Q. et al. (2026). τ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge. arXiv:2603.04370.
• Yao, S. et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023. arXiv:2210.03629.
• Liu, X. et al. (2024). AgentBench: Evaluating LLMs as Agents. ICLR 2024. arXiv:2308.03688.