Big-Bench Hard

BIG-Bench Hard (BBH) is a benchmark suite designed to evaluate the multi-step reasoning capabilities of large language models (LLMs). It consists of 23 challenging tasks selected from the broader BIG-Bench collaborative benchmark. BBH was introduced by Mirac Suzgun and colleagues in the 2022 paper Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them, which was published in the Findings of ACL 2023.

The tasks were selected because no language model evaluated in the original BIG-Bench study had surpassed the corresponding average reported human-rater score. The purpose of BBH was to determine whether these tasks were fundamentally beyond the capabilities of contemporary models or whether their apparent difficulty resulted partly from inadequate prompting methods.

BBH became particularly influential in research on chain-of-thought prompting (CoT), in-context learning, emergent model capabilities, and inference-time reasoning. The original experiments showed that sufficiently large models could improve substantially when demonstrations contained explicit intermediate reasoning steps. The public release contains 6,511 evaluation examples, task definitions, manually constructed chain-of-thought prompts, and outputs produced by the Codex model.

Origin and Construction

BBH was derived from BIG-Bench (Beyond the Imitation Game Benchmark), a collaboratively developed evaluation suite containing more than 200 text-based tasks in areas such as natural-language understanding, mathematics, commonsense reasoning, symbolic manipulation, and question answering.

BIG-Bench was originally intended to contain tasks beyond the capabilities of contemporary language models. By the time BBH was created, however, the strongest model reported in the BIG-Bench study had already exceeded the average human-rater result on approximately 65% of its tasks. The BBH authors therefore focused on the remaining difficult tasks and applied a sequence of filtering criteria.

Filtering procedure used to construct BIG-Bench Hard

Remaining tasks	Filtering stage
209	All BIG-Bench task entries considered by the BBH authors.
187	Tasks with more than three subtasks were removed.
130	Tasks with fewer than 103 examples were removed. A minimum of three examples was needed for few-shot demonstrations and at least 100 for evaluation.
85	Tasks without reported human-rater baselines were removed.
78	Tasks that could not be evaluated through multiple-choice accuracy or exact matching were removed.
36	Tasks on which the best previously reported model exceeded the average human-rater score were removed.
23	Thirteen additional tasks judged to be outside the scope of the study were removed after manual inspection.

The manually excluded tasks required highly specialized knowledge, involved unusually long or impractical problem-solving procedures, contained ambiguous formulations, or were considered unsuitable for evaluation with chain-of-thought prompting. Examples discussed by the authors included chess-state tracking, extremely long text classification, and ambiguous moral-permissibility judgments.

The final benchmark contains 23 conceptual tasks but 27 task files. This difference arises because two tasks have three difficulty variants:

• Logical Deduction contains variants with three, five, and seven objects.
• Tracking Shuffled Objects contains variants with three, five, and seven objects.

For 24 of the 27 task files, the authors randomly selected 250 evaluation examples. The three exceptions use all available evaluation examples:

• Causal Judgement — 187 examples.
• Penguins in a Table — 146 examples.
• Snarks — 178 examples.

Together, these files contain 6,511 evaluation examples. Three additional examples from each task were reserved for few-shot prompting and were not included in the evaluation set.

The word “Hard” is therefore relative to a particular historical evaluation setting. BBH was not formed by measuring an intrinsic or permanent level of task difficulty; it was selected according to the performance of models and human raters available during the construction of the benchmark.

Tasks and Capabilities

The BBH authors divided the suite into 11 algorithmic tasks and 12 natural-language or world-knowledge tasks. An algorithmic task was defined as one that could, after its instructions were understood, be solved by a deterministic rule-based procedure without requiring additional natural-language knowledge.

Group	Task	Capability evaluated
Algorithmic	Boolean Expressions	Evaluating expressions containing Boolean constants and the operators and, or, and not.
Natural language	Causal Judgement	Determining how a typical person would answer a causal question about a short moral, intentional, or counterfactual scenario.
Natural language	Date Understanding	Inferring a calendar date from several statements expressing temporal relationships.
Natural language	Disambiguation QA	Resolving an ambiguous pronoun or determining that the available information does not identify a unique antecedent.
Algorithmic	Dyck Languages	Predicting the closing brackets required to complete an incomplete Dyck-4 sequence.
Natural language	Formal Fallacies	Determining whether a conclusion logically follows from informally expressed premises, with particular emphasis on negation and syllogistic reasoning.
Algorithmic	Geometric Shapes	Identifying the geometric figure described by an SVG path containing several drawing commands.
Natural language	Hyperbaton	Choosing the English sentence that follows the conventional ordering of multiple adjectives.
Algorithmic	Logical Deduction	Inferring the order or relative position of three, five, or seven objects from a collection of constraints.
Natural language	Movie Recommendation	Selecting a film most similar to a group of movies that a hypothetical user has liked.
Algorithmic	Multi-Step Arithmetic	Evaluating arithmetic expressions containing several nested operations.
Algorithmic	Navigate	Following a sequence of movement and rotation instructions and determining whether an agent returns to its starting position.
Algorithmic	Object Counting	Counting objects belonging to a requested semantic category in a natural-language inventory.
Natural language	Penguins in a Table	Reading and updating a small table of penguins and answering questions about their attributes.
Natural language	Reasoning about Colored Objects	Inferring the color, position, or identity of objects described in a short scene.
Natural language	Ruin Names	Recognizing a humorous one-character modification of the name of a film, band, artist, or other cultural entity.
Natural language	Salient Translation Error Detection	Classifying the principal error in an English translation of a German source sentence.
Natural language	Snarks	Distinguishing a sarcastic sentence from a closely related non-sarcastic alternative.
Natural language	Sports Understanding	Determining whether a statement about a sport, athlete, action, or competition is plausible.
Algorithmic	Temporal Sequences	Identifying a possible time interval for an event from a schedule and a series of temporal constraints.
Algorithmic	Tracking Shuffled Objects	Tracking the ownership or position of three, five, or seven objects after a sequence of exchanges.
Algorithmic	Web of Lies	Inferring the truth values of statements made by people who may be lying or telling the truth.
Algorithmic	Word Sorting	Arranging a list of words in alphabetical order.

The task descriptions and examples are stored in the official repository. Although most tasks use English-language inputs, the suite covers heterogeneous abilities: arithmetic, formal logic, spatial reasoning, temporal reasoning, coreference resolution, grammar, humor, translation analysis, factual knowledge, and social judgments.

Data Format and Release

The official BBH release stores the evaluation data in 27 JSON files inside the bbh directory. Each file has a compact structure consisting of two top-level fields:

Field	Description
canary	A fixed warning string and globally unique identifier intended to help detect accidental inclusion of benchmark data in training corpora.
examples	An array containing the evaluation examples for the task.
input	The problem statement, instructions, and answer choices where applicable.
target	The expected answer used for exact-match evaluation.

A simplified representation of a task file is:

{
  "canary": "benchmark contamination warning and identifier",
  "examples": [
    {
      "input": "Task-specific problem statement",
      "target": "Expected answer"
    }
  ]
}

The benchmark is primarily an evaluation collection rather than a conventional training, validation, and test dataset. Demonstrations used for few-shot evaluation are distributed separately in the cot-prompts directory. The repository also includes generated outputs from code-davinci-002, allowing later researchers to inspect the model responses used in the original experiments.

The contamination canary does not technically prevent benchmark examples from entering model-training corpora. It provides a recognizable sequence that dataset curators or auditors can search for when examining training data or model outputs.

Evaluation Methodology

Answer-Only and Chain-of-Thought Prompting

The original study compared two few-shot prompting configurations:

• Answer-only prompting. Each demonstration contained a problem and its final answer without an explicit reasoning trace.
• Chain-of-thought prompting. Each demonstration contained the problem, intermediate reasoning steps, and the final answer.

Both configurations included task instructions and, for multiple-choice problems, the available answer options. The authors manually constructed three chain-of-thought demonstrations for each task. The reasoning annotations began with the phrase Let's think step by step, following research on zero-shot and few-shot reasoning prompts.

Chain-of-thought prompting was intended to make intermediate computation explicit. Instead of mapping a complex input directly to a short label, a model could decompose the problem into smaller operations, preserve temporary states, and then generate the required answer.

Answer Extraction and Accuracy

In the original experiments, the models used greedy decoding with a sampling temperature of zero. For multiple-choice tasks, all options were placed in the prompt simultaneously and the model generated a textual response. This differs from evaluation methods that independently score the conditional probability of each answer option.

The evaluator extracted the final answer from the generated text, typically after a phrase such as the answer is, and compared it with the reference target using exact matching. Each task score was calculated as:

$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}=\frac{\text{number of correct responses}}{\text{number of evaluated examples}}$

The three Logical Deduction variants were averaged into one conceptual task score, as were the three Tracking Shuffled Objects variants. The headline BBH score was then calculated as the unweighted mean of the 23 conceptual task scores. Consequently, every task contributed equally to the aggregate result even though the task files contained different numbers of examples.

Because the answer spaces differ substantially, the probability of succeeding through random guessing is not uniform across BBH. Some tasks are binary, while others have three, four, five, or more possible outputs. The original paper reported an aggregate random baseline of approximately 25.7% under its task-level averaging procedure.

Comparability of Results

Later evaluations frequently modify the original protocol. Relevant variables include:

• Zero-shot versus three-shot prompting.
• Direct-answer versus chain-of-thought instructions.
• Manually written versus automatically generated demonstrations.
• Greedy decoding versus stochastic sampling.
• Majority voting or self-consistency over multiple responses.
• Maximum context and generation lengths.
• Chat templates and system instructions.
• Answer-extraction regular expressions.
• Case, whitespace, punctuation, and label normalization.
• Use of tools, code execution, or external retrieval.

A reported “BBH score” is therefore not fully interpretable without a description of the prompt, decoding procedure, task aggregation, and answer parser. Results obtained with materially different protocols should not be treated as entries in a single strictly comparable leaderboard.

Original Results and Findings

The initial study evaluated several sizes of PaLM, InstructGPT, and Codex. The principal results for the largest model in each family are summarized below.

Results reported in the original BIG-Bench Hard study

System or baseline	Answer-only average	Chain-of-thought average	Tasks above average human-rater answer-only → CoT
Average reported human-rater	67.7%	—	—
Maximum reported human-rater	94.4%	—	23 / 23
PaLM 540B	52.3%	65.2%	6 / 23 → 10 / 23
InstructGPT (text-davinci-002)	51.8%	68.4%	4 / 23 → 15 / 23
Codex (code-davinci-002)	56.6%	73.9%	5 / 23 → 17 / 23

Codex with chain-of-thought prompting achieved the strongest aggregate result in the study. Its score of 73.9% exceeded the average reported human-rater baseline of 67.7%, but remained more than 20 percentage points below the maximum reported human-rater baseline of 94.4%. The model surpassed the average human score on 17 of the 23 tasks, compared with only five tasks under answer-only prompting.

The effect of chain-of-thought prompting differed by task family. For Codex, it increased the average score on the 11 algorithmic tasks from 45.9% to 74.4%, an improvement of 28.5 percentage points. On the 12 natural-language tasks, the improvement was smaller, from 66.4% to 73.5%.

The largest task-level gains for Codex were reported on:

• Tracking Shuffled Objects — an increase of 60.4 percentage points.
• Multi-Step Arithmetic — an increase of 46.4 percentage points.
• Navigate — an increase of 46.0 percentage points.
• Temporal Sequences — an increase of 19.8 percentage points.

These tasks require maintaining and updating an intermediate state across several operations. The authors argued that explicit reasoning traces helped sufficiently capable models decompose such problems into sequential steps.

The benefit of chain-of-thought prompting was also dependent on model scale. For small InstructGPT and PaLM models, chain-of-thought produced little improvement or sometimes reduced accuracy. The positive difference grew at larger scales. The authors described this as an emergent prompting effect and highlighted Multi-Step Arithmetic, Tracking Shuffled Objects, and Web of Lies as tasks whose performance remained close to random under direct prompting but rose sharply with chain-of-thought at larger model sizes.

Chain-of-thought was not universally beneficial. In the original experiments, it reduced or failed to improve results on tasks including Causal Judgement, Ruin Names, and Snarks. These tasks rely more heavily on social interpretation, humor, cultural knowledge, or pragmatic language understanding than on a deterministic sequence of symbolic operations.

Adoption and Saturation

BBH subsequently became a widely reported general-reasoning benchmark in model cards, technical reports, open evaluation harnesses, and academic studies. Its compact size, public prompts, diverse task suite, and single aggregate score made it easier to run than the complete BIG-Bench collection.

The benchmark also progressed rapidly toward saturation. By 2024, several frontier models were reported to achieve aggregate BBH scores above 90% under commonly used few-shot protocols. A 2025 analysis concluded that state-of-the-art models had reached near-perfect accuracy on many individual tasks, reducing BBH's usefulness for differentiating the strongest systems.

Saturation does not make every BBH task valueless. Task-level error patterns can still reveal weaknesses in symbolic state tracking, formal logic, temporal inference, or linguistic interpretation. However, a near-ceiling aggregate score provides limited resolution for ranking frontier models and can conceal the small number of tasks on which meaningful differences remain.

Limitations and Criticism

Historically Relative Definition of Difficulty

BBH's selection procedure defines difficulty relative to the models, prompts, and human baselines available during the BIG-Bench evaluation. A task entered BBH because previously reported models failed to surpass its average human-rater score—not because the task satisfied an independent measure of cognitive complexity.

The procedure also creates a form of selection or survivorship bias. Tasks that were already solved by earlier models were removed, but 13 tasks that remained extremely difficult were also excluded because the authors considered them unsuitable for the intended chain-of-thought study. BBH therefore represents a tractable middle region of the original benchmark rather than the complete set of its hardest tasks.

Human-Rater Baselines

The reported human baselines should not be interpreted as definitive estimates of ordinary or expert human ability. The original BIG-Bench human evaluation took place over an extended period during which the formatting and content of some tasks changed. Raters answered subsampled task sets, worked for limited periods, and were sometimes permitted to use external resources such as the Internet.

Some instructions were also difficult to follow or could have been formulated more clearly. The BBH authors consequently warned that exceeding the average human-rater score should not be treated as proof of general language understanding or human-level reasoning.

Public Data and Training Contamination

BBH is a static, publicly available benchmark. Its task files, prompts, labels, and example model outputs have been accessible online since the benchmark was released. These materials may be copied into web crawls, benchmark compilations, synthetic training sets, or instruction-tuning corpora.

The repository includes a canary string warning that benchmark data should not appear in training corpora. The canary can assist with auditing, but it cannot detect paraphrased examples, separately copied task contents, or training data from the original BIG-Bench sources. A high score from a model with undisclosed training data may therefore reflect an unknown combination of reasoning ability, familiarity with task templates, and direct or indirect memorization.

Heterogeneous Chance Baselines and Shortcuts

The tasks do not share a common answer format or random baseline. Eight of the 23 conceptual tasks use binary labels, and another five have no more than five answer choices. High chance accuracy can therefore contribute materially to the aggregate score, particularly when models exploit answer-frequency or formatting patterns.

Later analysis also identified opportunities for task-specific shortcuts. Some problems can be solved by recognizing recurring templates or superficial regularities rather than carrying out the intended reasoning procedure. For example, an SVG-based geometry task may reward familiarity with common path patterns, while repeated linguistic templates may allow a model to infer the label distribution.

The authors of BIG-Bench Extra Hard characterized BBH inputs as comparatively short, with a task-level macro-average of approximately 700 characters, and noted that many examples require only a small number of reasoning operations. These properties contributed to the rapid improvement and eventual saturation of the benchmark.

Aggregate Score and Task Weighting

The unweighted 23-task average gives the same importance to every conceptual task. A task containing 146 evaluation examples contributes as much as a collection of three 250-example subtasks. This makes the aggregate score easy to interpret as an average over capabilities, but it is not equivalent to accuracy over all 6,511 examples.

The single average also combines qualitatively different abilities, including arithmetic, sarcasm detection, cultural knowledge, formal deduction, and spatial tracking. Two models can receive the same aggregate score while succeeding on substantially different task groups. Reporting per-task results, algorithmic and natural-language subaverages, and confidence intervals provides a more informative evaluation.

Prompt and Parser Sensitivity

BBH scores depend on details that are not properties of the model alone. The wording of instructions, ordering of demonstrations, inclusion of answer choices, use of reasoning cues, and maximum generation length can all affect performance. Exact-match evaluation additionally depends on whether the parser correctly extracts a valid answer from explanatory text.

A model may reason correctly but lose credit because it returns an unexpected label format. Conversely, a model may produce an invalid rationale followed by the correct short answer and receive full credit. Standard BBH accuracy therefore measures final-answer agreement rather than the faithfulness or logical validity of the generated reasoning trace.

Language and Cultural Scope

BBH is predominantly an English-language benchmark. Several tasks rely on English grammar, English-language humor, Western entertainment, familiar sports, or social judgments about intention and causation. Salient Translation Error Detection additionally assumes knowledge of German-to-English translation.

The suite does not comprehensively evaluate multilingual reasoning, long-context reasoning, advanced mathematics, programming, formal proof construction, multimodal perception, interactive planning, or real-world tool use. Performance on BBH should therefore be interpreted as evidence about a limited collection of text-based reasoning tasks rather than a general measure of intelligence.

BIG-Bench Extra Hard

To restore discriminative difficulty after BBH began to saturate, researchers introduced BIG-Bench Extra Hard (BBEH), published at ACL 2025. BBEH replaces each of the 23 BBH tasks with a newly constructed counterpart designed to test a similar underlying capability while requiring longer inputs, more reasoning steps, larger output spaces, or resistance to simple shortcuts.

The replacement tasks preserve the broad diversity of BBH but are not merely larger samples of the same examples. They were redesigned to increase compositional and computational complexity. In the original BBEH evaluation, the strongest general-purpose model achieved 23.9%, while the strongest reasoning-specialized model achieved 54.2%, leaving substantially more room for improvement than contemporary BBH evaluations.

BBEH is consequently positioned as a successor rather than a revised scoring system for BBH. Results on the two benchmarks are not directly comparable because their questions, answer spaces, input lengths, and difficulty distributions differ.

External links

• Official BIG-Bench Hard repository
• Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
• Official BIG-Bench repository
• Official BIG-Bench Extra Hard repository
• BIG-Bench Extra Hard paper

See also

• BIG-Bench
• Chain-of-thought prompting
• In-context learning
• Prompt engineering

Literature

• Srivastava, A. et al. (2023). Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models. Transactions on Machine Learning Research. [9].
• Wei, J. et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS. arXiv:2201.11903.
• Kojima, T. et al. (2022). Large Language Models are Zero-Shot Reasoners. NeurIPS. arXiv:2205.11916.
• Suzgun, M. et al. (2023). Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. Findings of ACL 2023. [10].
• Wei, J. et al. (2022). Emergent Abilities of Large Language Models. Transactions on Machine Learning Research. [11].
• Kazemi, M. et al. (2025). BIG-Bench Extra Hard. ACL 2025. [12].