BIG-bench (benchmark)

BIG-bench (an acronym for Beyond the Imitation Game benchmark) is a large-scale collection of tasks (benchmark) created to evaluate the capabilities and limitations of large language models (LLMs). The project was developed in 2021–2022 through a collaborative effort of over 450 researchers from 132 organizations under the aegis of Google.

The benchmark includes 204 diverse tasks covering a wide range of fields: linguistics, mathematics, programming, common sense reasoning, biology, physics, and the evaluation of social biases. The primary goal of BIG-bench is to go beyond the "imitation game" (the Turing test) and test models on tasks considered difficult or unsolvable for existing architectures. The benchmark is designed not only to measure current abilities but also to extrapolate their future capabilities as scale increases.

Development and Structure

BIG-bench was initiated by a group of researchers from Google, who organized an open call for task submissions from the scientific community. As a result, the final set includes 204 tasks from dozens of independent teams. Each task was designed to be a challenge for LLMs and has its own format and evaluation metric (e.g., multiple-choice accuracy, scoring of freely generated responses).

The tasks range from standard academic questions to unconventional puzzles, such as:

• Solving mathematical and logical problems.
• Understanding emoji sequences.
• Solving chess problems from a text description.
• Identifying social stereotypes in model responses.

The entire benchmark and its code are publicly available on GitHub, allowing researchers to test new models and propose additional tasks.

Model Evaluation and Human Baseline

In the original 2022 paper, large-scale testing was conducted on models including the GPT family from OpenAI, as well as dense and sparse models from Google, such as PaLM and Switch Transformers.

To compare the results, a human baseline was established. Expert raters performed all tasks using the resources available to them. Two metrics were defined:

• Average expert score: approximately 45/100 on a conditional normalized scale.
• Best expert score: approximately 80/100 (when at least one expert solved the task optimally).

Even the largest models of that time performed significantly worse than humans. For example, the best of them (including GPT-3) scored only around 15/100, highlighting the difficulty of the tasks and the significant potential for future progress.

Key Results and Findings

Analysis of the results on BIG-bench revealed several key patterns:

1. The effect of scale. Model accuracy increases with the number of parameters across almost all task categories.
2. Emergent abilities. On many tasks, model performance remains at the level of random guessing for a long time, but after reaching a certain "critical" scale, there is a sharp jump in quality. This phenomenon is known as emergent behavior.
3. Social biases. As the size of the model increases, the level of social stereotypes learned from the training data can also increase. However, it was shown that proper prompt formulation (prompting) can mitigate this effect.

Evolution of the Benchmark

As models became more powerful, some BIG-bench tasks ceased to be challenging. This led to the creation of more difficult subsets.

Big-bench Hard (BBH)

In 2022, researchers identified the 23 most difficult tasks on which all models initially performed below the average human level. This set was named BIG-bench Hard (BBH). Experiments showed that using the Chain-of-Thought (CoT) technique—where the model generates a chain of reasoning before giving an answer—dramatically improves performance. With CoT, the PaLM model (540 billion parameters) was able to surpass the average human score on 10 of the 23 tasks, and Codex (a version of GPT-3) did so on 17 of the 23.

Big-bench Extra Hard (BBEH)

By 2024, when even the tasks in BBH were being solved by state-of-the-art models, the next stage was proposed: BIG-bench Extra Hard (BBEH). The authors from DeepMind replaced each of the 23 BBH tasks with a new one, similar in reasoning type but significantly more difficult. Initial tests on BBEH showed that even the most powerful contemporary LLMs are far from solving them, providing a long-term challenge for future models.

Big-bench Lite (BBL)

For faster and less resource-intensive testing, a lightweight version was created: BIG-bench Lite (BBL). It consists of a sample of 24 tasks that reflect the diversity of the full set. BBL allows developers to quickly evaluate their models and compare them on a public leaderboard.

External Links

• Official BIG-bench repository on GitHub
• BIG-bench page on Papers With Code

Further Reading

• Liang, P. et al. (2022). Holistic Evaluation of Language Models (HELM). arXiv:2211.09110.
• Chang, Y. et al. (2023). A Survey on Evaluation of Large Language Models. arXiv:2307.03109.
• Ni, S. et al. (2025). A Survey on Large Language Model Benchmarks. arXiv:2508.15361.
• Biderman, S. et al. (2024). The Language Model Evaluation Harness (lm-eval): Guidance and Lessons Learned. arXiv:2405.14782.
• Kiela, D. et al. (2021). Dynabench: Rethinking Benchmarking in NLP. arXiv:2104.14337.
• Ma, Z. et al. (2021). Dynaboard: An Evaluation‑As‑A‑Service Platform for Holistic Next‑Generation Benchmarking. arXiv:2106.06052.
• Goel, K. et al. (2021). Robustness Gym: Unifying the NLP Evaluation Landscape. arXiv:2101.04840.
• Xu, C. et al. (2024). Benchmark Data Contamination of Large Language Models: A Survey. arXiv:2406.04244.
• Liu, S. et al. (2025). A Comprehensive Survey on Safety Evaluation of LLMs. arXiv:2506.11094.
• Chiang, W.-L. et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv:2403.04132.
• Boubdir, M. et al. (2023). Elo Uncovered: Robustness and Best Practices in Language Model Evaluation. arXiv:2311.17295.
• Huang, L. et al. (2023). A Survey on Hallucination in Large Language Models. arXiv:2311.05232.