Chinchilla (language model)

Chinchilla is a large language model (LLM) developed by the DeepMind research team and introduced in March 2022. The model contains approximately 70 billion parameters and was trained on a text corpus of 1.4 trillion tokens.

The key feature of Chinchilla is its compute-optimal approach to training. Unlike previous models where the main focus was on increasing the number of parameters, Chinchilla was created based on the hypothesis that both model size and the volume of training data must be scaled proportionally. Thanks to this approach, Chinchilla demonstrated superior performance over significantly larger models, such as Gopher (280 billion parameters) and GPT-3 (175 billion), across a wide range of language tasks.

Background and History

The development of Chinchilla was the result of research into LLM scaling conducted at DeepMind, building upon the Gopher family of models. The Gopher model, introduced in 2021, had 280 billion parameters but was trained on a relatively small corpus of 300 billion tokens. At the time, the prevailing approach in the industry was that model performance grew primarily by increasing their size (number of parameters), while the amount of data remained relatively constant. This view drew on earlier scaling laws by Kaplan et al. (2020), which Chinchilla would later revise by showing that training data had been substantially undervalued.

The Compute-Optimal Training Hypothesis

DeepMind researchers hypothesized that many large models, including Gopher, were undertrained relative to their size. They did not achieve the maximum possible quality for a given computational budget because they lacked sufficient training data.

The core of the hypothesis was that for optimal use of computational resources, model size and the volume of training data should be increased proportionally to each other. In other words, doubling the number of model parameters requires approximately doubling the number of training tokens. This conclusion contradicted previous research that had overestimated the value of increasing model size, as those studies were conducted with a fixed amount of data.

To test this hypothesis, the DeepMind team conducted extensive experiments, training over 400 models ranging from 70 million to over 16 billion parameters on datasets of 5 to 500 billion tokens. The results confirmed that parallel scaling is the optimal strategy. Based on these findings, the Chinchilla model was developed as a practical test of this new paradigm.

Architecture and Training

Architectural Features

Chinchilla belongs to the family of autoregressive transformers and uses essentially the same architecture as Gopher: a decoder-only transformer broadly similar to GPT-2/GPT-3, but adopting RMSNorm instead of LayerNorm and relative positional encodings instead of absolute ones. Its main differences from Gopher are aimed at reducing size while maintaining network depth:

• Parameters: ~70 billion parameters, distributed across 80 layers.
• Model Width: The number of self-attention heads was reduced to 64 (compared to 128 in Gopher), and the internal layer dimension to 8192 (compared to ~16384 in Gopher).
• Optimizer: It uses AdamW instead of Adam, which improves convergence on large datasets.

This architecture allowed Chinchilla to maintain the same network depth as Gopher but with a significantly smaller number of parameters, reducing memory and computational requirements.

Scaling and Training Data

To validate the hypothesis, Chinchilla was trained with the same computational budget as Gopher, but with resources reallocated in favor of data. The 70-billion-parameter model was trained on a corpus of 1.4 trillion tokens, which is approximately four times the amount of data used for Gopher.

This ratio, approximately 20 tokens for every parameter, became known as the Chinchilla point (or "Chinchilla-optimal") and serves as a benchmark for the compute-optimal training of modern LLMs. The experiment confirmed that Chinchilla, being trained closer to this optimal limit, was able to realize its potential more fully than undertrained, albeit larger, models.

Results and Performance

Across a wide range of standard benchmarks, Chinchilla demonstrated a significant advantage over previous models. It confidently outperformed not only Gopher but also other state-of-the-art LLMs of its time, including OpenAI's GPT-3 (175 billion parameters) and Megatron-Turing NLG (530 billion parameters).

The most indicative result came from the comprehensive MMLU (Measuring Massive Multitask Language Understanding) benchmark, which evaluates knowledge and reasoning across hundreds of diverse tasks. Chinchilla achieved an average 5-shot accuracy of 67.6%, setting a new record for models of its class and improving on Gopher's 60.0% by 7.6 percentage points; on the same benchmark GPT-3 scored 43.9%.

In addition to its high performance, Chinchilla also proved to be economical to use. Its smaller size (70 billion vs. 175+ billion for its counterparts) means that it requires significantly fewer computational resources for inference and fine-tuning, simplifying its practical application.

Significance and Impact

The Chinchilla research has had a fundamental impact on approaches to training large language models.

• Chinchilla scaling laws: The identified optimal ratio between model size and data volume became a de facto standard and a guide for subsequent developments in the industry, later refined to also account for inference costs (see below).
• Shift in focus from size to data: The work encouraged the industry to pay more attention to creating, cleaning, and expanding training corpora, rather than just indiscriminately increasing the number of parameters.
• Application in multimodal systems: Chinchilla was used as the core language component in DeepMind's multimodal model Flamingo, which is capable of understanding images and text.

Although the Chinchilla model itself was not publicly released, its concepts and the results published in the research paper changed the development trajectory of the entire LLM field, charting a path toward more efficient and balanced growth in artificial intelligence capabilities.

Later Developments and Critiques

Inference-aware scaling ("beyond Chinchilla")

A widely noted limitation of the Chinchilla scaling laws is that they optimize only for the compute cost of training and ignore the cost of inference. Because a smaller model is cheaper to run, a model that will serve many requests can be worth training well beyond the 20-tokens-per-parameter ratio. Meta's LLaMA models (2023) illustrated this in practice: their smaller models were deliberately trained on far more data than Chinchilla would deem optimal — the 7B model on about 1 trillion tokens (~140 tokens per parameter) — trading extra training compute for lower inference cost. This idea was formalized by Sardana et al. (2024), who modified the Chinchilla laws to account for inference demand and argued that models expected to serve large request volumes should be trained "smaller and longer than Chinchilla-optimal"; they also observed that model quality continued to improve as the token-to-parameter ratio was pushed to extreme values. Later models pushed the ratio much further still — LLaMA 3's 8-billion-parameter model was trained on 15 trillion tokens — reflecting a broad shift toward "over-training" relatively small models.

Replication of the scaling law (2024)

In April 2024, researchers at Epoch AI (Besiroglu, Erdil, Barnett, and You) published a replication of the third of the three methods Hoffmann et al. used to estimate their scaling law — the parametric fit to the loss surface. They reported that the paper's published coefficients fit the reconstructed data poorly and came with implausibly narrow confidence intervals, so narrow that they would require far more experiments than were actually run. One of the original authors subsequently acknowledged an optimizer-configuration error that explained the overly tight intervals. Importantly, the replication did not overturn the central conclusion: the re-derivation reconciled the third method with the other two, and the finding that parameters and training tokens should be scaled at roughly equal rates (about 20 tokens per parameter) has held up in later work, though the exact optimum is now understood to depend on the quality of the training data.

External links

• Chinchilla (language model) — Wikipedia

Literature

• Hendrycks, D.; Gimpel, K. (2016). Gaussian Error Linear Units (GELUs). arXiv:1606.08415.
• Loshchilov, I.; Hutter, F. (2017). Decoupled Weight Decay Regularization. arXiv:1711.05101.
• Shoeybi, M.; et al. (2019). Megatron‑LM: Training Multi‑Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053.
• Kaplan, J.; et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361.
• Brown, T. B.; et al. (2020). Language Models are Few‑Shot Learners. arXiv:2005.14165.
• Rajbhandari, S.; et al. (2020). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. arXiv:1910.02054.
• Press, O.; et al. (2021). Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. arXiv:2108.12409.
• Rae, J.; et al. (2021). Scaling Language Models: Methods, Analysis & Insights from Training Gopher. arXiv:2112.11446.
• Hoffmann, J.; et al. (2022). Training Compute‑Optimal Large Language Models. arXiv:2203.15556.
• Alayrac, J.‑B.; et al. (2022). Flamingo: A Visual Language Model for Few‑Shot Learning. arXiv:2204.14198.
• Hendrycks, D.; et al. (2020). Measuring Massive Multitask Language Understanding. arXiv:2009.03300.
• Touvron, H.; et al. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971.
• Sardana, N.; et al. (2024). Beyond Chinchilla‑Optimal: Accounting for Inference in Language Model Scaling Laws. arXiv:2401.00448.
• Besiroglu, T.; et al. (2024). Chinchilla Scaling: A Replication Attempt. arXiv:2404.10102.