BLOOM (language model)

From Systems Analysis Wiki
Jump to navigation Jump to search

BLOOM (BigScience Large Open-science Open-access Multilingual Language Model) is a large, open-access language model (LLM) containing 176 billion parameters. It was developed in 2022 as part of the BigScience project—an international collaboration of over 1,000 researchers from 70 countries, spearheaded by Hugging Face[1].

BLOOM is an autoregressive transformer model capable of generating coherent text in 46 natural languages and 13 programming languages. The model was trained on the Jean Zay supercomputer in France and became one of the first truly open-access alternatives to closed models like GPT-3 from OpenAI[2].

Background and Development

The BigScience initiative was launched in May 2021 as a one-year research workshop, with the goal of democratizing AI research by collaboratively creating a large, open-source language model[1][3]. At that time, state-of-the-art LLMs like GPT-3 were developed in a closed-source manner by large corporations that did not disclose their architecture, training data, or source code. The BigScience project brought together over a thousand volunteer researchers from around the world to create a competitive and fully open model[4].

The project received a grant for computing resources on the French supercomputer Jean Zay (grant 2021-A0101012475), owned by GENCI and operated at IDRIS, the computing centre of the French National Centre for Scientific Research (CNRS)[2]. The model's training took place from March 11 to July 6, 2022[5]. The development was conducted with maximum transparency: the team published information about data selection, training configurations, and held public discussions, following the project's adopted ethical charter[6].

Architecture and Training

Model Architecture

BLOOM is built on a decoder-only autoregressive transformer architecture, similar to the GPT-3 model[2].

Architectural specifications of BLOOM[7]
Parameter Value
Type Decoder-only transformer
Parameters 176,247,271,424
Layers 70
Attention heads 112
Hidden size 14,336
Sequence length 2048 tokens
Activation function GELU
Positional encoding ALiBi (Attention with Linear Biases)
Vocabulary 250,680 tokens (byte-level BPE)
Training tokens ≈366 billion

Two architectural choices distinguish BLOOM from a standard GPT-style model, both selected after systematic ablation experiments[2][8]:

  • ALiBi positional embeddings: instead of adding positional information at the embedding layer, ALiBi attenuates attention scores according to the distance between keys and queries. Although originally motivated by extrapolation to longer sequences, it also produced smoother training and better downstream results than both learned and rotary embeddings[9].
  • Embedding LayerNorm: an additional normalization layer placed immediately after the embedding layer, which markedly improved training stability at 176B scale (at some cost in performance)[2].

The tokenizer is a byte-level BPE model with a deliberately large vocabulary of 250,680 tokens, chosen so that low-resource languages would not be excessively over-segmented; because it operates on bytes, tokenization never produces unknown tokens[2].

Besides the flagship 176B model, the family includes smaller checkpoints of 560M, 1.1B, 1.7B, 3B, and 7.1B parameters, trained on the same corpus[7].

Training Data

BLOOM was trained on the specially created ROOTS (The Responsible Open-science Open-collaboration Text Sources) text corpus. The total data volume was 1.6 terabytes of cleaned and deduplicated text (≈366 billion tokens seen during training)[10].

The corpus includes texts in 59 languages[10][7]:

  • 46 natural languages, including English (30% of tokens), Chinese, French, Spanish, Arabic, as well as many low-resource languages (e.g., Chi Tumbuka — 0.00002% of tokens).
  • 13 programming languages, including Python, Java, JavaScript, and C++.

This multilingual and multi-domain dataset was intentionally compiled to make the model useful for a wide range of language communities. The corpus was documented in unusual detail, with a published datasheet describing sources, preprocessing, and known limitations[10].

Training Infrastructure

Training ran for about 3.5 months and consumed 1,082,990 GPU-hours on 48 nodes of 8 NVIDIA A100 80 GB GPUs each (384 GPUs in total, with spare nodes held in reserve against hardware failures)[2]. The model was implemented using the Megatron-LM[11] and DeepSpeed[12] frameworks, developed by Nvidia and Microsoft, respectively, combined into a Megatron-DeepSpeed fork that applied 3D parallelism (data, tensor, and pipeline parallelism) and bfloat16 mixed-precision training[13].

Carbon Footprint

BLOOM was the first LLM of its scale for which a full life-cycle carbon assessment was published. Its final training run was estimated to have emitted approximately 24.7 tonnes of CO₂eq when counting only dynamic power consumption, and 50.5 tonnes when equipment manufacturing and idle consumption are included — a comparatively low figure for a model of this size, partly because the Jean Zay supercomputer runs on France's largely low-carbon (nuclear) electricity grid[14].

Performance and Application

BLOOM's results depend strongly on the type of task[2]:

  • On multilingual tasks — machine translation on Flores-101 and multilingual summarization on WikiLingua — BLOOM performs well and outperforms the comparably sized OPT-175B from Meta, with translation quality for several low-resource languages approaching that of dedicated supervised systems.
  • On English-only benchmarks, the picture is more modest. On the HELM evaluation suite the authors report that BLOOM is roughly on par with previous-generation English-only models such as GPT-3 davinci v1, but behind more recent models including InstructGPT davinci v2 and OPT; it does, however, rank among the best models evaluated for fairness[2][15].
  • In the zero-shot setting on SuperGLUE, performance is close to the random baseline on many tasks; a single in-context example yields only marginal improvement[2].

The model is capable of performing a wide range of tasks in a zero-shot setting (without additional training), including:

  • Generating text in a given style.
  • Summarizing documents.
  • Answering questions based on context.
  • Translating between languages.
  • Generating simple program code (though well below the level of specialized models such as Codex)[2].

To improve its practical utility, the BigScience team later conducted additional multilingual multitask fine-tuning on the xP3 prompt corpus, creating the BLOOMZ version, which follows user instructions far more accurately and substantially improves zero-shot task generalization[16].

Limitations and Criticism

Two limitations shaped BLOOM's practical fate[2]:

  • Not compute-optimal. BLOOM was trained on roughly 366 billion tokens for 176 billion parameters — about 2 tokens per parameter. The Chinchilla scaling study, published while BLOOM was already being trained, argued that compute-optimal training requires on the order of 20 tokens per parameter, meaning BLOOM (like GPT-3 and OPT-175B) was substantially undertrained relative to its size[17]. Subsequent open models — LLaMA, Mistral, Falcon and their successors — achieved comparable or better quality with far fewer parameters trained on far more data, and quickly displaced BLOOM in practical use.
  • Deployment cost. Inference with the full 176B model requires more than 350 GB of accelerator memory, placing it out of reach of most users. This motivated Petals, a BigScience-affiliated system for running BLOOM-scale models collaboratively over the internet, BitTorrent-style, by distributing layers across volunteer machines[18].

Licensing and Open Access

The full 176-billion-parameter BLOOM model, its source code, and data were released in July 2022. The model is distributed under the specially developed BigScience BLOOM RAIL (Responsible AI License) v1.0 (identifier `bigscience-bloom-rail-1.0`), dated 19 May 2022[19].

This license permits free use, modification, and redistribution of the model but imposes a series of use-based restrictions. In particular, it is forbidden to use BLOOM for purposes that contradict the BigScience ethical norms, such as[19][6]:

  • Mass surveillance.
  • Algorithmic discrimination.
  • Spreading disinformation.
  • Controlling lethal autonomous weapons.

According to the RAIL Initiative, the BLOOM RAIL license was the first of its kind: the first open license devoted specifically to licensing a machine learning model, combining a permissive intellectual-property grant with behavioural use restrictions[20]. Models released in parallel, such as Meta's OPT-175B and SEER, also carried behavioural-use clauses, but permitted research use only and were therefore not open in the same sense[21].

In August 2022 the license was generalized into the BigScience OpenRAIL-M license, applicable to any AI model, which in turn became the template for CreativeML OpenRAIL-M (used by Stable Diffusion) and later BigCode OpenRAIL-M (used by StarCoder)[21][19].

Legacy

Although BLOOM itself has been superseded as a practical tool, its main contribution proved to be procedural rather than architectural: the project established a template for open, documented model releases — a detailed model card, a corpus datasheet, an ethical charter, published training logs, and a full carbon-footprint accounting — and produced the OpenRAIL licensing family now used well beyond BigScience[3][14][21]. BLOOM also remains a reference point in research on multilingual models, including efforts to extend it to further languages[22].

Literature

  • Hendrycks, D.; Gimpel, K. (2016). Gaussian Error Linear Units (GELUs). arXiv:1606.08415.
  • Shoeybi, M.; et al. (2019). Megatron‑LM: Training Multi‑Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053.
  • Rajbhandari, S.; et al. (2020). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. arXiv:1910.02054.
  • Press, O.; et al. (2021). Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. arXiv:2108.12409.
  • Hoffmann, J.; et al. (2022). Training Compute‑Optimal Large Language Models. arXiv:2203.15556.
  • Le Scao, T.; et al. (2022). What Language Model to Train if You Have One Million GPU Hours?. arXiv:2210.15424.
  • Le Scao, T.; et al. (2022). BLOOM: A 176B‑Parameter Open‑Access Multilingual Language Model. arXiv:2211.05100.
  • Luccioni, A. S.; Viguier, S.; Ligozat, A.-L. (2023). Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model. JMLR 24. arXiv:2211.02001.
  • Muennighoff, N.; et al. (2022). Crosslingual Generalization through Multitask Finetuning (BLOOMZ & mT0). arXiv:2211.01786.
  • BigScience Workshop (2022). BigScience OpenRAIL‑M License v1.0. Online specification.
  • Akiki, C.; et al. (2022). BigScience: A Case Study in the Social Construction of a Multilingual Large Language Model. arXiv:2212.04960.
  • Yong, Z.‑X.; et al. (2022). BLOOM+1: Adding Language Support to BLOOM for Zero‑Shot Prompting. arXiv:2212.09535.
  • Laurençon, H.; et al. (2022). The BigScience ROOTS Corpus: A 1.6 TB Composite Multilingual Dataset. arXiv:2303.03915.
  • Borzunov, A.; et al. (2023). Distributed Inference and Fine-tuning of Large Language Models Over The Internet (Petals). arXiv:2312.08361.
  • Liang, P.; et al. (2022). Holistic Evaluation of Language Models (HELM). arXiv:2211.09110.

References

  1. ↑ 1.0 1.1 "BLOOM". BigScience Blog. [1]
  2. ↑ 2.00 2.01 2.02 2.03 2.04 2.05 2.06 2.07 2.08 2.09 2.10 2.11 Le Scao, T., et al. (2022). "BLOOM: A 176B-Parameter Open-Access Multilingual Language Model". arXiv:2211.05100. [2]
  3. ↑ 3.0 3.1 Akiki, C.; et al. (2022). "BigScience: A Case Study in the Social Construction of a Multilingual Large Language Model". arXiv:2212.04960. [3]
  4. ↑ "BigScience Releases 176B Parameter AI Language Model BLOOM". InfoQ. 26 Jul 2022. [4]
  5. ↑ "Researchers open-source neural network with 176B parameters". SiliconANGLE. [5]
  6. ↑ 6.0 6.1 "BigScience BLOOM Responsible AI License (RAIL) 1.0". OECD.AI Catalogue of Tools & Metrics. [6]
  7. ↑ 7.0 7.1 7.2 "bigscience/bloom". Hugging Face. [7]
  8. ↑ Le Scao, T.; et al. (2022). "What Language Model to Train if You Have One Million GPU Hours?". Findings of EMNLP 2022, arXiv:2210.15424. [8]
  9. ↑ Press, O.; Smith, N. A.; Lewis, M. (2022). "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation". ICLR 2022, arXiv:2108.12409. [9]
  10. ↑ 10.0 10.1 10.2 Laurençon, H.; Saulnier, L.; Wang, T.; et al. (2022). "The BigScience ROOTS Corpus: A 1.6 TB Composite Multilingual Dataset". NeurIPS 2022 Datasets and Benchmarks Track, arXiv:2303.03915. [10]
  11. ↑ Shoeybi, M.; et al. (2019). "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism". arXiv:1909.08053. [11]
  12. ↑ Rajbhandari, S.; et al. (2020). "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models". arXiv:1910.02054. [12]
  13. ↑ "The Technology Behind BLOOM Training". Hugging Face Blog. [13]
  14. ↑ 14.0 14.1 Luccioni, A. S.; Viguier, S.; Ligozat, A.-L. (2023). "Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model". Journal of Machine Learning Research, 24, arXiv:2211.02001. [14]
  15. ↑ Liang, P.; et al. (2022). "Holistic Evaluation of Language Models (HELM)". arXiv:2211.09110. [15]
  16. ↑ Muennighoff, N.; et al. (2023). "Crosslingual Generalization through Multitask Finetuning (BLOOMZ & mT0)". ACL 2023, arXiv:2211.01786. [16]
  17. ↑ Hoffmann, J.; et al. (2022). "Training Compute-Optimal Large Language Models". arXiv:2203.15556. [17]
  18. ↑ Borzunov, A.; et al. (2023). "Distributed Inference and Fine-tuning of Large Language Models Over The Internet". NeurIPS 2023, arXiv:2312.08361. [18]
  19. ↑ 19.0 19.1 19.2 "The BigScience OpenRAIL-M License". Responsible AI Licenses (RAIL) Initiative. [19]
  20. ↑ "FAQ". Responsible AI Licenses (RAIL). [20]
  21. ↑ 21.0 21.1 21.2 "OpenRAIL: Towards open and responsible AI licensing frameworks". Hugging Face Blog. [21]
  22. ↑ Yong, Z.-X.; et al. (2023). "BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting". ACL 2023, arXiv:2212.09535. [22]
Retrieved from "https://systems-analysis.ru/eng/index.php?title=BLOOM_(language_model)&oldid=3131"