Falcon (language model family)

From Systems Analysis Wiki
Jump to navigation Jump to search

Falcon is a family of open-source large language models (LLMs) developed by the Technology Innovation Institute (TII) in Abu Dhabi, UAE. The Falcon models have become a significant contribution to the development of accessible artificial intelligence, consistently ranking high on performance benchmarks such as the Hugging Face Open LLM Leaderboard[1].

The family includes models of various sizes and specializations, from compact versions designed to run on consumer hardware to large-scale models that compete with those from leading technology companies. Key features of Falcon include an advanced architecture, training on the high-quality RefinedWeb dataset, and a permissive Apache 2.0 license[2].

History and Development

The first version of the Falcon models was introduced in June 2023. In September 2023, the Falcon-180B model was released, which at the time was the largest and most powerful open-source LLM in the world, surpassing Meta's Llama 2 70B in parameter count[3][4].

Further development of the family included the release of new generations and specialized versions:

  • Falcon 2 (2024): The second iteration with enhanced capabilities, including a multimodal version, Falcon 2 11B VLM (Vision Language Model)[5].
  • Falcon 3 (December 2024): The latest generation, trained on 14 trillion tokens, with expanded multimodal features and optimized to run on lightweight hardware, including laptops[6][7].
  • Specialized Models: Models adapted for specific tasks have been released, such as Falcon Arabic and Falcon Mamba.
Key Models in the Falcon Family
Model Parameters (billions) Key Features License
Falcon-180B 180 Largest first-generation model; trained on 3.5 trillion tokens; outperforms GPT-3.5[3]. TII Falcon License 1.0 (with restrictions on commercial use)[4]
Falcon-40B 40 Baseline high-performance model; trained on 1 trillion tokens. Apache 2.0
Falcon-7B 7 Compact model requiring ~15 GB of GPU memory; suitable for consumer hardware[1]. Apache 2.0
Falcon-1.3B 1.3 Smallest model for resource-constrained devices. Apache 2.0
Falcon 2 11B 11 Second generation; competes with Llama 3 8B and Gemma 7B; multimodal (VLM) version available[5]. Apache 2.0
Falcon 3 N/A Trained on 14 trillion tokens; multimodality (text, image, audio, video); runs on laptops[6]. Apache 2.0
Falcon Arabic 7 Specialized model for the Arabic language (standard and dialects); Falcon 3 architecture[8]. Apache 2.0
Falcon Mamba N/A Experimental model based on the Mamba (SSM) architecture instead of a Transformer[9]. Apache 2.0

Architecture and Technical Features

Transformer Architecture

Most Falcon models are built on a decoder-only Transformer architecture. Key architectural decisions include:

  • Multi-Query Attention (MQA): Unlike standard Multi-Head Attention, where each head has its own key/value pair, in MQA, all attention heads share a single key and value set. This significantly reduces memory consumption and speeds up inference without a substantial loss in quality[1].
  • Rotary Positional Embeddings (RoPE): RoPE is used to encode token position information, similar to other modern LLMs.
  • FlashAttention: Used to optimize attention mechanism computations.

Mamba Architecture (State Space Model)

The Falcon Mamba model is innovative as it departs from the traditional Transformer architecture in favor of a State Space Model (SSM). The Mamba architecture processes data sequences linearly, allowing it to be significantly more efficient with very long contexts and require fewer computational resources compared to Transformers[9].

Training Data

The foundation for training Falcon models is the high-quality RefinedWeb dataset, created by TII[4]. It consists of trillions of tokens extracted from Common Crawl, with rigorous filtering and deduplication to improve quality.

  • For Falcon-180B, an expanded dataset of 3.5 trillion tokens was used, which consisted of ~85% RefinedWeb, along with curated data from books, dialogues, and code[3].
  • Falcon Arabic was trained on a high-quality native (not translated) Arabic dataset, covering both Modern Standard Arabic and regional dialects[10].

Specialized Models

Falcon Arabic

Falcon Arabic is a 7-billion-parameter model specifically optimized for the Arabic language. It demonstrates outstanding results on Arabic benchmarks (Open Arabic LLM Leaderboard) and is capable of understanding both Modern Standard Arabic (MSA) and numerous regional dialects. This enables the model to provide culturally aware and accurate responses for Arabic-speaking users[8]. In terms of performance, it surpasses models that are up to 10 times larger[11].

Multimodal Capabilities

  • Falcon 2 11B VLM was the first multimodal model in the family, capable of processing both text and images[5].
  • Falcon 3 significantly expanded these capabilities by adding support for video and audio. A full voice mode is planned for release in January 2025[6].

Performance and Issues

Comparison with Competitors

Falcon models consistently demonstrate high performance.

  • Falcon-180B outperforms GPT-3.5 and Llama 2 70B on most academic benchmarks, such as MMLU, HellaSwag, and LAMBADA, although it falls short of GPT-4[3].
  • Falcon 2 11B demonstrates performance on par with or exceeding Meta Llama 3 8B and Google Gemma 7B[5].
  • Falcon 3 ranked first on the global Hugging Face Leaderboard among models of its size at the time of its release[6].

Limitations and Issues

  • Performance on Different Languages: The majority of the training data is in English[12]. As a result, the models' performance in other languages, including Russian, can be significantly lower.[13].
  • Hallucinations: Like all LLMs, Falcon models are prone to generating inaccurate or fabricated information (hallucinations), which requires a cautious approach when using them in mission-critical applications[14].
  • Licensing Restrictions: Although most models are distributed under the Apache 2.0 license, the flagship Falcon-180B model has its own TII Falcon LLM License, which imposes royalty obligations for commercial use with revenue exceeding $1 million, limiting its adoption in business[4][15].

Further Reading

  • Ainslie, J. et al. (2023). GQA: Training Generalized Multi‑Query Transformer Models from Multi‑Head Checkpoints. arXiv:2305.13245.
  • Almazrouei, E. et al. (2023). The Falcon Series of Open Language Models. arXiv:2311.16867.
  • Dao, T. et al. (2022). FlashAttention: Fast and Memory‑Efficient Exact Attention with IO‑Awareness. arXiv:2205.14135.
  • Ding, Y. et al. (2024). LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens. arXiv:2402.13753.
  • Fedus, W.; Zoph, B.; Shazeer, N. (2021). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv:2101.03961.
  • Gu, A.; Dao, T. (2023). Mamba: Linear‑Time Sequence Modeling with Selective State Spaces. arXiv:2312.00752.
  • Penedo, G. et al. (2023). The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only. arXiv:2306.01116.
  • Peng, B. et al. (2023). YaRN: Efficient Context Window Extension of Large Language Models. arXiv:2309.00071.
  • Shazeer, N. (2019). Fast Transformer Decoding: One Write‑Head is All You Need. arXiv:1911.02150.
  • Su, J. et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864.

Notes

  1. ↑ 1.0 1.1 1.2 "Falcon: The \"T-shirt-sized\" 7B and 40B models that are democratizing the LLM landscape". Hugging Face Blog. [1]
  2. ↑ "Falcon Model". Hugging Face Transformers documentation. [2]
  3. ↑ 3.0 3.1 3.2 3.3 "Falcon 180B open-source language model outperforms GPT-3.5 and Llama 2". The Decoder. [3]
  4. ↑ 4.0 4.1 4.2 4.3 "Falcon 180B: The World's Largest Open Language Model". Neurohive.
  5. ↑ 5.0 5.1 5.2 5.3 "Falcon 2: UAE’s Technology Innovation Institute Releases New Series of AI Models Outperforming Meta’s Llama 3". AETOSWire.
  6. ↑ 6.0 6.1 6.2 6.3 "Falcon 3: UAE’s Technology Innovation Institute Launches World’s Most Powerful Small AI Models Capable of Running Even on Lightweight Devices, Including Laptops". AETOSWire.
  7. ↑ "Technology Innovation Institute launches Falcon 3 model to enhance access to AI through light infrastructures". Abu Dhabi Media Office. [4]
  8. ↑ 8.0 8.1 "Falcon Arabic". FalconLLM TII. [5]
  9. ↑ 9.0 9.1 "Falcon Mamba — a new step in the development of language models without an attention mechanism". Pikabu.
  10. ↑ "Middle East's Leading AI Powerhouse TII Launches Two New AI Models". TII News. [6]
  11. ↑ "Middle East's leading AI powerhouse, TII,launches two new AI models". Falcon Foundation. [7]
  12. ↑ Almazrouei, Ebtesam, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, et al. "The Falcon Series of Open Language Models". arXiv, 29 November 2023. https://doi.org/10.48550/arXiv.2311.16867.[8]
  13. ↑ "Middle East’s leading AI powerhouse, TII, launches two new AI models". AETOSWire.
  14. ↑ "Falcon-180B: review, launch, and first impressions". Habr.
  15. ↑ "Falcon 180B License Discussion". Hugging Face. [9]


Retrieved from "https://systems-analysis.ru/eng/index.php?title=Falcon_(language_model_family)&oldid=1087"