Google’s large language models
Jump to navigation
Jump to search
Google's large language models are a series of large language models (LLMs) developed by various divisions within Google, including Google AI (formerly Google Brain) and DeepMind. As one of the pioneers in deep learning and the Transformer architecture, Google has made fundamental contributions to the development of modern LLMs. The history of these models' development reflects a journey from specialized language understanding systems to large-scale multimodal and agentic systems that underpin many of Google's products and define the trajectory of the entire AI industry.
History and evolution of Google's models
Early achievements and neural machine translation (2011–2016)
The foundations for LLM development at Google were laid by the Google Brain project (2011), which was dedicated to the application of deep neural networks. One of the first breakthroughs was the Word2Vec algorithm (2013), created by Tomas Mikolov. It allowed words to be represented as vectors (embeddings) that captured their semantic context, which became a fundamental method for language understanding in neural networks.
The next step was the shift to sequence-to-sequence models, such as seq2seq (2014), which formed the basis of Google Neural Machine Translation (GNMT) (2016). The transition of Google Translate to an LSTM-based neural network architecture significantly improved the quality of machine translation. In parallel, the subsidiary DeepMind, acquired by Google in 2014, demonstrated the power of deep learning with the victory of the AlphaGo system over the world Go champion, solidifying belief in the potential of AI.
The Transformer revolution and the birth of BERT (2017–2018)
In 2017, researchers at Google Brain introduced the Transformer architecture in the paper "Attention Is All You Need." This architecture, based on the self-attention mechanism, allowed sequences to be processed in parallel rather than sequentially, which became a revolution in NLP and the foundation for all modern LLMs.
Riding this wave of success, Google introduced the BERT (Bidirectional Encoder Representations from Transformers) model in 2018. BERT was the first deeply bidirectional model, which considered the context of a word from both the left and the right simultaneously. This enabled it to achieve state-of-the-art results on numerous language understanding tasks (GLUE, SQuAD) and set a new industry standard. BERT was released in two versions (BASE with 110 million parameters and LARGE with 340 million) with open source code and weights, which contributed to its widespread adoption. Since 2019, BERT has been used in Google Search to better understand user queries.
Increasing scale and the era of conversational models (2019–2022)
After BERT, Google continued to experiment with scale and architecture:
- T5 (Text-to-Text Transfer Transformer, 2019): A unified model that treats every NLP task as a "text-to-text" problem. Trained on the massive C4 (Colossal Clean Crawled Corpus), T5 was also released as open source in several sizes (up to 11 billion parameters).
- Meena (2020): Google's first specialized conversational model with 2.6 billion parameters, which demonstrated high-quality open-domain dialogue.
- LaMDA (Language Model for Dialogue Applications, 2021): A family of conversational models (up to 137 billion parameters) trained on a vast corpus of dialogues (1.56 trillion words). LaMDA was aimed at creating more natural and meaningful conversations and became widely known after a Google engineer claimed it was "sentient."
- Gopher and Chinchilla (DeepMind, 2021–2022): In parallel, DeepMind investigated scaling laws. The Gopher model (280 billion parameters) showed how scale affects quality. And the Chinchilla model (70 billion) demonstrated that for optimal performance, the key is not the maximum number of parameters but the correct balance between model size and the amount of training data. This finding became known as "Chinchilla's law" and influenced LLM training strategies across the industry.
The era of massive and multimodal models (2022–present)
- PaLM (Pathways Language Model, 2022): At the time of its announcement, it was Google's largest dense model, with 540 billion parameters, trained on the new Pathways distributed infrastructure. PaLM demonstrated breakthrough capabilities in logical reasoning, especially using the Chain-of-Thought (CoT) prompting technique. Specialized versions were built on it, such as Med-PaLM for medicine. In 2023, an improved version, PaLM 2 (~340 billion parameters), was released, which became the foundation for the updated Bard chatbot.
- Gemini (2023–present): A new generation of models created by the combined Google DeepMind team. Gemini was designed from the ground up as a natively multimodal system, capable of processing text, code, images, audio, and video. It was released in several versions:
- Gemini Ultra: The most powerful model for complex tasks.
- Gemini Pro: A versatile model for a wide range of tasks.
- Gemini Nano: A compact model for on-device operation on mobile devices.
In 2024–2025, the family was expanded with Gemini 1.5 (with a context window of up to 1 million tokens) and Gemini 2.0, which gained agentic capabilities.
Summary table of key models
| Model | Release year | Release date | Parameters (est.) | Architecture | Key features | arXiv paper | Stable Google source |
|---|---|---|---|---|---|---|---|
| Transformer | 2017 | 2017-06-12 | ~65–213 million | Encoder-Decoder | Self-attention sequence model; foundational architecture for modern language models. | arXiv:1706.03762 | Google Research |
| BERT | 2018 | 2018-11-02 | 110–340 million | Encoder | Bidirectional masked-language pre-training; major breakthrough in language understanding tasks. | arXiv:1810.04805 | Google Research Blog |
| XLNet | 2019 | 2019-06-19 | 110–340 million | Autoregressive Transformer-XL | Permutation language modeling; bidirectional context without BERT-style masking mismatch. | arXiv:1906.08237 | Google Research Blog |
| ALBERT | 2019 | 2019-12-20 | ~12–235 million | Encoder | Parameter-sharing and factorized embeddings; more efficient BERT-style pre-training. | arXiv:1909.11942 | Google Research Blog |
| T5 | 2019 | 2019-10-23 | 60 million – 11 billion | Encoder-Decoder | Unified "text-to-text" framework for translation, summarization, classification, QA, and other NLP tasks. | arXiv:1910.10683 | Google Research Blog |
| Meena | 2020 | 2020-01-28 | 2.6 billion | Seq2Seq / Evolved Transformer | Open-domain multi-turn chatbot; introduced Sensibleness and Specificity Average (SSA) evaluation. | arXiv:2001.09977 | Google Research Blog |
| ELECTRA | 2020 | 2020-03-10 | ~14–335 million | Encoder + Generator/Discriminator | Replaced-token detection objective; more sample-efficient than masked-language modeling. | arXiv:2003.10555 | Google Research Blog |
| mT5 | 2020 | 2020-10-22 | 300 million – 13 billion | Encoder-Decoder | Multilingual T5 trained across 101 languages. | arXiv:2010.11934 | Google Research |
| Switch Transformer | 2021 | 2021-01-11 | up to 1.6 trillion | Sparse MoE Encoder-Decoder | Simplified expert routing; trillion-parameter sparse scaling with controlled compute cost. | arXiv:2101.03961 | — |
| LaMDA | 2021 | 2021-05-18 | up to 137 billion | Decoder | Dialogue-specialized Transformer family; focused on open-ended conversation, safety, and factual grounding. | arXiv:2201.08239 | Google Blog |
| Gopher | 2021 | 2021-12-08 | 280 billion | Decoder | DeepMind large-scale language model; broad benchmark, ethics, and retrieval-related analysis. | arXiv:2112.11446 | Google DeepMind Blog |
| RETRO | 2021 | 2021-12-08 | up to 7.5 billion | Retrieval-Enhanced Decoder | Uses an external retrieval database; competitive performance with far fewer parameters. | arXiv:2112.04426 | Google DeepMind Blog |
| GLaM | 2021 | 2021-12-09 | 1.2 trillion total, sparse active | Sparse MoE Decoder | Generalist sparse language model; lower training energy and inference compute than comparable dense models. | arXiv:2112.06905 | Google Research Blog |
| Chinchilla | 2022 | 2022-03-29 | 70 billion | Decoder | Compute-optimal scaling result; showed the importance of training smaller models on more tokens. | arXiv:2203.15556 | Google DeepMind Blog |
| PaLM | 2022 | 2022-04-04 | 540 billion | Dense Decoder | Pathways-trained language model; strong few-shot reasoning, multilingual, and code capabilities. | arXiv:2204.02311 | Google Research Blog |
| UL2 | 2022 | 2022-05-10 | up to 20 billion | Encoder-Decoder | Unified denoising framework combining multiple language-learning objectives. | arXiv:2205.05131 | Google Research Blog |
| Flan-T5 / Flan-PaLM | 2022 | 2022-10-20 | T5: up to 11B; PaLM: 540B | Instruction-tuned Encoder-Decoder / Decoder | Instruction tuning and chain-of-thought data; strong zero-shot and few-shot generalization. | arXiv:2210.11416 | Google Research Blog |
| PaLM 2 | 2023 | 2023-05-10 | Not disclosed | Transformer | Improved multilingual, reasoning, and coding capabilities; more compute-efficient than PaLM. | arXiv:2305.10403 | Google Blog |
| Gemini 1.0 | 2023 | 2023-12-06 | Not disclosed; Nano: 1.8B / 3.25B | Native Multimodal Transformer | Ultra, Pro, and Nano family; text, image, audio, and video understanding. | arXiv:2312.11805 | Google Blog |
| Gemini 1.5 | 2024 | 2024-02-15 | Not disclosed | Multimodal MoE | Long-context Gemini generation; up to 1–2 million tokens, with Pro and Flash variants. | arXiv:2403.05530 | Google Blog |
| Gemma | 2024 | 2024-02-21 | 2B / 7B | Decoder / Open-weight | Lightweight open models based on Gemini research; released as pre-trained and instruction-tuned variants. | arXiv:2403.08295 | Google Blog |
| CodeGemma | 2024 | 2024-04-09 | 2B / 7B | Decoder / Code-specialized | Open code models for code completion, code generation, instruction following, and mathematical reasoning. | arXiv:2406.11409 | Google DeepMind |
| PaliGemma | 2024 | 2024-05-14 | 3 billion | Vision-Language Model | Combines SigLIP vision encoder with Gemma language model; captioning, VQA, OCR-like tasks, detection, and segmentation. | arXiv:2407.07726 | Google AI for Developers |
| Gemma 2 | 2024 | 2024-06-27 | 9B / 27B | Decoder / Open-weight | Higher-performing and more inference-efficient successor to Gemma; stronger safety tuning. | arXiv:2408.00118 | Google Blog |
| PaliGemma 2 | 2024 | 2024-12-04 | 3B / 10B / 28B | Vision-Language Model | Based on Gemma 2; broader VLM transfer tasks, OCR, object detection, segmentation, and captioning. | arXiv:2412.03555 | Google DeepMind |
| Gemini 2.0 | 2024 | 2024-12-11 | Not disclosed | Native Multimodal + Tools | Agentic-era Gemini line; native image/audio output, native tool use, and stronger multimodal interaction. | — | Google Blog |
| Gemma 3 | 2025 | 2025-03-12 | 1B / 4B / 12B / 27B | Multimodal Open-weight | Image-and-text input, text output, 128K context window, and support for 140+ languages. | arXiv:2503.19786 | Google Blog |
| Gemini 2.5 | 2025 | 2025-03-25 | Not disclosed | Multimodal Reasoning Model | "Thinking" model family; improved reasoning, coding, mathematics, and complex task performance. | — | Google DeepMind Blog |
| Gemma 3n | 2025 | 2025-06-26 | E2B / E4B | On-device Multimodal Open-weight | Mobile-first Gemma variant; optimized for local multimodal use with efficient memory requirements. | — | Google Developers Blog |
| Gemini 3 | 2025 | 2025-11-18 | Not disclosed | Native Multimodal Reasoning Model | Gemini 3 Pro/Flash generation; upgraded reasoning, multimodal understanding, coding, and agentic capabilities. | — | Google Blog |
| Gemini 3.1 | 2026 | 2026-02-19 | Not disclosed | Native Multimodal Reasoning Model | Updated Gemini 3 generation; stronger core intelligence, improved agentic behavior, and Pro/Flash/Live variants. | — | Google Blog |
| Gemma 4 | 2026 | 2026-04-02 | E2B / E4B / 12B / 31B / 26B A4B | Multimodal Open-weight; Dense / MoE variants | Open-weight agentic model family; up to 256K context, selected audio support, QAT and MTP variants. | — | Google AI for Developers |
| Gemini 3.5 Flash | 2026 | 2026-05-19 | Not disclosed | Native Multimodal Agentic Model | Frontier-level Flash model optimized for coding, agents, long-horizon workflows, and built-in computer use. | — | Google DeepMind |
| DiffusionGemma | 2026 | 2026-06-10 | 26B MoE | Text Diffusion / MoE | Experimental open text-generation model using diffusion-style block generation instead of standard token-by-token decoding. | — | Google Blog |
Gemini
| No. | Date | Model | Note | Source | Developer | arXiv / Technical report | Model Card |
|---|---|---|---|---|---|---|---|
| 1 | 2023-12-06 | Gemini 1.0 | First Gemini generation; covered Ultra, Pro and Nano variants; native multimodal model family for text, image, audio and video understanding. | — | arXiv:2312.11805 Google DeepMind PDF |
HTML: — PDF: Google DeepMind PDF | |
| 2 | 2024-02-15 | Gemini 1.5 | Long-context Gemini generation; includes Gemini 1.5 Pro and Gemini 1.5 Flash. | Vertex AI model versions | arXiv:2403.05530 Google DeepMind PDF |
HTML: — PDF: Google DeepMind PDF | |
| 3 | 2024-12-11 | Gemini 2.0 Flash | First Gemini 2.0 model; introduced stronger agentic capabilities, native tool use and multimodal output. GA followed on 2025-02-05. | Gemini API docs | — | HTML: — PDF: Model Card PDF | |
| 4 | 2025-02-05 | Gemini 2.0 Flash-Lite | Cost-efficient Gemini 2.0 model; preview released on 2025-02-05 and GA followed on 2025-02-25. | Google Developers Blog | Gemini API docs | — | HTML: — PDF: Model Card PDF |
| 5 | 2025-03-25 | Gemini 2.5 Pro | First Gemini 2.5 “thinking” Pro model; stable GA version followed on 2025-06-17. | Gemini API docs | arXiv:2507.06261 Google DeepMind technical report PDF |
HTML: — PDF: Model Card PDF | |
| 6 | 2025-04-17 / 2025-08-26 | Gemini 2.5 Flash / Gemini 2.5 Flash Image | Hybrid reasoning Flash model; the image generation/editing variant was later released as Gemini 2.5 Flash Image / Nano Banana. | Google Developers Blog Google Developers Blog |
Gemini API docs | arXiv:2507.06261 Google DeepMind technical report PDF |
HTML: — PDF: Model Card PDF |
| 7 | 2025-06-17 | Gemini 2.5 Flash-Lite | Fastest and most budget-friendly Gemini 2.5 model; preview on 2025-06-17 and stable GA on 2025-07-22. | Google Developers Blog Google Developers Blog |
Gemini API docs | arXiv:2507.06261 Google DeepMind technical report PDF |
HTML: — PDF: Model Card PDF |
| 8 | 2025-08-01 | Gemini 2.5 Deep Think | Enhanced reasoning model/mode for especially complex math, science and coding problems. | — | arXiv:2507.06261 Google DeepMind technical report PDF |
HTML: — PDF: Model Card PDF | |
| 9 | 2025-10-07 | Gemini 2.5 Computer Use | Specialized Gemini 2.5 model for browser/UI-control agents using screenshots and generated UI actions. | Gemini API docs | arXiv:2507.06261 Google DeepMind technical report PDF |
HTML: — PDF: Model Card PDF | |
| 10 | 2025-11-18 | Gemini 3 Pro | First Gemini 3 model; major reasoning, multimodal and coding upgrade. | Gemini API docs | — | HTML: — PDF: Model Card PDF | |
| 11 | 2025-11-20 | Gemini 3 Pro Image / Nano Banana Pro | Advanced image generation and editing model based on Gemini 3 Pro. | Gemini API docs | — | HTML: — PDF: Model Card PDF | |
| 12 | 2025-12-17 | Gemini 3 Flash | Gemini 3 Flash model optimized for frontier-class performance at lower latency and cost. | Gemini API docs | — | HTML: — PDF: Model Card PDF | |
| 13 | 2026-02-19 | Gemini 3.1 Pro | Upgraded Gemini 3.1 Pro model for complex reasoning, coding, multimodal tasks and agentic workflows. | Gemini API docs | — | HTML: Google DeepMind HTML PDF: Model Card PDF | |
| 14 | 2026-02-26 | Gemini 3.1 Flash Image / Nano Banana 2 | Gemini 3.1 image generation/editing model based on Gemini 3 Flash. | Gemini API docs | — | HTML: Google DeepMind HTML PDF: Model Card PDF | |
| 15 | 2026-03-03 | Gemini 3.1 Flash-Lite | Frontier-class lightweight Gemini 3.1 model focused on cost efficiency and high-frequency tasks. | Gemini API docs | — | HTML: Google DeepMind HTML PDF: Model Card PDF | |
| 16 | 2026-03-26 / 2026-04-15 | Gemini 3.1 Flash Audio / Flash Live / TTS | Audio-focused Gemini 3.1 Flash card covering Flash Live and text-to-speech variants. | Google |
Gemini API docs | — | HTML: Google DeepMind HTML PDF: Model Card PDF |
| 17 | 2026-05-19 | Gemini 3.5 Flash | Gemini 3.5 Flash model for sustained frontier performance on agentic and coding tasks. | Gemini API docs | — | HTML: Google DeepMind HTML PDF: Model Card PDF | |
| 18 | 2026-05-29 | Gemini Omni Flash | Conversational multimodal model for video generation and conversational video editing; API public preview followed on 2026-06-30. | Gemini API docs | — | HTML: Google DeepMind HTML PDF: Model Card PDF | |
| 19 | 2026-06-09 | Gemini 3.5 Audio / Live Translate | Gemini 3.5 audio model for near real-time speech-to-speech translation. | Gemini API docs | — | HTML: Google DeepMind HTML PDF: Model Card PDF | |
| 20 | 2026-06-30 | Gemini 3.1 Flash-Lite Image / Nano Banana Lite | Low-latency, cost-efficient Gemini image generation and editing model for high-volume use cases. | Gemini API docs | — | HTML: Google DeepMind HTML PDF: Model Card PDF |
Application in products and the ecosystem
Google actively integrates its LLMs across its entire product line:
- Google Search: BERT, MUM, and Gemini are used to better understand complex queries and provide direct answers in the form of AI Overviews (formerly SGE).
- Google Assistant and Bard (now Gemini): The transition from simple voice commands to full-fledged conversational assistants based on LaMDA, PaLM 2, and Gemini.
- Google Workspace: The Duet AI features (now Gemini for Workspace) help write emails in Gmail, create texts in Docs, and generate presentations in Slides.
- Android: Gemini Nano enables AI features to run locally on devices like Pixel, enhancing privacy and speed.
- Google Cloud AI: The Vertex AI platform provides businesses with access to PaLM and Gemini models via API to build their own applications.
External links
Bibliography
- Vaswani, A. et al. (2017). Attention Is All You Need. NIPS.
- Devlin, J. et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL.
- Raffel, C. et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR.
- Thoppilan, R. et al. (2022). LaMDA: Language Models for Dialog Applications. arXiv:2201.08239.
- Hoffmann, R. et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556.
- Chowdhery, A. et al. (2022). PaLM: Scaling Language Modeling with Pathways. JMLR.
- Gemini Team, Google (2023). Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312.11805.