PaLM (Pathways Language Model)

PaLM (Pathways Language Model) is a family of large language models (LLMs) developed by Google. The first version of the model, introduced in April 2022, contained 540 billion parameters and became one of the largest language models in the world at the time, demonstrating breakthrough capabilities that resulted from massive scaling.

The key technological foundation of PaLM was Pathways, a new machine learning systems architecture from Google that enables the efficient coordination of distributed computations across thousands of accelerator chips. PaLM was the first large-scale demonstration of this system, showcasing unprecedented training efficiency at an immense scale.

Pathways System: The Foundation for Scaling

The Pathways concept, introduced by Google in 2021, envisioned a single neural network capable of efficiently generalizing knowledge across different domains and performing thousands of tasks simultaneously. PaLM became the first large-scale application of this system: its training was parallelized across 6144 specialized TPU v4 processors, organized into two cloud clusters (TPU v4 Pods).

At the time of its creation, this was the largest TPU configuration ever used to train a single model. The system achieved a record utilization efficiency of 57.8% hardware FLOPs utilization (corresponding to a model FLOPs utilization, or MFU, of 46.2% — roughly double the 21.3% reported for GPT-3), which made it possible to significantly surpass previous projects in scale and successfully train a model with over half a trillion parameters.

Architecture and Training Data

Model Architecture

PaLM is a dense (non-sparse) language model with a decoder-only architecture, similar to models in the GPT series. This architecture is oriented towards next-token prediction tasks and is well-suited for text generation. Unlike the standard transformer architecture, PaLM uses several key modifications to enhance efficiency:

• Parallel Layers: The attention and feed-forward layers are computed in parallel, which accelerated training by approximately 15%.
• SwiGLU Activation: Use of the SwiGLU activation function instead of the standard ReLU, which significantly improved model quality.
• Multi-Query Attention: The key and value projections are shared across attention heads (only the queries remain separate), which sharply reduces the memory cost of autoregressive decoding — a technique that later became standard for fast LLM inference.
• RoPE Embeddings: Rotary position embeddings are used instead of absolute or relative ones, improving behaviour on long sequences.
• Other choices: Shared input and output embedding matrices, no bias terms in dense layers or layer norms (which improved training stability at scale), and a SentencePiece vocabulary of 256k tokens.

The family was trained at three scales — 8 billion, 62 billion, and 540 billion parameters — which allowed the authors to study the effect of scale directly; the largest model has 118 layers.

Training Data

PaLM was trained on a high-quality data corpus of 780 billion tokens. The dataset was multilingual and diverse, including:

• Social media conversations (50% of the corpus).
• Filtered web documents (27%) and books (13%).
• Articles from Wikipedia (4%) and news (1%).
• Source code from GitHub (5% of the corpus).

Approximately 78% of the data was in English, while the remaining 22% was a multilingual set. A special "lossless" tokenization method was used, which preserved all whitespace (critical for code) and split unrecognized Unicode characters into bytes.

Capabilities and Results

Emergent Abilities and Few-Shot Learning

PaLM demonstrated that increasing the scale of the model, the volume of data, and the computational power can lead to emergent (unexpectedly arising) abilities. On many tasks, the model's performance increased sharply and non-linearly only upon reaching the maximum scale, indicating the appearance of new, previously unobserved capabilities.

The model was evaluated in few-shot learning mode (without fine-tuning, with a few examples in the prompt) and surpassed previous large models (such as GPT-3, Gopher, Chinchilla, GLaM, Megatron-Turing NLG, and LaMDA) on 28 out of 29 popular NLP benchmarks. On the comprehensive BIG-bench suite of tasks, PaLM became the first model reported to surpass the average score of the human raters who attempted the same tasks.

Chain-of-Thought Reasoning

One of PaLM's most notable achievements was its ability for multi-step logical reasoning when using the "chain-of-thought prompting" technique. This method involves providing the model with examples where the solution to a problem is broken down into steps. After learning from such examples, PaLM was able to generate its own "chain of thought" to solve new complex tasks, such as:

• Mathematical Problems: On the GSM8K test (grade school-level math problems), PaLM solved 58% of the tasks, surpassing the previous state-of-the-art result achieved by a fine-tuned model.
• Common Sense Tasks: The model was able to generate detailed explanations for non-trivial problems, for example, providing interpretations of previously unseen jokes.

This capability made the model's "thinking" process more transparent and human-like.

Code Generation and Multilingualism

Despite source code constituting only 5% of the training data, PaLM performed comparably to the specialized OpenAI Codex 12B model on code generation and transformation tasks, while having been trained on roughly 50 times less Python code. The model also showed strong capabilities in multilingual tasks, including translation.

Evolution and Successors: The PaLM Family

PaLM became the foundation for an entire family of models developed by Google.

PaLM 2

Introduced in May 2023, PaLM 2 became a more efficient and multilingual successor. Instead of pursuing a higher parameter count, the focus shifted to the quality of training data and architectural efficiency. PaLM 2 is trained on texts in over 100 languages and demonstrates improved capabilities in logic, programming, and translation. The model is released in four sizes (from smallest to largest): Gecko, Otter, Bison, and Unicorn. The most compact version (Gecko) is lightweight enough to run on mobile devices offline. In contrast to the original PaLM, Google withheld the parameter count, dataset size, and architectural details of PaLM 2 from its technical report.

Instruction-Tuned and Specialized Versions

Based on PaLM and PaLM 2, versions for specific domains and usage modes were created:

• Flan-PaLM: PaLM fine-tuned on a large collection of tasks phrased as instructions (instruction tuning), which markedly improved zero-shot generalization and made it the base model for several downstream systems, including the first Med-PaLM.
• Minerva: A version of PaLM (62B and 540B) further trained on scientific and mathematical texts from arXiv and the web, aimed at quantitative reasoning.
• Med-PaLM and Med-PaLM 2: Specialized models for medicine. Med-PaLM (built on Flan-PaLM, late 2022) was the first AI system to exceed the commonly quoted pass mark on USMLE-style questions from the MedQA dataset (67.2%); Med-PaLM 2 (2023, built on PaLM 2) raised this to 86.5%, which Google described as expert-level performance.
• Sec-PaLM 2: A model focused on cybersecurity, trained to identify vulnerabilities and analyze malicious code.

PaLM-E: Multimodal Version

PaLM-E (Pathways Language Model Embodied) is a multimodal model that combines the PaLM language model with visual data from a Vision Transformer (ViT). This allows the model to process both text and images, solving tasks related to the physical world, such as controlling robots. Its largest configuration, PaLM-E-562B, combines the 540B PaLM with a 22-billion-parameter ViT and was, at the time, the largest vision-language model ever published; it reached state-of-the-art results on the OK-VQA benchmark without task-specific fine-tuning.

Supersession by Gemini

PaLM's role as Google's flagship model was short-lived. In December 2023, Google introduced the Gemini family, and the Bard chatbot — which had run on a lightweight LaMDA model and then on PaLM 2 — was migrated to Gemini. The developer-facing PaLM API was subsequently replaced by the Gemini API, and the legacy PaLM 2 endpoints on Vertex AI (text-bison, chat-bison, and related models) were retired in April 2025. The PaLM family is therefore no longer available as a product and is now mainly of historical and research interest.

Ethical Aspects and Limitations

The creators of PaLM emphasize the need for a responsible approach to developing large language models. The official scientific paper included an analysis of potential biases and toxicity in the generated text. To ensure transparency, Google published a Model Card and a Datasheet for PaLM, documenting the dataset's characteristics, testing results, and identified limitations. These measures align with modern practices for responsible AI and are intended to mitigate risks associated with biases and the generation of harmful content.

Departure from Compute-Optimal Scaling

PaLM was trained on 780 billion tokens for 540 billion parameters — a ratio of roughly 1.4 tokens per parameter. One month before its release, DeepMind's Chinchilla study argued that compute-optimal training requires approximately 20 tokens per parameter, meaning that models of PaLM's generation were substantially undertrained relative to their size. By that standard, a considerably smaller model trained on far more data could have matched PaLM's quality at lower cost — an insight that shaped subsequent models, including PaLM 2, which explicitly shifted the emphasis from parameter count toward data quality and volume.

The Debate over Emergent Abilities

The claim that abilities "emerge" abruptly at large scale — for which PaLM was a central piece of evidence — was later challenged. Schaeffer et al. argued that such sharp, unpredictable jumps can be an artifact of the researcher's choice of evaluation metric: discontinuous metrics (such as exact-match accuracy) produce apparent leaps, whereas continuous metrics reveal smooth, predictable improvement with scale. The question of whether emergence reflects a genuine qualitative change in model behaviour remains open.

External links

• Official Google Blog about PaLM
• Official PaLM 2 page (Google AI)
• PaLM — Wikipedia

Literature

• Chowdhery, A. et al. (2022). PaLM: Scaling Language Modeling with Pathways. arXiv:2204.02311.
• Anil, R. et al. (2023). PaLM 2 Technical Report. arXiv:2305.10403.
• Driess, D. et al. (2023). PaLM-E: An Embodied Multimodal Language Model. arXiv:2303.03378.
• Singhal, K. et al. (2022). Large Language Models Encode Clinical Knowledge. arXiv:2212.13138.
• Singhal, K. et al. (2023). Towards Expert-Level Medical Question Answering with Large Language Models. arXiv:2305.09617.
• Lewkowycz, A. et al. (2022). Solving Quantitative Reasoning Problems with Language Models (Minerva). arXiv:2206.14858.
• Barham, P. et al. (2022). Pathways: Asynchronous Distributed Dataflow for ML. arXiv:2203.12533.
• Wei, J. et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903.
• Zhang, Z. et al. (2022). Automatic Chain of Thought Prompting in Large Language Models. arXiv:2210.03493.
• Wei, J. et al. (2022). Emergent Abilities of Large Language Models. arXiv:2206.07682.
• Schaeffer, R. et al. (2023). Are Emergent Abilities of Large Language Models a Mirage?. arXiv:2304.15004.
• Lu, S. et al. (2023). Are Emergent Abilities in Large Language Models just In-Context Learning?. arXiv:2309.01809.
• Chung, H. W. et al. (2022). Scaling Instruction-Finetuned Language Models (Flan-PaLM). arXiv:2210.11416.
• Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361.
• Hoffmann, J. et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556.
• Rae, J. W. et al. (2021). Scaling Language Models: Methods, Analysis & Insights from Training Gopher. arXiv:2112.11446.
• Diao, S. et al. (2023). Active Prompting with Chain-of-Thought for Large Language Models. arXiv:2302.12246.