Yi (01.AI)

Yi is a family of large language models (LLM) and vision-language models (VLM) developed by 01.AI (零一万物) under the leadership of Kai-Fu Lee. The models were trained from scratch on a high-quality bilingual (English and Chinese) corpus of 3.1 trillion tokens and include variants for text generation, long-context processing (up to 200K tokens), multimodal input (text + images), code generation, and efficient inference based on the Mixture-of-Experts (MoE) architecture. The open-weight models are distributed under the Apache 2.0 license, which permits commercial use (they were initially released under a bespoke Yi model license and re-licensed under Apache 2.0 in March 2024); the closed flagship models (Yi-Large, Yi-Lightning) are available through an API.

History and development timeline

01.AI was founded in March 2023 by Kai-Fu Lee, former head of Microsoft Research Asia and Google China, and is headquartered in Beijing. The company focuses on efficient LLMs, with an emphasis on data quality and engineering optimization of infrastructure. By November 2023, 01.AI had reached unicorn status (a valuation above US$1 billion) after a funding round with participation from Alibaba.

First generation (2023–2024)

The first open models, Yi-6B and Yi-34B, were released on November 2, 2023. Over the following weeks the lineup was expanded with 200K-context variants (November 5, 2023) and with chat and quantized models (November 23, 2023). The multimodal Yi-VL-6B and Yi-VL-34B followed in January 2024. In March 2024, 01.AI introduced Yi-9B, derived from Yi-6B by depth upscaling, published the technical report arXiv:2403.04652, and re-licensed the open models under Apache 2.0.

Yi-1.5 and specialized models (2024)

On May 13, 2024, the Yi-1.5 series (6B/9B/34B) was released — the result of continued pre-training on an additional 500 billion tokens and fine-tuning on 3 million instruction examples. In May 2024 the proprietary Yi-Large model with API access was introduced and positioned as a SOTA offering; at the same event the company announced that its first Mixture-of-Experts model, Yi-XLarge, was in training. In September 2024 the specialized Yi-Coder series (1.5B/9B) for code generation was released, with a 128K-token context window and support for 52 programming languages.

Yi-Lightning (2024)

In October 2024 the flagship Yi-Lightning model was introduced, built on a Mixture-of-Experts (MoE) architecture and optimized for inference speed and efficiency. On its debut, Yi-Lightning ranked 6th overall on the LMSYS Chatbot Arena leaderboard (Arena score 1287, tied with Grok-2-08-13), 2nd in the Chinese category, 3rd in Math and Multi-Turn, and 4th in Coding and Hard Prompts. The technical report was published in December 2024 (arXiv:2412.01253).

Strategy shift (2025)

In mid-December 2024, according to media reports, 01.AI disbanded its pre-training algorithm and infrastructure teams, whose members received offers from Alibaba's Tongyi team and Alibaba Cloud; in January 2025 Kai-Fu Lee denied rumors of an asset sale to Alibaba Cloud and disclosed that the company's 2024 revenue had exceeded 100 million yuan (about US$14 million). In March 2025 the company launched the Wanzhi (万智; English branding: Worldwise) Enterprise LLM One-Stop Platform — an integrated software-and-hardware solution for on-premise deployment of DeepSeek models, with support for the DeepSeek, Qwen, and Yi model families — and stopped pre-training its own large language models, focusing instead on enterprise solutions, multi-agent systems, and products built on third-party models, primarily DeepSeek.

Summary timeline

Date	Event
November 2023	Release of Yi-6B and Yi-34B (base, chat, and 200K variants)
January 2024	Multimodal Yi-VL-6B and Yi-VL-34B
March 2024	Yi-9B (depth-upscaled); technical report arXiv:2403.04652; re-licensing under Apache 2.0
May 2024	Yi-1.5 (6B/9B/34B); Yi-Large (closed, API); Yi-XLarge (MoE) announced as in training
September 2024	Yi-Coder-1.5B/9B (code specialization, 128K context)
October 2024	Yi-Lightning (MoE architecture, flagship model)
December 2024	Yi-Lightning technical report (arXiv:2412.01253); reported dissolution of the pre-training and infrastructure teams
March 2025	Wanzhi (Worldwise) enterprise platform; end of in-house LLM pre-training, pivot to DeepSeek-based enterprise solutions

Theoretical foundations and architecture

Base architecture

All Yi language models are based on the decoder-only Transformer architecture described by Vaswani et al. The models were trained from scratch and are not derivatives of LLaMA, despite similarities in implementation.

The core multi-head self-attention mechanism is described by the formula:

$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{Q{K}^{\mathrm{\top }}}{\sqrt{d_k}}\right)V$

where $Q$, $K$, $V$ are the matrices of queries, keys, and values respectively, and $d_k$ is the key dimension.

Key architectural modifications in Yi:

• Grouped-Query Attention (GQA) — query heads are split into groups sharing key/value heads, which reduces memory consumption while preserving quality; unlike Llama 2, GQA is applied to both the 6B and the 34B model.
• SwiGLU activation — replaces the standard ReLU/GELU and reduces the activation size of the post-attention layer from 4h to 8/3h (where h is the hidden size).
• Rotary Position Embedding (RoPE) with an adjusted base frequency (ABF), enabling context extension up to 200K tokens without architectural changes.
• Vocabulary: 64,000 BPE tokens (SentencePiece), with numbers split into individual digits and a Unicode-byte fallback for rare characters.

Base model configurations

Architecture parameters from the technical report arXiv:2403.04652:

Parameter	Yi-6B	Yi-34B
Hidden size	4096	7168
Query heads	32	56
KV heads	4	8
Number of layers	32	60
Pre-training sequence length	4096	4096
Maximum learning rate	3×10⁻⁴	1.5×10⁻⁴

Source: arXiv:2403.04652, Table 1.

Yi-Lightning architecture (MoE)

Yi-Lightning (2024) uses an enhanced Mixture-of-Experts architecture with the following features:

• Fine-grained expert segmentation — the FFN of each expert is partitioned into smaller functional units to improve specialization and parameter utilization.
• Multi-level load balancing — a combination of Switch-Transformer (ST), Expert Parallel (EP), and Partitioned EP (PEP) auxiliary losses for even token distribution across experts and expert-parallel groups.
• Hybrid attention — blocks alternate three sliding-window attention layers with one full-attention layer, with KV-cache reuse between consecutive full-attention layers.
• KV-cache memory reduction of up to 82.8% compared to the standard approach on long sequences.
• Context window extended to 64K tokens during training (the public API serves a 16K window); the tokenizer vocabulary was expanded to 100,352 tokens.

The exact number of experts and the total parameter count of Yi-Lightning have not been publicly disclosed.

Multimodal variants (Yi-VL)

In the multimodal Yi-VL models, the language model is connected to a CLIP ViT-H/14 visual encoder through a two-layer MLP projection. An image $I$ is converted by the visual encoder into a sequence of embeddings $\{v_1,\dots ,v_K\}$, which are fed to the language model together with the text tokens. Training proceeds in three stages with increasing image resolution: 224×224 → 448×448 pixels.

Training pipeline and alignment

Pre-training

The base Yi-6B and Yi-34B models were pre-trained on a bilingual corpus of 3.1 trillion tokens. The data passes through a cascaded filtering and deduplication pipeline: heuristic filters, learned scorers (perplexity, quality, coherence, safety), semantic clustering, and MinHash deduplication. Sources include Common Crawl web documents, books, and academic papers.

For Yi-1.5, continued pre-training was performed on an additional 500 billion tokens of high-quality data.

Supervised fine-tuning (SFT)

The first-generation chat models were fine-tuned on a small but carefully curated set of fewer than 10,000 multi-turn examples, each manually verified and edited by the company's machine-learning engineers. For Yi-1.5, the SFT dataset was scaled up to 3 million examples.

Alignment with reinforcement learning

Yi-Lightning uses a multi-stage alignment process: two-stage SFT (1.3 million and 300,000 examples) followed by RLHF with direct preference optimization (DPO) in offline and online phases. Synthetic data for mathematics and coding is generated using search algorithms, including Monte Carlo Tree Search (MCTS). The RAISE (Responsible AI Safety Engine) framework implements four layers of protection: pre-training data filtering, safety fine-tuning, input prompt control, and output response control.

Context extension

The extension of the context window to 200K tokens was implemented via lightweight continued pre-training on 5 billion tokens (a mixture of pre-training data, length-upsampled books, and synthetic multi-document question–answer data) using the RoPE ABF technique. After the March 2024 update, accuracy on the Needle-in-a-Haystack task (retrieving a target fragment from a long text) reaches 99.8%.

Depth upscaling

Yi-9B was obtained by duplicating layers 12–28 of the base Yi-6B model (32 → 48 layers), followed by pre-training on about 800 billion tokens in two stages. The method increases model capacity without training from scratch.

Model family

Core language models

Model	Parameters	Type	Context	Release date	License	Notes
Yi-6B / Yi-34B	6B / 34B	Base / Chat	4K (extendable to 32K at inference)	November 2023	Apache 2.0	Base models trained from scratch
Yi-6B-200K / Yi-34B-200K	6B / 34B	Base	200K	November 2023	Apache 2.0	Continued pre-training on long sequences
Yi-9B	9B	Base	4K (200K variant available)	March 2024	Apache 2.0	Depth-upscaled from Yi-6B + 800B tokens
Yi-1.5-6B/9B/34B	6 / 9 / 34B	Base / Chat	4K / 16K / 32K	May 2024	Apache 2.0	+500B tokens and 3M SFT examples
Yi-Large	Not disclosed	LLM (dense)	Not disclosed	May 2024	Proprietary (API)	Positioned as a SOTA model
Yi-Lightning	MoE (expert count not disclosed)	LLM	64K (16K via API)	October 2024	Proprietary (API)	6th place on LMSYS Chatbot Arena

Sources: arXiv:2403.04652; arXiv:2412.01253; GitHub 01-ai/Yi.

Specialized models

Model	Parameters	Modalities	Context	Release date	Notes
Yi-VL-6B / Yi-VL-34B	6 / 34B	Text + images (448×448)	Base model default	January 2024	CLIP ViT-H/14 visual encoder, three-stage training
Yi-Coder-1.5B / Yi-Coder-9B	1.5 / 9B	Text (code)	128K	September 2024	52 programming languages

Sources: arXiv:2403.04652; GitHub 01-ai/Yi.

API identifiers

Examples on the 01.AI platform and third-party providers: yi-large, yi-lightning, yi-34b-chat, 01-ai/Yi-1.5-34B-Chat (Hugging Face), yi-34b (Fireworks AI, Replicate).

Evaluation and benchmarks

Base model results

Data from the technical report arXiv:2403.04652 (MMLU, C-Eval, and CMMLU — 5-shot; BBH — 3-shot; GSM8K — 8-shot; HumanEval — 0-shot, pass@1):

Benchmark	Yi-6B	Yi-34B	Llama 2-34B	Llama 2-70B	Qwen-14B	GPT-3.5
MMLU	63.2	76.3	62.6	69.7	66.7	69.1
BBH	42.8	54.3	44.1	64.9	53.4	70.1
C-Eval	72.0	81.4	—	50.1	72.1	52.5
CMMLU	75.5	83.7	—	53.3	71.0	55.5
GSM8K	32.5	67.2	42.2	56.8	61.3	57.1
HumanEval	15.9	23.2	22.6	31.7	32.3	48.1

Source: arXiv:2403.04652, Tables 2–3.

Yi-34B outperformed the twice-larger Llama 2-70B on most benchmarks (with the notable exceptions of BBH and code) and led open models on MMLU, C-Eval, and CMMLU at the time of release.

Yi-Lightning results

Data from the technical report arXiv:2412.01253 (GPQA, MATH, HumanEval — 0-shot; WildBench and Arena-Hard — LLM-as-a-judge with GPT-4o-0513):

Benchmark	Yi-Lightning	Qwen2.5-72B-Instruct	Llama 3.1-70B-Instruct
GPQA	50.9	49.1	45.1
MATH	76.4	82.7	67.1
HumanEval	83.5	86.0	76.2
WildBench	65.1	59.9	49.0
Arena-Hard	91.8	90.5	74.0

Source: arXiv:2412.01253, Table 1.

Human preference evaluations

Yi-34B-Chat ranked second on AlpacaEval with a 94.08% win rate (behind only GPT-4 Turbo) and held an Elo rating of 1110 on LMSYS Chatbot Arena (data as of December 21, 2023). Yi-34B led open models on the Hugging Face Open LLM Leaderboard (pre-trained models) and on C-Eval at the time of release.

Yi-Lightning took 6th place overall on LMSYS Chatbot Arena (Arena score 1287, tied with Grok-2-08-13), placing 2nd in Chinese, 3rd in Math and Multi-Turn, and 4th in Coding and Hard Prompts as of the report's publication.

Note. Direct comparison of models from different releases and configurations requires uniform testing conditions (identical benchmark versions and generation parameters). Preference ratings on crowdsourced platforms depend on the participant pool and on the current set of models in the system.

Applications

The Yi family is used across a wide range of natural-language-processing and multimodal tasks:

• Chat assistants and dialogue systems — based on the Yi-34B-Chat and Yi-1.5 chat variants.
• Code generation and analysis — Yi-Coder supports 52 programming languages.
• Long-document analysis — 200K-context variants for legal, technical, and analytical tasks.
• Multimodal analysis — image description and visual question answering with Yi-VL.
• Enterprise agents — RAG systems, knowledge retrieval, and data classification via the 01.AI platform.
• Local deployment — via vLLM, llama.cpp, and Hugging Face Transformers; quantized versions (AWQ/GPTQ, 4/8-bit) run on consumer GPUs (e.g., a single RTX 4090 for the 4-bit Yi-34B).

The API models (Yi-Large, Yi-Lightning) are used for chatbots, multilingual services, and low-cost inference. As of 2024, Yi-Lightning inference cost 0.99 CNY (about US$0.14) per million tokens.

Limitations and open problems

• Hallucinations. The models are prone to factual errors typical of LLMs, especially in complex reasoning and long generation chains.
• Math and code in early versions. The base Yi-34B trails specialized frontier models on hard coding and mathematics (e.g., 14.4 on 4-shot MATH). This was partially addressed in Yi-1.5 and Yi-Lightning.
• Long context. Extension to 200K requires additional pre-training and compute; a slight degradation on short contexts is possible (MMLU of Yi-34B drops from 76.3 to 75.6 after 200K adaptation).
• Closed models. Yi-Large and Yi-Lightning do not provide downloadable weights, which limits independent verification of the architecture and data.
• Arena–benchmark gap. Yi-Lightning performs strongly in human preference evaluations (Chatbot Arena) but trails some competitors on static academic benchmarks — an instance of the broader metrics-mismatch problem noted in the technical report itself.
• Reproducibility. Not all training details (exact dataset composition, domain distribution, hyperparameters) are fully disclosed.
• Prompt sensitivity. Like other LLMs, Yi models are sensitive to prompt wording, which affects output stability.

According to the technical report, 4-bit and 8-bit quantization causes near-zero quality degradation (less than a 1% drop on MMLU/CMMLU), which enables deployment of the 34B chat model on consumer GPUs.

Ethical and regulatory aspects

Yi-Lightning incorporates RAISE (Responsible AI Safety Engine), a safety framework with four layers of protection:

• Filtering of toxic content, PII, and harmful material at the pre-training stage.
• Safety fine-tuning during SFT and RLHF, with reward mechanisms that encourage safe responses.
• Input control (prompt classification).
• Output control (response filtering).

The Apache 2.0 license for the open models permits commercial use; individual hosting providers may impose additional terms. The company's public-facing services operate in accordance with Chinese regulatory requirements for generative AI.

Outlook and research directions

The Yi series demonstrates the effectiveness of an approach that prioritizes data quality over parameter count, along with lightweight scaling techniques (continued pre-training, depth upscaling, MoE optimization).

Since early 2025, 01.AI's focus has shifted to applied solutions:

• Enterprise deployments and multi-agent systems on the Wanzhi (Worldwise) Enterprise LLM Platform, built on third-party models — primarily DeepSeek, alongside Qwen and Yi.
• Inference optimization (FP8 quantization, hardware-aware operators) to reduce deployment costs.

The open Yi models continue to be used by the community for research, fine-tuning, and distillation.

External links

• Official Yi GitHub repository
• Yi-1.5 GitHub repository
• Official Yi models page
• 01-ai organization on Hugging Face

See also

• Mixture-of-Experts (MoE)
• Transformer architecture
• GLM (Zhipu AI)
• Kimi (Moonshot AI)

Literature

• Young A. et al. Yi: Open Foundation Models by 01.AI. arXiv:2403.04652, 2024. https://arxiv.org/abs/2403.04652
• 01.AI (Wake A. et al.). Yi-Lightning Technical Report. arXiv:2412.01253, 2024. https://arxiv.org/abs/2412.01253
• Vaswani A. et al. Attention Is All You Need. NeurIPS, 2017. https://arxiv.org/abs/1706.03762