Mixtral (Mistral AI)

Mixtral 8x7B is a text-only, open-weight large language model (LLM) developed by the French company Mistral AI and released on December 11, 2023. It is a decoder-only sparse Mixture-of-Experts (SMoE) model in which each transformer layer contains eight feed-forward experts and a router activates two of them for each token. The model has 46.7 billion total parameters, approximately 12.9 billion active parameters per token, and a fully dense context window of 32,768 tokens.

At launch, Mistral reported that Mixtral 8x7B outperformed Llama 2 70B on most of its listed benchmarks while using roughly one-fifth as many active parameters, and matched or exceeded GPT-3.5 on most of the evaluations included in the technical report. Mistral also reported inference up to six times faster than Llama 2 70B. These comparisons were produced by the model developer using late-2023 models and specific evaluation and deployment configurations; they do not establish a hardware-independent speed ratio or superiority over later systems.

The base and instruction-tuned weights are distributed under the Apache License 2.0, permitting academic and commercial use under the license terms. The instruction model was post-trained with supervised fine-tuning and Direct Preference Optimization (DPO). As of July 11, 2026, the downloadable weights remain available, but Mistral's official lifecycle register marks both Mixtral 8x7B Base and Mixtral 8x7B Instruct as retired from March 30, 2025. The current model card lists Mistral Small 4 as the replacement for the hosted model.

Development History

Mistral AI was founded in April 2023 by Arthur Mensch, Guillaume Lample, and Timothée Lacroix. The founders had previously worked in artificial-intelligence research at Google and Meta.

On September 27, 2023, the company released Mistral 7B, a 7.3-billion-parameter dense model under the Apache 2.0 license. Mistral 7B used Grouped Query Attention (GQA) and Sliding Window Attention (SWA). Although Mixtral retained several elements of the Mistral 7B architecture, it did not retain SWA in its final configuration.

On December 11, 2023, Mistral announced Mixtral 8x7B, its first publicly released Mixture-of-Experts model. Mistral described it as the strongest permissively licensed open-weight model available at the time and reported an MT-Bench score of 8.30 for the instruction-tuned version. A technical paper describing the architecture and evaluations was submitted to arXiv on January 8, 2024.

On April 17, 2024, Mistral continued the family with Mixtral 8x22B. That model has approximately 141 billion total parameters, 39 billion active parameters, and a 64K context window, and was also released under Apache 2.0.

Mistral subsequently classified Mixtral 8x7B as a legacy model. Both the Base and Instruct versions reached their official retirement date on March 30, 2025. Retirement refers to Mistral's supported model lifecycle and hosted offering; it did not revoke the Apache 2.0 license or remove the previously published weights.

Architecture: Sparse Mixture of Experts (SMoE)

Mixtral 8x7B is based on a decoder-only transformer architecture. Its main distinction from a dense transformer is that every standard feed-forward sub-block is replaced by a sparse Mixture-of-Experts layer.

Key architectural features:

• Transformer Backbone: The model contains 32 decoder layers with a hidden-state dimension of 4,096. It uses 32 attention heads, eight key-value heads, Grouped Query Attention, Rotary Position Embeddings (RoPE), RMSNorm, and SwiGLU feed-forward experts. Its vocabulary contains 32,000 tokens.
• MoE Structure: Each transformer layer contains eight feed-forward expert blocks. For every token and at every layer, a learned router assigns scores to the experts and selects the two experts with the highest routing scores. The layer output is a weighted combination of the outputs from those two experts. Experts and routers were trained jointly rather than by training eight independent models and combining them afterward.
• Parameter Accounting: The model has approximately 46.7 billion total parameters and uses approximately 12.9 billion active parameters to process each token. Despite the name "8x7B", it is not a collection of eight complete 7-billion-parameter models: the attention layers, embeddings, and other components are shared, while the feed-forward blocks are replicated as experts.
• Compute and Memory Requirements: The active-parameter count is closely related to arithmetic performed for an individual token, but it does not describe the full deployment cost. All approximately 47 billion parameters generally need to be stored or made accessible during inference. The technical report notes that routing, expert memory loads, hardware utilization, batching, and expert parallelism can add overhead. Consequently, Mixtral cannot universally be treated as having the same memory requirements or real-world latency as a dense 13-billion-parameter model.
• Attention and Context Length: Mixtral supports a context of 32,768 tokens with fully dense attention. Contrary to a frequently repeated description, the final Mixtral 8x7B architecture does not use Sliding Window Attention. Its official configuration sets sliding_window to null, and the technical paper explicitly distinguishes Mixtral's fully dense 32K context from the SWA architecture of Mistral 7B.
• Router Behaviour: The word "expert" does not necessarily mean that individual blocks specialize in clearly separated subjects such as mathematics, programming, or biology. In the routing analysis published by Mistral, the authors found no obvious topic-based specialization. Expert selections appeared more strongly associated with syntax, token patterns, and local sequence structure.

Training

The Mixtral 8x7B family includes two principal releases:

1. Mixtral-8x7B-v0.1 (base model): A pre-trained model optimized through autoregressive next-token prediction. Mistral states that it was trained on multilingual data extracted from the open Web and reports capabilities in English, French, German, Spanish, and Italian, as well as programming code.
2. Mixtral-8x7B-Instruct-v0.1 (instruction model): A version post-trained first through supervised fine-tuning on instruction data and then through DPO on paired preference data. It was designed for dialogue and instruction-following applications.

The public technical report does not disclose the total number of pre-training tokens, the complete list and provenance of training datasets, the precise data-filtering pipeline, total training compute, or a fully reproducible end-to-end training procedure. Statements that Mixtral was trained on a specific quantified corpus are therefore unsupported unless attributed to a separate verified disclosure.

Neither the base nor the initial Instruct model includes a built-in moderation system. The official Instruct model card states that the model may follow harmful or otherwise undesirable instructions unless external prompting, guardrails, fine-tuning, or moderation systems are applied.

Performance

In Mistral's technical report, Mixtral 8x7B matched or exceeded Llama 2 70B on most of the evaluated tasks while using approximately five times fewer active parameters per token. Mistral's launch announcement additionally claimed inference up to six times faster than Llama 2 70B. The latter is a developer-reported deployment result rather than a fixed architectural ratio: actual latency and throughput depend on batch size, quantization, memory bandwidth, expert-parallel configuration, inference software, and accelerator hardware.

Developer-reported comparison of Mixtral 8x7B with Llama 2 70B and GPT-3.5

Metric	Llama 2 70B	GPT-3.5	Mixtral 8x7B
MMLU (5-shot)	69.9%	70.0%	70.6%
GSM-8K (5-shot)	53.6%	57.1%	58.4%
MBPP (pass@1)	49.8%	52.2%	60.7%
MT-Bench (instruction/chat models)	6.86	8.32	8.30

The MMLU, GSM-8K, and MBPP figures are automatic benchmark results, whereas MT-Bench compares dialogue-oriented versions. For MT-Bench, the GPT-3.5 system was gpt-3.5-turbo-1106. Mistral re-evaluated the compared models using its own evaluation pipeline, and the figures should therefore be described as developer-reported rather than independent results. They represent the competitive landscape of late 2023 and do not indicate Mixtral's position relative to models released in 2024–2026.

• Multilinguality: Mistral increased the proportion of multilingual material relative to Mistral 7B. In the developer's evaluations, Mixtral 8x7B outperformed Llama 2 70B on translated versions of ARC Challenge, HellaSwag, and MMLU in French, German, Spanish, and Italian.
• Long-Context Evaluation: Mixtral achieved 100% retrieval accuracy in Mistral's synthetic passkey test across tested positions and sequence lengths up to 32K tokens. This test measures retrieval of a deliberately inserted key and should not be interpreted as proof of perfect comprehension, reasoning, or factual recall across arbitrary 32K documents.
• Bias Benchmarks: On the authors' evaluations, Mixtral obtained 56.0% accuracy on the BBQ benchmark, compared with 51.5% for Llama 2 70B. On the BOLD dataset, the authors reported generally more positive sentiment and similar variation across demographic categories. BBQ and BOLD evaluate aspects of social bias and sentiment; they are not hallucination benchmarks and do not support a conclusion that Mixtral hallucinates less frequently.
• Instruction Following: Mixtral 8x7B Instruct scored 8.30 on MT-Bench. The technical paper described it as the highest-scoring open-weight model on that benchmark as of December 2023 and reported that it ranked above GPT-3.5 Turbo, Claude 2.1, Gemini Pro, and Llama 2 70B Chat in a December 2023 LMSys Chatbot Arena snapshot. These rankings are historical and changed as new models and evaluation data became available.

Licensing and Availability

Both Mixtral-8x7B-v0.1 and Mixtral-8x7B-Instruct-v0.1 were released under the Apache License 2.0. The license permits use, modification, redistribution, and commercial deployment, subject to its notice, attribution, patent, and other legal provisions.

Mistral's legal register classifies the two versions as "Open Source GPAIM". However, definitions of open-source AI are not uniform. The Open Source Initiative's Open Source AI Definition 1.0 requires not only freely usable parameters but also sufficiently detailed training-data information and the complete code used to derive the parameters. Because Mixtral's full training corpus, data-processing code, and end-to-end training pipeline were not released, open-weight is the more precise general description of the published model artifacts under that definition.

The base and Instruct weights remain available through Mistral's official pages and Hugging Face. They can be run through compatible inference frameworks such as Transformers and vLLM. Mistral's historical reference inference repository remains accessible on GitHub but is archived and is no longer actively maintained.

Mistral's official lifecycle documentation lists March 30, 2025 as the retirement date for both the Base and Instruct versions and identifies Mistral Small 4 as the replacement for the former hosted offering. This retirement does not prevent continued self-hosting or redistribution of the Apache-2.0-licensed weights.

External links

• Mixtral of experts — official Mistral AI announcement
• Mixtral of Experts — technical report
• Mixtral 8x7B — official model card
• Mixtral 8x7B model lifecycle
• Mixtral-8x7B-v0.1 base weights
• Mixtral-8x7B-Instruct-v0.1 weights

See also

• DBRX (language model)
• Jamba (language model)

Literature

• Jiang, Albert Q.; Sablayrolles, Alexandre; Roux, Antoine; et al. (2024). Mixtral of Experts. arXiv:2401.04088.
• Jiang, Albert Q.; Sablayrolles, Alexandre; Mensch, Arthur; et al. (2023). Mistral 7B. arXiv:2310.06825.
• Shazeer, Noam; Mirhoseini, Azalia; Maziarz, Krzysztof; et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. arXiv:1701.06538.
• Lepikhin, Dmitry; Lee, HyoukJoong; Xu, Yuanzhong; et al. (2020). GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. arXiv:2006.16668.
• Fedus, William; Zoph, Barret; Shazeer, Noam (2021). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv:2101.03961.
• Ainslie, Joshua; Lee-Thorp, James; de Jong, Michiel; et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv:2305.13245.
• Rafailov, Rafael; Sharma, Archit; Mitchell, Eric; et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290.