IN THIS BREAKDOWN · 10 SECTIONS
You can read 20% of this breakdown.
The paper in one minute
Mixtral 8x7B is Mistral 7B with one change: every feed-forward block becomes a mixture of eight experts. Each expert is a SwiGLU block of the same shape as Mistral 7B's, and a small router, a linear layer over the token's hidden state, scores all eight, keeps the top two, softmaxes those two scores and sums the two experts' outputs with those weights. Attention is untouched and shared across experts, so the name "8x7B" overstates the size: the model holds about 47 billion parameters, not 56.
The distinction the paper makes explicit is between total (sparse) parameters, which grow with the number of experts and set the memory bill, and active parameters, which grow with the number used per token and set the compute bill. Mixtral uses about 13 billion per token.
On Mistral's re-run of standard benchmarks, Mixtral beat or matched Llama 2 70B on most tasks, with large leads in code and math, and beat it on French, German, Spanish and Italian versions of ARC, HellaSwag and MMLU. It found a passkey anywhere in a 32k-token context every time. An instruct version trained with supervised fine-tuning and DPO scored 8.30 on MT-Bench and ranked above GPT-3.5 Turbo, Claude 2.1 and Gemini Pro on the LMSys arena in December 2023. Both models were released under Apache 2.0.
The routing analysis is the surprise. Experts did not specialize by subject: arXiv papers, biology abstracts and philosophy were routed almost identically. Routing followed syntax instead, and consecutive tokens often went to the same experts.
Research notes: the problem, approach and results
total parameters stored against active parameters used per token (Section 3)
MMLU, against 69.9% for Llama 2 70B with about 5x fewer active parameters (Table 2)
MT-Bench score of Mixtral 8x7B Instruct, against 8.32 for GPT-3.5 Turbo (Table 3)
Open models that matched GPT-3.5 needed about 70B dense parameters, which are expensive to run for every token. Sparse mixture-of-experts models could cut that compute, but no strong open-weight MoE model with a permissive license existed.
Take the Mistral 7B decoder (32 layers, grouped-query attention, 32k context) and replace every feed-forward block with 8 SwiGLU experts and a router that picks the top 2 per token, weighting them by a softmax over the two scores. Fine-tune an instruct version with supervised fine-tuning and DPO, and release both under Apache 2.0.
With 13B active parameters, Mixtral matched or beat Llama 2 70B on most of 12 benchmarks, with large gains in code (HumanEval 40.2% vs 29.3%) and math, and beat it in French, German, Spanish and Italian. Mixtral Instruct scored 8.30 on MT-Bench. Routing showed no topic specialization.
Why it matters to an FDE
Mixtral is the model that put mixture-of-experts into ordinary deployments, and it is a clean case study in what MoE changes for whoever runs it.
Budget memory and compute separately. Mixtral costs about as much compute per token as a 13B dense model but needs memory for 47B parameters: about 87 GiB at 16 bits, which means two 80 GB GPUs or quantization. Its KV cache is small (it keeps Mistral 7B's attention), so long contexts are cheaper than for Llama 2 70B. A customer who sizes hardware by "it is like a 13B model" will run out of memory.
The license removes a blocker. Apache 2.0 weights are usable commercially without the custom terms of other open models, which often matters more to a customer's legal team than a benchmark point.
MoE wants batches. The authors note that routing adds overhead and that MoE models suit batched workloads. At a single request at a time, the GPU loads two experts' weights per layer per token and works well below its peak; at higher concurrency, more of each expert's weights are reused per load.
Experts are not topic specialists. Don't plan to prune experts by domain or route a customer's legal documents to a "legal expert". The paper finds no such structure.
Understand the mechanism. Try it on a real problem.
Unlock this breakdown and 290 more papers.
Premium opens every paper in full, plus every answer, concept, course and guide on the site. One payment for six months of access.
6 months · One payment · No auto-renewal
- ◇1 architecture diagram
- </>1 runnable code experiment
- ▦2 results tables
- ?5 interview questions with answers
- 03Key takeaways
The routing formula, parameter accounting, where it wins and loses, locality, and two misreadings
Free account - 04How it works
The router, the expert block, the shape in Table 1, and how serving engines run it
Premium - 05The numbers that matter
Benchmarks against Llama 2, comparison with GPT-3.5, multilingual and long-context results, routing
Premium - 06Try it yourself
Rebuild the 47B and 13B counts, compare serving needs, audit Table 2, and simulate locality
Premium - 07Where it breaks
Memory cost, small-batch efficiency, missing training details, evaluation choices and slips
Premium - 08Using it on a deployment
Fitting Mixtral on GPUs, batching for MoE efficiency, and why not to prune experts
Premium - 09Interview angles
Five questions this paper prepares you for, with answer hints
Premium - 10Read next
Where to go from here on this site
Premium