DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

DeepSeek-V2 attacks two different costs with two different changes. Mixture-of-experts layers cut the compute per token. Multi-head latent attention cuts the memory each token occupies in the KV cache, which mixture-of-experts does nothing about and which decides how many requests fit on a GPU at long context.

Multi-head latent attention (MLA). Ordinary attention caches a key and a value vector for every head in every layer. MLA instead projects each token's hidden state down to one latent vector of 512 numbers, caches that, and reconstructs the per-head keys and values from it with up-projection matrices. Because those up-projections can be merged into the query and output weights at inference time, the model never needs to materialize full keys and values. Rotary position embeddings do not survive that merge, so MLA adds a small separate key of 64 numbers, shared across heads, that carries position. Per token per layer, the cache holds 576 numbers instead of the 32,768 that full multi-head attention would need at this size.

DeepSeekMoE. Every feed-forward block except the first becomes 2 shared experts that see every token plus 160 small routed experts, of which 6 are chosen per token. Many small experts let each specialize; shared experts hold common knowledge so the routed ones need not duplicate it. Each token's experts are limited to 3 of the 8 devices that hold them, and three balance losses keep experts, devices and communication even.

The model has 236B parameters with 21B active per token, trained on 8.1 trillion tokens, extended from 4K to 128K context with YaRN, then fine-tuned on 1.5 million instruction examples and trained with reinforcement learning using GRPO. It matched the best open models of mid-2024 on English benchmarks while being far cheaper to train and serve than DeepSeek's earlier dense 67B model.

At long contexts, the KV cache, not the weights, usually decides how many users a GPU can serve. DeepSeek-V2 is the paper that made the cache itself an architecture target, and its descendants (DeepSeek-V3 and R1) are models customers ask about.

Cache size sets concurrency. A model whose cache is a tenth the size can hold roughly ten times as many concurrent long conversations in the same leftover memory. When comparing models for a long-document or agent workload, compute KV bytes per token from the architecture, not just the parameter count.

Efficiency headlines combine several changes. The paper's 93.3% cache reduction comes from MLA together with quantizing the cache to about 6 bits. MLA alone, at the same precision, is about an 82% reduction against DeepSeek 67B. Both numbers are useful; quoting the wrong one to a customer sets the wrong expectation for their own deployment.

MoE plus MLA needs serving support. Absorbing the up-projections, fine-grained expert kernels and expert parallelism all need engine support. Before committing, check that the customer's serving stack implements MLA efficiently.