DeepSeek-V3 Technical Report

DeepSeek-V3 is a large mixture-of-experts (MoE) Transformer from DeepSeek-AI: 671B parameters in total, of which 37B are used for any one token. It was released with open weights in December 2024 alongside this technical report, which is unusually detailed about how a frontier-scale model was trained cheaply.

Two pieces come from DeepSeek-V2. Multi-head Latent Attention (MLA) compresses each token's keys and values into a 512-dimensional latent vector plus a small rotary-position key, and caches only those, shrinking the KV cache. DeepSeekMoE replaces most feed-forward layers with 256 small routed experts and 1 shared expert; each token goes to the shared expert and its top 8 routed experts.

Two pieces are new. Auxiliary-loss-free load balancing keeps experts evenly used by adding a per-expert bias to routing scores, raised for underused experts and lowered for overloaded ones after each training step, instead of adding a balancing term to the loss that can hurt quality. Multi-token prediction (MTP) adds a small module that predicts the token after next, which densifies the training signal; at inference it can be dropped or used as a draft for speculative decoding.

Training efficiency gets as much space as architecture: FP8 matrix multiplication with fine-grained scaling, a pipeline schedule called DualPipe that hides the cross-node traffic expert routing creates, custom all-to-all kernels, and memory savings that avoid tensor parallelism entirely. Pre-training used 2,048 H800 GPUs and 14.8 trillion tokens, with no irrecoverable loss spikes or rollbacks. Context was then extended to 128K tokens, and post-training included supervised fine-tuning and reinforcement learning, with reasoning distilled from DeepSeek-R1.

The report gives the total as 2.788 million H800 GPU-hours, $5.576 million at an assumed $2 per hour, for the final training run only.

DeepSeek-V3 reset expectations about what a frontier-quality model costs to train, and its open weights made it a serious option for customers who need self-hosting.

Read cost claims precisely. $5.6M is the rental value of the final run's GPU-hours. It excludes the research, failed runs, ablations, data work, staff and the DeepSeek-R1 model whose reasoning was distilled into V3. Customers quoting it as "what a frontier model costs" are comparing different things.

Active parameters set compute, total parameters set memory. Per token, V3 computes like a 37B model, but all 671B parameters must be held somewhere. The report's recommended serving units are 32 GPUs for prefill and 320 for decoding: a large cluster, which the authors list as a limitation for small teams.

KV cache design changes long-context economics. MLA's small cache makes 128K-token contexts far cheaper to hold than standard attention would be.

Multi-token prediction pays at inference. Reused as a speculative draft, the extra prediction was accepted 85% to 90% of the time, giving 1.8 times the decoding speed.