DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

DeepSeek-V4 is a report on two mixture-of-experts models designed to make very long contexts more practical. The Pro configuration has 1.6 trillion total parameters with 49 billion activated per token. Flash has 284 billion total and thirteen billion activated. Both support a context of approximately one million tokens in the reported setup.

The central architectural change is hybrid compressed attention. Compressed Sparse Attention first reduces the history into compressed entries, then selects a subset for attention. Heavily Compressed Attention reduces the history much more aggressively and attends densely to that smaller representation. Both include a local uncompressed window.

The report also describes constrained residual mixing, Muon-based optimization, specialized kernels and a cache layout designed for heterogeneous state. The efficiency result therefore comes from an integrated architecture and implementation, not one setting that can be applied unchanged to any transformer.

This page reviews the April 2026 preview report. Its first-submission date is April even though the arXiv identifier begins with a different month. Official checkpoints and the accompanying inference implementation are the starting points for artifact-specific evaluation. Historical benchmark results should be read with their reasoning budgets and tool configurations attached.

A customer may want an agent to retain a long investigation, inspect a large codebase or compare many documents. Merely accepting a large input is insufficient: the system must process it within a useful latency and memory budget, then use the relevant details correctly.

V4 gives concrete examples of how architecture changes that budget. Compressed state can reduce per-request memory, while sparse selection limits some attention work. Yet model weights, indexer state, local windows and incomplete compression blocks still need management. Activated parameters do not describe all resident weights.

Use KV-cache reasoning and mixture-of-experts fundamentals to evaluate those costs separately. Then test the customer's actual long-context tasks. A favorable FLOP ratio is useful evidence about one computation, but it is not an end-to-end service-level guarantee.