Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding

Generating one token for a long-context request reads two big things from GPU memory: the KV cache for attention, which grows with context length and batch size, and the FFN weights, which every token needs regardless of context. At a million tokens of context, both matter.

Tensor parallelism (TP) splits a layer across GPUs so each reads less. For attention it splits by head. Modern models share KV heads across query heads (grouped-query attention with K = 8 KV heads is common; DeepSeek's MLA effectively has one). Once TP is wider than K, each extra GPU still needs a full copy of at least one KV head, so KV reads per GPU stop shrinking. The paper's roofline: at batch 8 and 1M tokens in FP4, KV reads per layer fall from 1,024 µs to 128 µs as TP goes from 1 to 8, then stay at 128 µs. Duplicated KV also eats the memory that would hold more requests.

KV parallelism (KVP) splits the sequence instead: each GPU holds a slice of the history for its heads, computes attention over its slice, and returns a partial output plus a log-sum-exp; one all-to-all combines them into the exact result (the same trick as Flash-Decoding). Prior work (Medha) did this but then ran the FFN on a fixed group of 8 GPUs, leaving FFN weight reads slow.

Helix reuses the same N GPUs in two layouts per layer: KVP × TP (TP ≤ K) for attention, then TP across all N (or TP × expert parallelism for MoE) for the FFN. HOP-B overlaps each request's all-to-all with the next request's attention. New KV entries rotate across KVP ranks every 16 tokens so memory fills evenly.

In NVIDIA's simulator of GB200 NVL72 with FP4 at a 1M-token context, Helix raised per-user speed up to 1.5x and supported up to 32x more concurrent users for DeepSeek-R1; for Llama-405B, 1.13x and 4x against TP.

Customers asking for whole-codebase, whole-archive or long-running agent contexts are asking for decode at hundreds of thousands to millions of tokens. The question that follows is how many GPUs per user, and what latency.

More GPUs don't help attention past the KV head count with plain TP. Adding GPUs to a TP group beyond 8 for a GQA model mostly copies KV cache. If a deployment's long-context latency won't improve with more GPUs, this is the first thing to check.

The shard layout decides concurrency. Duplicated KV leaves less memory for other requests; at a million tokens, a single request's KV cache for a 405B-class model is over 100 GB in FP4 by our arithmetic. Sharding the sequence is what turns a large NVLink domain into more concurrent users.

Short contexts reduce to something familiar. The paper notes that below about 4K tokens Helix becomes data-parallel attention with tensor-parallel FFN, already common in serving engines. The new part matters only for long histories.