Kimi Linear: An Expressive, Efficient Attention Architecture

Kimi Linear combines recurrent attention with periodic full-attention layers. Its recurrent component, Kimi Delta Attention, updates a fixed-size matrix state rather than retaining a separate key and value for every past token. Channel-wise gates let different parts of that state decay at different rates.

The delta rule then corrects the state's prediction for a new key-value association. This is more selective than simply adding every association forever, and more flexible than applying one shared forget rate to an entire head. It still uses finite state, so the full model retains global attention to help with precise retrieval.

The reported architecture repeats three KDA layers followed by one full Multi-Head Latent Attention layer. In matched experiments, it improves several quality measures while reducing long-context state and decoding costs. Some individual tasks regress, so the abstract's broad superiority claim should be read alongside the actual tables.

This page reviews the November 2025 revision of the October paper. The official repository links the kernels, integration and released checkpoints. The key deployment question is whether the hybrid's quality and state savings help the customer's request distribution, not whether “linear attention” guarantees a fixed-memory service.

Long agent trajectories can make token-level attention state expensive. A recurrent layer offers bounded state per head, which can reduce memory traffic during decoding. That can be useful when the application generates long outputs or keeps many active sequences.

But an agent may also need to recover an exact identifier or obey an earlier exception. Finite-state compression does not guarantee perfect recall of arbitrary history. Kimi Linear's hybrid design acknowledges this tension by preserving some full-attention layers instead of replacing every layer with recurrence.

Use attention fundamentals and KV-cache accounting to understand the mechanism, then evaluate exact retrieval and state tracking alongside ordinary answer quality. The architecture is valuable when its complete tradeoff fits the workload, including interruptions, prefix reuse and the inference runtime's support for recurrent state.