TENT: A Declarative Slice Spraying Engine for Performant and Resilient Data Movement in Disaggregated LLM Serving

In disaggregated serving (prefill and decode on different machines) and multi-turn agent serving, the KV cache moves between GPUs and machines all the time, and so do model weights during online RL. These are large "elephant" transfers on the critical path to the first token. They share the network with small, latency-critical "mice" such as MoE expert traffic.

The authors ran Mooncake Transfer Engine on thousands of GPUs and found three recurring problems with the usual design, where the application binds a transfer to a transport and set of NICs at setup and the engine stripes it evenly. Silos: NVLink and RDMA descriptors don't mix, so paths like "remote DRAM to local GPU over NVLink" fall back to TCP. Stranding: even striping ignores a slow, congested or cross-NUMA rail, and the whole transfer waits for it. Fragility: one cluster logged 382 hardware failures a month, and each link fault surfaced as an application error needing manual repair.

TENT inverts the design. The application declares an intent (source, destination, length, priority) and gets a handle. The engine then works in three phases. Orchestration picks transports per request from a unified view of memory segments and their reachable links, and can stage through host or GPU memory to stitch a multi-hop path. Slice spraying cuts large transfers into slices of at least 64 KB, each carrying its absolute destination offset, and sends each slice to the rail with the lowest predicted finish time, β0 + β1·(queued bytes + L)/bandwidth, using live telemetry. Resilience marks rails healthy, degraded or paused, retries failed slices on other rails (safe because writes go to fixed offsets), and can switch transport entirely.

In SGLang's hierarchical KV cache, TENT gave 1.36x the input throughput of Mooncake TE and lower time to first token; RL weight updates ran 20-26% faster; a NIC failure caused a sub-50 ms dip with no errors. It is open source in the Mooncake repository.

Every serving optimization that moves KV between machines (prefix caching across nodes, prefill/decode disaggregation, multi-turn session caches) assumes the transfer is fast and reliable. In production clusters it often is neither: links degrade, NICs on the far NUMA node are slower, and something fails every couple of hours.

Tail latency comes from the slowest link. If a transfer is striped evenly, one degraded rail sets its finish time. When a customer's P90 time to first token jumps at peak hours while averages look fine, look at the transfer layer.

Failures should be routing events, not incidents. At 382 failures a month, a transfer layer that throws errors to the application means pages and restarts; one that reroutes means a brief slowdown. That difference shows up in on-call load and SLO attainment.

Cache hit rate is not the whole story. A cache that hits but transfers slowly can lose to recomputing. TENT's round-1 numbers show caching costing a little time when there is nothing to reuse; the gains come in later turns.