Fast Recovery for LLM Serving via Decoupled Device Memory Lifetime in Dynamo

An LLM serving replica is one engine spread across several GPUs (eight B200s here), running for weeks. Failures are routine: in 18 weeks of NVIDIA's own Dynamo cluster (about 918 GPUs, about 150 replicas), replicas restarted 5,762 times, 46 a day, losing 2,967 GPU-hours to recovery alone. Each restart rebuilds the whole engine: CUDA contexts, NCCL communicators, model weights, kernel tuning and CUDA graph capture. Even with weights in the page cache and kernels precompiled, that takes 80 to 157 seconds for the paper's four models; in production the median was 3.5 minutes. Operators keep spare GPUs to absorb the gap.

The authors' argument: for inference, what a failure costs is ready serving capacity, not request progress. Interrupted requests can be replayed in seconds; it's the initialized engine that's slow to rebuild. And in their data, 77% of restarts were device-preserving: the engine process died but the GPUs were healthy and their memory trustworthy. Only 3 episodes needed a GPU reset.

Dynamo adds three mechanisms that keep initialized state alive:

• GPU Memory Service (GMS). One process per GPU owns the physical memory holding model weights, using CUDA's virtual memory APIs. Engines map it read-only at fixed addresses. When an engine crashes, the weights stay resident and a new engine reattaches without copying.
• Shadow Engines. A second engine fully initializes on the same GPUs, maps the shared weights, reserves KV-cache addresses without memory, and parks. When the primary dies, the shadow installs fresh KV pages and takes over in seconds. It costs 4-8 GiB per GPU, not a second copy of the model.
• Snapshots. An initialized engine is checkpointed once (process tree with CRIU, GPU state with cuda-checkpoint) with its empty KV pages released, and restored instead of reinitialized. This covers failures that need a GPU reset and refills a used shadow without slowing serving.

On eight B200s, shadow promotion took 4.6 to 6.2 seconds across four models on vLLM and SGLang, 13 to 29 times faster than a warm restart.

Customers running their own model fleets pay for failures twice: in user-visible slowdowns while a replica is down, and in idle spare GPUs held to cover those minutes. The paper reframes reliability as a recovery-time problem with a concrete production base rate.

Ask how long a replica takes to come back, measured. Time from crash to serving, with weights cached. If it's minutes, that number drives both incident impact and overprovisioning.

Most failures are software, not hardware. In this trace, crashes, hangs and out-of-memory errors that left GPUs healthy dominated. Designs that assume "a failure means a dead GPU" overpay.

Recovery speed buys back capacity. The faster a replica recovers, the fewer spares are needed and the less the survivors are pushed into queueing. With minutes of recovery, a two-replica deployment under load saw median time to first token rise from 1.3 s to about 20 s.