Do the weight math first, then choose your poison: shrink the weights, split them across GPUs, or spill to host memory. Each buys you the fit at a different cost, and one of them is almost always wrong for production.
Updated Sep 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.
Do the weight math first, then choose your poison: shrink the weights, split them across GPUs, or spill to host memory. Each buys you the fit at a different cost, and one of them is almost always wrong for production.
The screen is the back-of-envelope: 70B params in FP16 is ~140 GB, so it physically cannot fit in 80 GB, and a candidate who can't state that fast is in trouble. The strong answer gives three different mechanisms (quantization, tensor/pipeline sharding, CPU/disk offload) with honest trade-offs, and flags offload as the one you avoid for latency-sensitive serving. The held-back follow-up is the KV cache: even after the weights fit, the cache for concurrent requests is the second memory wall.
No comments yet — be the first to share your approach.
