The GPU-platform capstone: dozens of models, spiky traffic, a fixed monthly GPU budget, and a p99 SLO that scale-to-zero would wreck. Token-based autoscaling, the three layers of LLM cold start, and the residency tiering that makes the budget math work.
Updated Aug 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.
The GPU-platform capstone: dozens of models, spiky traffic, a fixed monthly GPU budget, and a p99 SLO that scale-to-zero would wreck. Token-based autoscaling, the three layers of LLM cold start, and the residency tiering that makes the budget math work.
The tell that someone has actually run LLM serving is that they scale on token throughput and KV-cache headroom, not requests, because one request can cost a hundred times another. The held-back follow-up is 'a cold model just got a request, what does the user see, and what's your number?', and a serious answer names tens of seconds for a cold large model and a tiered-residency plan to avoid it on hot paths. Watch for the candidate who reaches for scale-to-zero everywhere because it demos well; on a user-facing route that's a self-inflicted brownout, and saying so is the senior move.
No comments yet — be the first to share your approach.
