What metrics do you autoscale inference pods on, and how do you handle cold starts?

CPU-based HPA on GPU inference never fires, the trap half of all candidates fall into within a minute. The signals that actually track load, the anatomy of a five-minute cold start, and which mitigations are worth their cost at each layer.

Updated Aug 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.