ML Infrastructure & GPUs: the practice test

FP16 stores each parameter in 2 bytes, so 70B times 2 is 140GB before anything else: no KV cache, no activations, no batching headroom. This one multiplication is the most-used capacity fact in the field: it says a 70B model cannot fit one 80GB device in half precision, that 8-bit brings it near 70GB, and 4-bit near 35GB. Optimizer state matters in training (where memory is several times the weights), not in serving checkpoints. Interviewers open with this to see whether you compute or recite.

Generation lengths vary wildly, so a static batch holds every finished sequence hostage until the longest one ends, and arriving requests queue behind the whole thing. Continuous (in-flight) batching schedules at the token-iteration level: a request that finishes frees its slot immediately and a queued request takes it on the next step. Throughput improvements of several-fold at similar latency made it the default in modern serving engines. It pairs naturally with paged KV memory, since requests now enter and exit constantly and their cache must be allocated and freed just as fluidly.