GO TO A SECTION

↑ ↓ to navigate · Enter to openExplore all sections →
FDEInterviews logo
← ML Infrastructure & GPUs / 38
medium★ EssentialNVIDIATogether AIHugging Face

A 70B model won't fit on one 80GB GPU. Give three ways to serve it and the trade-offs.

Do the weight math first, then choose your poison: shrink the weights, split them across GPUs, or spill to host memory. Each buys you the fit at a different cost, and one of them is almost always wrong for production.

Updated Sep 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.

Do the weight math first, then choose your poison: shrink the weights, split them across GPUs, or spill to host memory. Each buys you the fit at a different cost, and one of them is almost always wrong for production.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 548 remaining answers · ₹2,000 / $25

Every answer, concept and course, every paper breakdown, all published video lessons, hands-on FDE Lab missions, Premium PDF guides and companion files, the full practice-test bank and work-sample downloads. Referral Premium excludes guide PDFs and their companion files.

6 months · One payment · No auto-renewal

Study alongside free video lessons.

Related course lessons

Keep practising

Next in this trackDesign GPT-scale MoE inference as a global service across regions. How do you lay it out?PremiumNext in this trackWrite a GPU kernel that masks logits to a grammar's DFA during decoding, with under 1% overhead.PremiumNext in this trackDesign KV-cache reuse for hours-long 32K chat sessions, including edits, hot weight swaps, and eviction.Premium
FEDITOR'S NOTE

The screen is the back-of-envelope: 70B params in FP16 is ~140 GB, so it physically cannot fit in 80 GB, and a candidate who can't state that fast is in trouble. The strong answer gives three different mechanisms (quantization, tensor/pipeline sharding, CPU/disk offload) with honest trade-offs, and flags offload as the one you avoid for latency-sensitive serving. The held-back follow-up is the KV cache: even after the weights fit, the cache for concurrent requests is the second memory wall.

DISCUSSION · 0

No comments yet — be the first to share your approach.