← 🖥️ ML Infrastructure & Serving
Advanced
Noisy Neighbors and KV Fair Share
On a shared model server the contended resource is KV cache, and it is consumed in proportion to context length. A limiter that counts requests therefore measures the wrong thing: one hundred-thousand-token request holds the cache of fifty short ones, so equal request counts hand one tenant most of the machine.
Get full Premium access · ₹2,000 / $25
Every answer, concept and course, all published video lessons, hands-on FDE Lab missions, Premium PDF guides and companion files, the full practice-test bank and work-sample downloads. Referral Premium excludes guide PDFs and their companion files.
6 months · One payment · No auto-renewal
Study alongside free video lessons.
RELATED CONCEPTS
PRACTICE THIS IN REAL QUESTIONS
System Design & Production EngineeringExplain how rate limiting works, fixed window, sliding window, token bucket. When does each break?→MLOps & ML EngineeringWhat problem does a feature store solve? Explain the online store versus the offline store.→System Design & Production EngineeringEstimate the capacity and cost of an app with 50k DAU making 10 LLM calls each. What do you provision for?→AI Security, Privacy & GovernanceDesign safety guardrails into the request flow of an LLM serving system, where do they sit and what does each catch?→SQL & Data EngineeringHow do you size Snowflake warehouses, when do you scale up vs scale out, and how do you keep the bill sane?→MLOps & ML EngineeringWhat is training-serving skew, and how do you keep online and offline features consistent?→