← 🖥️ ML Infrastructure & Serving
Advanced
Noisy Neighbors and KV Fair Share
On a shared model server the contended resource is KV cache, and it is consumed in proportion to context length. A limiter that counts requests therefore measures the wrong thing: one hundred-thousand-token request holds the cache of fifty short ones, so equal request counts hand one tenant most of the machine.
Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew
RELATED CONCEPTS
PRACTICE THIS IN REAL QUESTIONS
System Design & Production EngineeringExplain how rate limiting works, fixed window, sliding window, token bucket. When does each break?→MLOps & ML EngineeringWhat problem does a feature store solve? Explain the online store versus the offline store.→System Design & Production EngineeringEstimate the capacity and cost of an app with 50k DAU making 10 LLM calls each. What do you provision for?→AI Security, Privacy & GovernanceDesign safety guardrails into the request flow of an LLM serving system, where do they sit and what does each catch?→SQL & Data EngineeringHow do you size Snowflake warehouses, when do you scale up vs scale out, and how do you keep the bill sane?→MLOps & ML EngineeringWhat is training-serving skew, and how do you keep online and offline features consistent?→