FDEInterviews logo
← RAG & Agent System Design / 61
hardOpenAIAnthropicDatabricks

One tenant started sending 100,000-token documents and everyone else's latency doubled. The rate limiter says nobody is over quota.

The rate limiter is working correctly and that is the problem. It counts requests, and one hundred-thousand-token request occupies the KV cache of fifty short ones, so admitting both tenants equally hands one of them eighty percent of the machine.

Updated Aug 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.

The rate limiter is working correctly and that is the problem. It counts requests, and one hundred-thousand-token request occupies the KV cache of fifty short ones, so admitting both tenants equally hands one of them eighty percent of the machine.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 543 remaining answers · ₹2,000 / $25

Every answer, concept and course, all published video lessons, hands-on FDE Lab missions, Premium PDF guides and companion files, the full practice-test bank and work-sample downloads. Referral Premium excludes guide PDFs and their companion files.

6 months · One payment · No auto-renewal

Study alongside free video lessons.

UP NEXT ON YOUR JOURNEY
Next in this trackFinance says one in five numbers from your text-to-SQL feature is wrong. Find which, and why, before they stop trusting all of them.Next in this trackDocument extraction scored 95% on the pilot sample and 60% in the first week of production. What happened, and what do you fix first?Next in this trackYour support agent resolved 69% of conversations last month and 60% this week. Nobody deployed anything. What do you check first?
FEDITOR'S NOTE

The rate limiter is not broken and the candidate who goes looking for a bug in it will spend the round there. It is counting requests, and requests are not the scarce resource. What separates a strong answer is naming KV cache as the contended thing and then noticing the consequence: a limiter that counts requests actively enforces the unfairness, because equal request counts mean wildly unequal cache consumption. Listen for whether they can state the quota in the right unit.

DISCUSSION · 0

No comments yet — be the first to share your approach.