The rate limiter is working correctly and that is the problem. It counts requests, and one hundred-thousand-token request occupies the KV cache of fifty short ones, so admitting both tenants equally hands one of them eighty percent of the machine.
Updated Aug 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.
The rate limiter is working correctly and that is the problem. It counts requests, and one hundred-thousand-token request occupies the KV cache of fifty short ones, so admitting both tenants equally hands one of them eighty percent of the machine.
The rate limiter is not broken and the candidate who goes looking for a bug in it will spend the round there. It is counting requests, and requests are not the scarce resource. What separates a strong answer is naming KV cache as the contended thing and then noticing the consequence: a limiter that counts requests actively enforces the unfairness, because equal request counts mean wildly unequal cache consumption. Listen for whether they can state the quota in the right unit.
No comments yet — be the first to share your approach.
