rate limiting

The rate limiter is working correctly and that is the problem. It counts requests, and one hundred-thousand-token request occupies the KV cache of fifty short ones, so admitting both tenants equally hands one of them eighty percent of the machine.

A verbatim-reported multi-part at OpenAI and Anthropic. Each level breaks the previous design on purpose: the burst-at-the-boundary flaw is planted, and naming it before the interviewer does is how you win the round.

Sliding-window limiters count requests; an API that bills by tokens needs to reserve a variable cost up front and refund it if the call fails. The trick is lazy refill: never run a timer, compute the current balance from elapsed time on each check.

A staple at OpenAI and Anthropic in both coding and design rounds. Most candidates can name the algorithms; few can say which one lets 2x your limit through, and that's the part that gets scored.

Rate limits protect your servers; quotas protect your business model, and most candidates design one and think they've designed both. The strong answer also covers the part customers actually feel: what hitting the limit looks like.

Every request emits a usage event worth real money, clients retry on timeouts, and tenants must be stopped at their quota and never overcharged. The hard parts are idempotent usage events and the succeeded-but-confirmation-lost case where the work happened but the client never heard back.

A reported Anthropic system-design prompt. The skeleton is standard API platform work; the AI-specific 20%, token-denominated limits, safety classifiers in the path, abuse identity, is what the round actually grades.