A dashboard needs the average tokens/sec over the last N seconds, updated on every sample, forever. The warm-up that screens whether you evict by time and keep a running sum instead of re-summing the window each call.
Updated Aug 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.
A dashboard needs the average tokens/sec over the last N seconds, updated on every sample, forever. The warm-up that screens whether you evict by time and keep a running sum instead of re-summing the window each call.
The fork in the road is count-window versus time-window. Many candidates build a fixed-size deque (last N samples), which is wrong when samples arrive irregularly, the metric you actually want is the last N seconds. The other tell is re-summing the whole window on every read, turning an O(1) operation into O(N) and melting under a high-frequency stream. Running sum plus time-based eviction is the answer.
No comments yet — be the first to share your approach.
