Design KV-cache reuse for hours-long 32K chat sessions, including edits, hot weight swaps, and eviction.
Reprefilling the whole history every turn makes a long chat O(n^2). The real design is a radix tree of KV blocks with copy-on-write branching, a weight-epoch tag, and an eviction policy that protects live turns.
Updated Aug 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.
Reprefilling the whole history every turn makes a long chat O(n^2). The real design is a radix tree of KV blocks with copy-on-write branching, a weight-epoch tag, and an eviction policy that protects live turns.
Lead with where the obvious approach breaks, because that is the judgment they are screening for — most candidates jump straight to the happy path and lose the room.
Then walk the failure back through the pipeline in order, naming the one metric the customer's exec sponsor actually cares about before you propose the fix.
