caching
FDE interview questions tagged caching, across every topic.
4 questions · 1 unlocked for you
Concepts behind "caching"
The curriculum that explains the ideas these questions test.
Core
Latency OptimizationMeasure p50, p95, and p99 before you touch anything, then find where the time actually goes: tokenization, retrieval, inference, or post-processing. A naive RAG pipeline that takes 1.5 seconds can usually reach sub-100ms perceived latency by caching, parallelizing retrieval, picking a smaller model, and streaming the first token, in that order of payoff.⚙️ System Design for AI in ProductionSign in
Core
Caching and EvictionA cache is bounded memory in front of expensive work, so the real design question is what to throw away when it fills. LRU evicts the least recently used entry and is the default; it is built from a hash map plus a doubly linked list to get O(1) get and put. TTL adds time-based expiry. Picking and implementing the right eviction policy is one of the most common practical FDE coding screens.💻 Coding & Engineering CraftSign in
Advanced
Prompt Caching and Semantic CachingTwo different caches solve two different bills. Prompt caching reuses the model's internal computation over a stable prefix, cutting cost and time-to-first-token on every call that shares it. Semantic caching skips the model entirely when a near-identical question has been answered before, and it is the one that can serve a wrong answer confidently.🧠 Foundations of LLMs & GenAI🔒 Premium