← 🧠 Foundations of LLMs & GenAI
Advanced
Mixture of Experts (MoE)
An MoE replaces the dense feed-forward block of a transformer with many parallel expert blocks plus a small router that activates only a few experts per token. This decouples total parameter count from per-token compute: the model can hold hundreds of billions of parameters while doing the work of a much smaller one on any given token. FDE loops probe it because the headline 'huge but cheap' hides a brutal serving cost, every expert must sit in memory even though most stay idle.
Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew
RELATED CONCEPTS
PRACTICE THIS IN REAL QUESTIONS
LLM & GenAI FundamentalsWalk me through what actually happens when an LLM generates a response.→RAG & Agent System DesignWalk me through a RAG pipeline end-to-end, and tell me where it usually breaks→ML Infrastructure & GPUsExplain how the CUDA execution model maps to hardware, grids, blocks, warps, SMs.→LLM & GenAI FundamentalsWalk me through the transformer architecture end-to-end, minus the heavy math.→ML Infrastructure & GPUsWalk me through the GPU memory hierarchy, registers, shared memory, L2, HBM. What lives where and why?→ML Infrastructure & GPUsWhat is warp divergence and why does it hurt performance?→