GO TO A SECTION

↑ ↓ to navigate · Enter to openExplore all sections →
FDEInterviews logo
← LLM & GenAI Fundamentals / 48
hardOpenAIAnthropicGoogle

Pre-norm vs post-norm transformers: why do large models use pre-norm, and how does it affect training stability?

Where you put the LayerNorm changes whether gradients survive a hundred layers. Pre-norm keeps a clean residual highway and well-scaled gradients so deep models train without warmup heroics; post-norm normalizes the output but fights vanishing gradients at depth.

Updated Aug 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.

Where you put the LayerNorm changes whether gradients survive a hundred layers. Pre-norm keeps a clean residual highway and well-scaled gradients so deep models train without warmup heroics; post-norm normalizes the output but fights vanishing gradients at depth.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 548 remaining answers · ₹2,000 / $25

Every answer, concept and course, every paper breakdown, all published video lessons, hands-on FDE Lab missions, Premium PDF guides and companion files, the full practice-test bank and work-sample downloads. Referral Premium excludes guide PDFs and their companion files.

6 months · One payment · No auto-renewal

Study alongside free video lessons.

Keep practising

Next in this trackExplain FlashAttention: why it never writes the full N x N matrix, and how it speeds up attention without changing the math.PremiumNext in this trackPrefix caching in a multi-turn chatbot: how do client-side and server-side caching cut redundant compute, and what breaks correctness?PremiumNext in this trackGRPO vs PPO-based RLHF: how does group-relative reward let you drop the value model?Premium
FEDITOR'S NOTE

The screen is whether the candidate understands that pre-norm leaves an unnormalized identity path through the residual stream (so gradients flow cleanly to early layers) while post-norm normalizes after the residual add, putting a LayerNorm in the backward path of every layer and making deep stacks gradient-unstable without careful warmup/init. The reserved follow-up: 'what does pre-norm cost you, and what does a final LayerNorm or QK-norm fix.' Saying 'pre-norm is just more stable' without naming the residual-path argument is shallow. Bonus if they mention pre-norm's representation-collapse tradeoff and why some models add extra norms back.

DISCUSSION · 0

No comments yet — be the first to share your approach.