Where you put the LayerNorm changes whether gradients survive a hundred layers. Pre-norm keeps a clean residual highway and well-scaled gradients so deep models train without warmup heroics; post-norm normalizes the output but fights vanishing gradients at depth.
Updated Aug 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.
Where you put the LayerNorm changes whether gradients survive a hundred layers. Pre-norm keeps a clean residual highway and well-scaled gradients so deep models train without warmup heroics; post-norm normalizes the output but fights vanishing gradients at depth.
The screen is whether the candidate understands that pre-norm leaves an unnormalized identity path through the residual stream (so gradients flow cleanly to early layers) while post-norm normalizes after the residual add, putting a LayerNorm in the backward path of every layer and making deep stacks gradient-unstable without careful warmup/init. The reserved follow-up: 'what does pre-norm cost you, and what does a final LayerNorm or QK-norm fix.' Saying 'pre-norm is just more stable' without naming the residual-path argument is shallow. Bonus if they mention pre-norm's representation-collapse tradeoff and why some models add extra norms back.
No comments yet — be the first to share your approach.
