Core
Normalization: Batch vs Layer
Normalizing activations keeps their scale stable as they flow through a deep net, which speeds training and lets you use higher learning rates. Batch norm normalizes across the batch and has nasty train/eval and small-batch pitfalls; layer norm normalizes across features per example, which is why transformers use it. RMSNorm strips it down further.
a free account unlocks the core curriculum tier · no card
Related concepts
Practise this in questions
LLM & GenAI FundamentalsPre-norm vs post-norm transformers: why do large models use pre-norm, and how does it affect training stability? →hard · PremiumLLM & GenAI FundamentalsWalk me through the transformer architecture end-to-end, minus the heavy math. →medium · FreeML Infrastructure & GPUsYour training run compiles and runs, but the loss is flat. Walk me through your debugging process. →hard · Free with sign-inML Infrastructure & GPUsHow would you migrate a PyTorch workload to TPU? →medium · PremiumMachine Learning & Data ScienceWhy did attention and transformers replace RNNs for sequence modeling? Explain it conceptually. →medium · PremiumLLM & GenAI FundamentalsDecoder-only, encoder-only, encoder-decoder: what is each good at, and why does generation use decoder-only? →easy · Premium