Core

Normalization: Batch vs Layer

Normalizing activations keeps their scale stable as they flow through a deep net, which speeds training and lets you use higher learning rates. Batch norm normalizes across the batch and has nasty train/eval and small-batch pitfalls; layer norm normalizes across features per example, which is why transformers use it. RMSNorm strips it down further.

a free account unlocks the core curriculum tier · no card

RELATED CONCEPTS

Gradient Descent & Learning Rate The Transformer, Intuitively

PRACTICE THIS IN REAL QUESTIONS

MLOps & ML EngineeringHow is CI/CD for ML models different from traditional DevOps CI/CD?→LLM & GenAI FundamentalsWalk me through the transformer architecture end-to-end, minus the heavy math.→ML Infrastructure & GPUsCompare data, tensor, and pipeline parallelism, when do you use each, and how do they combine into 3D parallelism?→MLOps & ML EngineeringHow do you ensure reproducibility in ML workflows?→ML Infrastructure & GPUsExplain how ring all-reduce works and derive its communication cost.→ML Infrastructure & GPUsWhat do ZeRO and FSDP actually shard, and how much memory does each stage save? Where does gradient checkpointing fit?→