Scaling Laws

Scaling laws say a model's loss falls as a smooth power law in its parameter count, training data, and compute, so you can predict the payoff of a bigger run before you make it. The Chinchilla result showed that for a fixed compute budget you should grow data and parameters together (roughly 20 tokens per parameter), revealing that many early giant models were badly undertrained. FDE loops test this because it governs every model-size and token-budget decision.