Information Theory for ML: Entropy, Cross-Entropy, KL and Perplexity

Four quantities from information theory keep showing up in ML: entropy measures the average surprise in a distribution, cross-entropy is the loss that trains classifiers and language models, KL divergence measures how far one distribution sits from another, and perplexity is the intuitive branching-factor view of a language model's loss. Knowing where each appears separates people who tuned a loss function from people who only imported one.