llm internals

Every chat model decodes with top-p, but few candidates can build it from logits: softmax, sort, take the smallest set of tokens whose cumulative probability crosses p, renormalize, sample. The screen for whether you understand decoding, not just call an API.

The loss every language model trains on, built from logits with no torch in sight. The screen for whether you understand log-softmax and the numerical-stability trick, not just import nn.CrossEntropyLoss.

BPE is how GPT and most LLMs turn text into tokens, and building a tiny trainable version from a corpus is a favorite deep-dive. The screen for whether you understand the merge loop, deterministic tie-breaking, and round-trip decode, not just tiktoken.encode.