Core
Attention and Self-Attention
Attention computes a weighted sum of value vectors, where the weights come from how well each token's query matches every other token's key. Self-attention applies this within one sequence so each token can pull from all the others, and the all-pairs comparison is why cost grows with the square of sequence length. FDE loops probe it because it explains context limits, latency, and the KV cache in one mental model.
a free account unlocks the core curriculum tier · no card
Related concepts
Practise this in questions
LLM & GenAI FundamentalsWalk me through the transformer architecture end-to-end, minus the heavy math. →medium · FreeLLM & GenAI FundamentalsWhy is the dot product the similarity score in attention and embeddings, and when should you normalize to cosine? →medium · FreeSystem Design & Production EngineeringEstimate the capacity and cost of an app with 50k DAU making 10 LLM calls each. What do you provision for? →medium · FreeSystem Design & Production EngineeringDesign observability for an LLM application. What do you log, trace, and alert on, and how is it different from normal services? →medium · Free with sign-inSystem Design & Production EngineeringHow do you version and roll out prompt changes like code, review, canary, rollback, audit? →medium · Free with sign-inSystem Design & Production EngineeringDesign a job queue for long-running LLM tasks, priorities, cancellation, progress, and poison messages. →medium · Free with sign-in