Core
RoPE and Positional Encodings
Self-attention has no built-in sense of word order, so transformers inject position information into the token vectors. Rotary position embeddings (RoPE) rotate query and key vectors by a position-dependent angle so relative position falls out of the dot product, which is why RoPE underpins almost every long-context model and why extending a context window means rescaling RoPE.
a free account unlocks the core curriculum tier · no card
Related concepts
Practise this in questions
LLM & GenAI FundamentalsWhat is RoPE, why does relative position help, and why does it extend to long context better than learned absolute embeddings? →medium · PremiumML Infrastructure & GPUsExtend a 4K-context 7B to 1M tokens: RoPE scaling, the data recipe, eval, and the inference math. →expert · PremiumLLM & GenAI FundamentalsWhat is a context window, and what are its practical limits in production? →easy · FreeML Infrastructure & GPUsLong-context traffic is OOM-ing your serving fleet at 3am. Triage the KV-cache blowup. →hard · PremiumLLM & GenAI FundamentalsWhen do you use long context, RAG, or prompt caching, and what are the failure modes of each? →hard · PremiumLLM & GenAI FundamentalsDoes increasing an LLM context window from 32K to 1M tokens guarantee reliable recall? →hard · Premium