rope

Rotary embeddings encode position as a rotation of the query and key vectors, so attention scores depend only on the gap between two tokens. That relative property is why RoPE generalizes past its training length where learned absolute tables fall off a cliff.

Going 4K to 1M is not one trick. It is RoPE rescaling chosen by frequency band, a long-doc continued-pretrain mix, needle evals that actually catch failures, and KV math that forces ring attention.