RoFormer: Enhanced Transformer with Rotary Position Embedding

Attention compares every token with every other, with no built-in sense of order. The original Transformer fixed this by adding a position vector, a set of sines and cosines, to each token's input. That works, but position and content get mixed in the same numbers, and the model has to learn how to extract relative distance. Relative-position methods added terms to the attention score instead, at the cost of extra parameters or incompatibility with faster linear attention.

RoPE, rotary position embedding, encodes position by rotation. Take the query and key vectors after their usual projections and split each into pairs of dimensions. Treat each pair as a point in a plane and rotate it by an angle equal to the token's position times a frequency; each pair has its own frequency, from fast to slow, set as 10,000 to the power of minus 2i/d, the same schedule as the original sinusoids. When a rotated query at position m meets a rotated key at position n, the angles subtract, so their dot product depends on the vectors and on n minus m, never on absolute position. Rotation also leaves vector lengths unchanged.

The authors derive this from the requirement that the score depend only on content and relative position, show an efficient elementwise way to compute it, prove a long-term decay property (an upper bound on the score shrinks as distance grows), and show it can be combined with linear attention. They test it as RoFormer: a small BLEU gain over Transformer-base in translation, faster-falling pre-training loss than BERT, mixed GLUE results, faster loss with Performer, and better accuracy on a Chinese legal document matching task when the model reads 1,024 rather than 512 characters.

The experiments are modest and the authors say they cannot fully explain why RoPE converges faster or does better on long text. RoPE spread because it is simple, has no learned parameters, and plays well with attention kernels and caches. LLaMA uses it, and context-extension methods for open models are built on adjusting its frequencies.

It is inside most open models you will deploy. LLaMA-family and many other open-weight models use RoPE. When you read a model card that mentions a "RoPE base" or "rope scaling", this is the mechanism being adjusted.

Context extension is a RoPE story. Models trained at one length are stretched to longer contexts by changing the rotation frequencies. When a customer asks whether a model really handles 128,000 tokens, the honest test is retrieval and reasoning quality at that length on their documents, because the frequencies were adjusted after training and quality can fall off.

Relative position is why long prompts behave as they do. Scores depend on distance, and the paper's bound shrinks with distance. Our computation shows a token's score against an identical token at distance 1,024 at about 21% of its value at distance 0. Put the instructions and key facts where the model is most likely to use them, and test, rather than assuming position does not matter.

Small published gains, big practical adoption. The paper's numbers are within noise in places and mix evaluation setups. Adoption came from simplicity and compatibility. When evaluating newer position methods for a customer's long-document task, measure on their data.

Fine-tuning keeps the same scheme. If you fine-tune a RoPE model on longer documents than it was trained on, the frequency settings and the training length together decide how well it generalizes; follow the model provider's guidance for scaling.