RoPE From First Principles

PyTorch implementation verifying the mathematical properties of Rotary Positional Embeddings (Su et al., 2021).

Derived in: "Geometric Foundations of Rotary Positional Embeddings" — LinkedIn post

Mathematical properties verified

Property	Theorem	Result
Correctness	Rotating [1,0] by π/2 gives [0,1]	Error: 4.37e-08
Norm preservation	‖R_θx‖ = ‖x‖ for all x, m	Max deviation: ~1e-07
Relative invariance	⟨f(q,m),f(k,n)⟩ = f(m−n)	Max deviation: ~1e-04
Frequency decay	θᵢ decreases as i increases	Verified numerically

Run

Run in Google Colab:

Run locally:

pip install torch matplotlib
jupyter notebook RoPE_Implementation.ipynb

Reference

Su et al. (2021) — RoFormer: Enhanced Transformer with Rotary Position Embedding
arxiv.org/abs/2104.09864