M.Sc. Mathematics | Deriving transformer architecture from first principles
Building the Mathematical Foundations of the Transformer
- one proof and one implementation at a time.
| Paper | Derivation | Implementation |
|---|---|---|
| RoPE (Su et al. 2021) | ||
| √dk Scaling (Vaswani 2017) |
Mechanistic interpretability — whether mathematical stability properties of attention mechanisms have detectable signatures in transformer circuits.