404khai · 404khai · Jul 29, 2026 · Jul 29, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -16,6 +16,19 @@ and this project aims to adhere to [Semantic Versioning](https://semver.org/).
 
 ---
 
+## [0.4.0] — 2026-07-29
+
+### Added
+
+- LLaMA-style RoPE (`model.OdysseyRoPE`, cache, math, visualizer)
+- `configs/model.yaml` with rope hyperparameters
+- Cross-implementation validator `scripts/validate_rope.py` (vs Phalanx)
+- Benchmarks + assets under `assets/rope/`
+- Papers: RoFormer, attention positional encodings, LLaMA RoPE
+- Experiment `ODY-0004` (validation PASS, max abs error ~4.8e-7)
+
+---
+
 ## [0.3.0] — 2026-07-28
 
 ### Added

diff --git a/EXPERIMENTS.md b/EXPERIMENTS.md
@@ -81,3 +81,21 @@ Details: [experiments/ODY-0002/README.md](experiments/ODY-0002/README.md)
 | Lessons | Embedding is a gather not a matmul; `math/` notes should ship with each neural phase; weight tying deferred |
 
 Details: [experiments/ODY-0003/README.md](experiments/ODY-0003/README.md)
+
+---
+
+## ODY-0004 — Rotary Positional Embeddings
+
+| Field | Value |
+| --- | --- |
+| ID | ODY-0004 |
+| Date | 2026-07-29 |
+| Phase | 4 |
+| Purpose | LLaMA-style RoPE + Phalanx numerical parity |
+| Config | `configs/model.yaml` / experiment `config.yaml` |
+| theta / rotary_dim | 10000 / 128 |
+| Result | **Successful** |
+| Validation | Max abs error ≈ 4.8e-7 vs Phalanx (`PASS` @ 1e-6) |
+| Lessons | Keep inv_freq + adjacent-pair rotate bit-identical to Runtime; validate every layer |
+
+Details: [experiments/ODY-0004/README.md](experiments/ODY-0004/README.md)
diff --git a/PAPERS.md b/PAPERS.md
@@ -39,12 +39,21 @@ Math companion: [math/embeddings.md](math/embeddings.md)
 
 ---
 
+## Phase 4 — RoPE *(complete)*
+
+| Paper / Study | Summary |
+| --- | --- |
+| RoFormer (Su et al.) | [papers/roformer.md](papers/roformer.md) |
+| Transformer positional encodings | [papers/attention_position.md](papers/attention_position.md) |
+| LLaMA RoPE notes | [papers/llama_rope.md](papers/llama_rope.md) |
+
+---
+
 ## Planned Reading (later phases)
 
 | Topic | Canonical paper / resource | Phase |
 | --- | --- | --- |
 | Attention | *Attention Is All You Need* (Vaswani et al., 2017) | 6+ |
-| RoPE | *RoFormer* (Su et al.) | 4 |
 | RMSNorm | *Root Mean Square Layer Normalization* | 5 |
 | SwiGLU | *GLU Variants Improve Transformer* (Shazeer) | 7 |
 | GPT-style LMs | GPT / Llama technical reports | 9–10 |

diff --git a/README.md b/README.md
@@ -6,7 +6,7 @@
 
 ---
 
-**Status:** Research Project — Phase 3 complete · **Spec v1.0.0** frozen  
+**Status:** Research Project — Phase 4 complete · **Spec v1.0.0** frozen  
 **Language:** Python 3.12+  
 **Framework:** PyTorch  
 **Target Runtime:** [Phalanx Runtime](https://github.com/404khai/phalanx)  
@@ -34,6 +34,7 @@ Odyssey is a research repository for building a small, carefully engineered deco
 | 1 | SentencePiece **reference** tokenizer |
 | 2 | **Owned** byte-level BPE library (`odyssey_tokenizer`) |
 | 3 | **Token embedding layer** (`OdysseyEmbedding`) |
+| 4 | **RoPE** (`OdysseyRoPE`) + Phalanx numerical validation |
 
 ```mermaid
 flowchart TD
@@ -42,12 +43,32 @@ flowchart TD
     TokenIDs --> EmbeddingLookup[Embedding Lookup]
     EmbeddingMatrix[Embedding Matrix] --> EmbeddingLookup
     EmbeddingLookup --> Vectors[Embedding Vectors]
-    Vectors --> RoPE[RoPE Phase 4]
+    Vectors --> RoPE[RoPE]
     RoPE --> TransformerBlock[Transformer Block]
 ```
 
 ---
 
+## Rotary Positional Embeddings (Phase 4)
+
+```python
+from model import OdysseyRoPE, load_rope_config
+
+rope = OdysseyRoPE(load_rope_config())
+q_rot, k_rot = rope.apply_rotary(q, k, position_offset=0)
+```
+
+Cross-check against Phalanx Runtime (must PASS):
+
+```bash
+python scripts/validate_rope.py
+# Max Error ~5e-7  →  PASS
+```
+
+Docs: [`docs/architecture/rope.md`](docs/architecture/rope.md) · Spec: [`spec/rope.md`](spec/rope.md)
+
+---
+
 ## Embedding Layer (Phase 3)
 
 ```python
@@ -180,7 +201,8 @@ MYPYPATH=tokenizer mypy --explicit-package-bases -p odyssey_tokenizer
 | 1 | SentencePiece reference | **Complete** |
 | 2 | Odyssey BPE library | **Complete** |
 | 3 | Embedding layer | **Complete** |
-| 4–20 | RoPE → Odyssey v1 | Planned |
+| 4 | RoPE (+ Phalanx validation) | **Complete** |
+| 5–20 | RMSNorm → Odyssey v1 | Planned |
 
 ---
 
@@ -192,6 +214,7 @@ MYPYPATH=tokenizer mypy --explicit-package-bases -p odyssey_tokenizer
 | ODY-0001 | SentencePiece baseline | Successful |
 | ODY-0002 | Odyssey BPE tokenizer | Successful |
 | ODY-0003 | Token embedding layer | Successful |
+| ODY-0004 | RoPE + Phalanx parity | Successful |
 
 ---
 

diff --git a/ROADMAP.md b/ROADMAP.md
@@ -42,11 +42,12 @@ Development proceeds **one phase at a time**. Complete, document, commit, then w
 
 ---
 
-## Phase 4 — Rotary Positional Embeddings
+## Phase 4 — Rotary Positional Embeddings *(complete)*
 
-- RoPE implementation
-- Visualization
-- Tests
+- LLaMA-style RoPE (`OdysseyRoPE`)
+- Cos/sin cache, partial rotary dims, linear scaling
+- Cross-implementation validation vs Phalanx (`scripts/validate_rope.py`)
+- ODY-0004 baseline
 
 ---
 

diff --git a/assets/rope/cos_sin_curves.png b/assets/rope/cos_sin_curves.png
diff --git a/assets/rope/inv_freq.png b/assets/rope/inv_freq.png
diff --git a/assets/rope/rotation_demo.png b/assets/rope/rotation_demo.png
diff --git a/configs/default.yaml b/configs/default.yaml
@@ -2,8 +2,8 @@
 # All hyperparameters should eventually flow from config files.
 
 experiment:
-  id: ODY-0003
-  name: odyssey-embedding-baseline
+  id: ODY-0004
+  name: odyssey-rope-baseline
   seed: 42
 
 model:
@@ -20,6 +20,14 @@ model:
     init_std: 0.02
     device: cpu
     dtype: float32
+  rope:
+    theta: 10000.0
+    rotary_dim: 64
+    max_position_embeddings: 2048
+    scaling: none
+    scaling_factor: 1.0
+    device: cpu
+    dtype: float32
 
 tokenizer:
   path: assets/tokenizer/bpe/odyssey.model

diff --git a/configs/model.yaml b/configs/model.yaml
@@ -0,0 +1,30 @@
+# Odyssey model configuration (Phase 4+)
+
+model:
+  name: odyssey-tiny
+  vocab_size: 32000
+  hidden_size: 768
+  intermediate_size: 2048
+  num_layers: 12
+  num_heads: 12
+  num_kv_heads: 12
+  context_length: 2048
+
+  embedding:
+    padding_idx: 0
+    init_strategy: xavier_uniform
+    init_std: 0.02
+    device: cpu
+    dtype: float32
+
+  # RoPE — must stay aligned with Phalanx layers::Rope / Odyssey Spec v1
+  rope:
+    theta: 10000.0
+    # head_dim defaults to hidden_size / num_heads (= 64 for Tiny)
+    # Experiment ODY-0004 also exercises rotary_dim=128 with head_dim=128
+    rotary_dim: 64
+    max_position_embeddings: 2048
+    scaling: none          # none | linear  (NTK / YaRN deferred)
+    scaling_factor: 1.0
+    device: cpu
+    dtype: float32
diff --git a/docs/architecture/README.md b/docs/architecture/README.md
@@ -7,7 +7,7 @@ Notes for Odyssey's decoder-only transformer.
 | Doc | Phase | Status |
 | --- | --- | --- |
 | [embeddings.md](embeddings.md) | 3 | Complete |
-| RoPE | 4 | Planned |
+| [rope.md](rope.md) | 4 | Complete |
 | RMSNorm | 5 | Planned |
 | Attention | 6 | Planned |
 

diff --git a/docs/architecture/rope.md b/docs/architecture/rope.md
@@ -0,0 +1,64 @@
+# Rotary Positional Embeddings (RoPE)
+
+## Motivation
+
+Token embeddings alone do not encode order. Absolute position tables add a vector per index but extrapolate poorly. **RoPE** rotates Q/K feature pairs by an angle that grows with position so attention sees relative offsets.
+
+## Mathematical Derivation
+
+See Spec [`spec/rope.md`](../../spec/rope.md) and pedagogy [`math/rope.md`](../../math/rope.md).
+
+\[
+\theta_i = 10000^{-2i/d_r}
+\]
+
+\[
+\begin{pmatrix} x_0' \\ x_1' \end{pmatrix}
+=
+\begin{pmatrix} \cos m\theta & -\sin m\theta \\ \sin m\theta & \cos m\theta \end{pmatrix}
+\begin{pmatrix} x_0 \\ x_1 \end{pmatrix}
+\]
+
+Rotation preserves pairwise (and thus subspace) L2 norms.
+
+## Implementation
+
+| Module | Role |
+| --- | --- |
+| `model/rope_math.py` | inv_freq + adjacent rotate |
+| `model/rope_cache.py` | lazy cos/sin cache |
+| `model/rope.py` | `OdysseyRoPE` public API |
+| `configs/model.yaml` | θ, rotary_dim, scaling |
+
+```python
+from model import OdysseyRoPE, load_rope_config
+rope = OdysseyRoPE(load_rope_config())
+q_rot, k_rot = rope.apply_rotary(q, k, position_offset=0)
+```
+
+## Tensor Shapes
+
+| Layout | Shape |
+| --- | --- |
+| Training | `(B, S, H, d)` |
+| Phalanx | `(S, H, d)` or `(S, d)` |
+| Output | Same as input |
+
+## Complexity
+
+- Time: \(O(B S H d)\) (same order as a cheap elementwise pass)
+- Cache memory: \(O(S_{\max} \cdot d_r)\) for cos+sin
+
+## Phalanx Compatibility
+
+Cross-check with:
+
+```bash
+python scripts/validate_rope.py
+```
+
+Tolerance default `1e-6` (float32). Report: `experiments/ODY-0004/rope_validation.json`.
+
+## Visualization
+
+`assets/rope/` — inv-freq spectrum, cos/sin curves, 2D rotation demo.
diff --git a/docs/architecture/rope_math.md b/docs/architecture/rope_math.md
@@ -0,0 +1,29 @@
+# RoPE Mathematics (Architecture Companion)
+
+Normative equations live in [`spec/rope.md`](../../spec/rope.md).
+
+## Variables
+
+| Symbol | Meaning |
+| --- | --- |
+| \(d\) | `head_dim` |
+| \(d_r\) | `rotary_dim` (even, ≤ \(d\)) |
+| \(\theta\) | base frequency (`theta`, default 10000) |
+| \(m\) | absolute position |
+| \(m'\) | \(m / \mathrm{factor}\) under linear scaling |
+| \(\omega_i\) | \(\theta^{-2i/d_r}\) |
+
+## Why Adjacent Pairing
+
+LLaMA stores pairs as consecutive dims. The complex view \((x_{2i}+i x_{2i+1})\) maps directly onto that layout without a gather/transpose of even/odd channels across the full head.
+
+## Why Not Rotate V
+
+RoPE’s relative-phase argument applies to the \(QK^\top\) product. Rotating \(V\) is not part of the standard derivation and would mix content with position unnecessarily.
+
+## Cache Layout (shared with Phalanx)
+
+```text
+cos[pos, pair], sin[pos, pair]
+flat index = pos * (d_r/2) + pair
+```
diff --git a/docs/architecture/rope_visualization.md b/docs/architecture/rope_visualization.md
@@ -0,0 +1,17 @@
+# RoPE Visualization Guide
+
+Artifacts are written to `assets/rope/` by `model.rope_visualizer.export_rope_assets`.
+
+| Figure | Meaning |
+| --- | --- |
+| `inv_freq.png` | Wavelength spectrum across pair indices |
+| `cos_sin_curves.png` | Cos/sin vs position for one pair |
+| `rotation_demo.png` | Unit vector `(1,0)` tracing a circle across positions |
+
+Regenerate:
+
+```bash
+python -c "from model.rope_visualizer import export_rope_assets; export_rope_assets()"
+```
+
+Intuition: each pair is a 2D vector spun by an angle \(m\omega_i\); magnitude stays constant, only orientation encodes position.
diff --git a/experiments/ODY-0004/README.md b/experiments/ODY-0004/README.md
@@ -0,0 +1,40 @@
+# ODY-0004 — Rotary Positional Embeddings
+
+| Field | Value |
+| --- | --- |
+| Phase | 4 |
+| Date | 2026-07-29 |
+| Purpose | Implement LLaMA-style RoPE + cross-validate vs Phalanx |
+| Result | **Successful** |
+
+## Configuration
+
+| Knob | Value |
+| --- | --- |
+| theta | 10000 |
+| rotary_dim | 128 (experiment) / 64 (Tiny default) |
+| head_dim | 128 (experiment) / 64 (Tiny) |
+| max_position_embeddings | 4096 (experiment) |
+| scaling | none (linear supported) |
+
+## Cross-Implementation Validation
+
+```bash
+python scripts/validate_rope.py --head-dim 128 --rotary-dim 128 --max-position 4096
+```
+
+See `rope_validation.json` for max/mean absolute error vs Phalanx `layers::Rope`.
+
+| Metric | Value |
+| --- | --- |
+| Max abs error | ≈ 4.77e-7 |
+| Mean abs error | ≈ 5.62e-9 |
+| Tolerance | 1e-6 |
+| Status | **PASS** |
+
+## Artifacts
+
+- Metrics: `metrics.json`
+- Validation: `rope_validation.json`
+- Plots: `assets/rope/`
+- Config snapshot: `config.yaml`
diff --git a/experiments/ODY-0004/config.yaml b/experiments/ODY-0004/config.yaml
@@ -0,0 +1,11 @@
+rope:
+  theta: 10000.0
+  head_dim: 128
+  rotary_dim: 128
+  max_position_embeddings: 4096
+  scaling: none
+  scaling_factor: 1.0
+
+experiment:
+  id: ODY-0004
+  seed: 0