Odyssey

A decoder-only transformer specializing in long-horizon reasoning, software architecture, and autonomous software engineering.

"Think before you build."

---

Status: Research Project — Phase 7 complete · Spec v1.0.0 frozen
Language: Python 3.12+
Framework: PyTorch
Target Runtime: Phalanx Runtime
Architecture: Decoder-only Transformer

---

Odyssey Specification (source of truth)

Architecture, tensor names, shapes, tokenizer, GGUF mapping, and the Phalanx runtime contract live in:

spec/ — Specification 1.0.0

Phalanx Runtime must follow this spec; pedagogical notes in math/ are non-normative.

---

Repository Overview

Odyssey is a research repository for building a small, carefully engineered decoder-only language model optimized for reasoning, not autocomplete.

Phase	Deliverable
0	Research repository foundation
1	SentencePiece reference tokenizer
2	Owned byte-level BPE library (odyssey_tokenizer)
3	Token embedding layer (OdysseyEmbedding)
4	RoPE (OdysseyRoPE) + Phalanx numerical validation
5	RMSNorm + pre-norm residuals + Phalanx validation
6	SwiGLU (OdysseySwiGLU) + Phalanx validation
7	GQA Attention (OdysseyAttention) + Phalanx validation

flowchart TD
    RawText[Raw Text] --> Tokenizer
    Tokenizer --> TokenIDs[Token IDs]
    TokenIDs --> EmbeddingLookup[Embedding Lookup]
    EmbeddingMatrix[Embedding Matrix] --> EmbeddingLookup
    EmbeddingLookup --> Vectors[Embedding Vectors]
    Vectors --> RoPE[RoPE]
    RoPE --> RMSNorm[RMSNorm]
    RMSNorm --> SwiGLU[SwiGLU]
    SwiGLU --> TransformerBlock[Transformer Block]

Loading

---

SwiGLU Feed-Forward (Phase 6)

from model import OdysseySwiGLU, load_feed_forward_config
ffn = OdysseySwiGLU(load_feed_forward_config())
y = ffn(x)  # (B, S, D) → (B, S, D)

Cross-check against Phalanx Runtime:

python scripts/validate_swiglu.py

Docs: docs/architecture/swiglu.md · Spec: spec/feedforward.md

---

Grouped Query Attention (Phase 7)

from model import OdysseyAttention, load_attention_config, OdysseyRoPE, RopeConfig
cfg = load_attention_config()
rope = OdysseyRoPE(RopeConfig(head_dim=cfg.head_dim, rotary_dim=cfg.head_dim))
attn = OdysseyAttention(cfg, rope=rope)
y = attn(x)  # (B, S, D) → (B, S, D)

Cross-runtime check:

python scripts/validate_attention.py

Docs: docs/architecture/attention.md · Spec: spec/attention.md

RMSNorm & Residuals (Phase 5)

from model import OdysseyRMSNorm, load_norm_config, residual_add

norm = OdysseyRMSNorm(load_norm_config())
h = residual_add(x, attn(norm(x)))  # Spec pre-norm ordering

Cross-check against Phalanx Runtime (must PASS):

python scripts/validate_rmsnorm.py
# or from monorepo root:
python ../validation/test_rmsnorm.py

Docs: docs/architecture/rmsnorm.md · Spec: spec/rmsnorm.md

---

Rotary Positional Embeddings (Phase 4)

from model import OdysseyRoPE, load_rope_config

rope = OdysseyRoPE(load_rope_config())
q_rot, k_rot = rope.apply_rotary(q, k, position_offset=0)

Cross-check against Phalanx Runtime (must PASS):

python scripts/validate_rope.py
# Max Error ~5e-7  →  PASS

Docs: docs/architecture/rope.md · Spec: spec/rope.md

---

Embedding Layer (Phase 3)

from model import EmbeddingConfig, OdysseyEmbedding, load_embedding_config

config = load_embedding_config()  # configs/embedding.yaml
emb = OdysseyEmbedding(config)
x = emb(token_ids)  # (batch, seq) → (batch, seq, hidden)
print(emb.inspect().format())

Knob	Default
Vocabulary	32,000
Hidden size	768
Parameters	24,576,000
Init	Xavier uniform
Memory (fp32)	~93.75 MiB

Math notes: math/embeddings.md · Architecture: docs/architecture/embeddings.md

python scripts/benchmark_embeddings.py --visualize

---

Odyssey Tokenizer

The tokenizer is a reusable library, not model-coupled code:

tokenizer/
├── odyssey_tokenizer/   # import odyssey_tokenizer
├── cli/                 # odyssey-tokenizer
├── benchmarks/
├── tests/
├── docs/
└── sentencepiece/       # Phase 1 reference only

Public API

from odyssey_tokenizer import OdysseyTokenizer

tokenizer = OdysseyTokenizer.load("assets/tokenizer/bpe/odyssey.model")
ids = tokenizer.encode("Build authentication API")
text = tokenizer.decode(ids)

Training and Phalanx Runtime can share the same artifacts (vocab.json + merges.txt). A future Rust port should preserve identical behavior.

Training

source venv/bin/activate
python scripts/prepare_tinystories_sample.py --max-stories 50000

odyssey-tokenizer train \
  --input datasets/raw/sample.txt \
  --vocab-size 2048 \
  --max-lines 1000 \
  --output assets/tokenizer/bpe/odyssey.model

CLI Usage

odyssey-tokenizer encode --model assets/tokenizer/bpe/odyssey.model --text "Hello"
odyssey-tokenizer decode --model assets/tokenizer/bpe/odyssey.model --ids 12,45,90
odyssey-tokenizer inspect --model assets/tokenizer/bpe/odyssey.model --text "Build authentication API" --show-merges
odyssey-tokenizer benchmark --model assets/tokenizer/bpe/odyssey.model --input datasets/raw/sample.txt --limit 200
odyssey-tokenizer visualize --model assets/tokenizer/bpe/odyssey.model --input datasets/raw/sample.txt

Docs: tokenizer/README.md

---

Math Notes

Equation-level companions for each neural component live in math/ (embeddings, RoPE outline, attention outline, …), including PyTorch vs Phalanx Runtime execution notes.

---

Vision

Odyssey exists to explore what an AI model looks like when it is optimized not for code completion, but for thinking like a senior software architect.

---

Goals

• Deliberate reasoning before code generation
• Software architecture and systems design capability
• Reproducible research experiments
• Clear documentation of every architectural decision
• Small but excellent models (Tiny → Base → Pro → Max)

---

Installation

cd odyssey
python3.12 -m venv venv
source venv/bin/activate
pip install -e ".[dev]"

Verify

pytest
black --check .
ruff check .
isort --check-only .
mypy odyssey model tokenizer training evaluation
MYPYPATH=tokenizer mypy --explicit-package-bases -p odyssey_tokenizer

---

Repository Progress

Phase	Focus	Status
0	Repository setup	Complete
1	SentencePiece reference	Complete
2	Odyssey BPE library	Complete
3	Embedding layer	Complete
4	RoPE (+ Phalanx validation)	Complete
5	RMSNorm + residuals (+ Phalanx validation)	Complete
6	SwiGLU FFN (+ Phalanx validation)	Complete
7	GQA Attention (+ Phalanx validation)	Complete
8–20	Decoder block → Odyssey v1	Planned

---

Experiment Tracking

ID	Purpose	Result
ODY-0000	Repository initialization	Successful
ODY-0001	SentencePiece baseline	Successful
ODY-0002	Odyssey BPE tokenizer	Successful
ODY-0003	Token embedding layer	Successful
ODY-0004	RoPE + Phalanx parity	Successful
ODY-0005	RMSNorm + Phalanx parity	Successful
ODY-0006	SwiGLU + Phalanx parity	Successful
ODY-0007	GQA Attention + Phalanx parity	Successful

---

License

MIT — see LICENSE.