Hephaistos

A GPT / Llama-style language model forged from scratch in Rust — no autograd, no tensor framework, no GPU.

Hand-written forward and backward passes, every gradient verified against numerical differentiation, trained on CPU, and exported to GGUF so the result runs in llama.cpp and Ollama.

---

What this is

Hephaistos is a complete, modern decoder-only transformer implemented in plain Rust. The only dependencies are a BPE tokenizer, an RNG, a thread pool, and JSON — there is no deep-learning framework underneath. Every matrix multiply, attention head, normalization, and the entire backpropagation are written by hand and run on flat Vec<f32> buffers.

It is small enough to read end to end, yet it is a real Llama-class architecture: RMSNorm, rotary position embeddings, SwiGLU feed-forward, AdamW with decoupled weight decay, dropout, validation-based checkpointing, top-k sampling, and a GGUF exporter that other tools can load.

The named modules build on each other in numbered phases (you can see them in the commit history and the doc comments), taking the project from an empty main to a trained model you can chat with in llama.cpp.

Why it's interesting

• Backprop by hand, then proved correct. Before a single line of backward code was trusted, a gradient-check harness was built — the "truth machine." Every analytic gradient is compared against an f64 central-difference estimate. "Loss goes down" does not prove gradients are right; matching numerical gradients does.
• One implementation, two precisions. The forward and backward passes are generic over the float type. The real model runs in f32; the gradient checker re-runs the exact same code in f64 so numerical round-off can't hide a bug.
• A genuine Llama block, not a toy: token embedding → RMSNorm → RoPE attention → RMSNorm → SwiGLU MLP, residual stream throughout, all biases dropped the way Llama does.
• Honest CPU performance. A cache-blocked matmul and an 8-lane dot product let the compiler emit SIMD, and rayon parallelizes over rows and the batch — while staying bit-identical to the naive reference loop.
• It leaves the sandbox. The trained checkpoint is exported as a llama-architecture GGUF (RoPE weights permuted to GGUF's interleaved layout, byte-level BPE tokenizer embedded), so it runs in llama.cpp and Ollama.
• Trustworthy by construction. cargo test runs the full gradient-check suite on every commit (see CI).

Architecture at a glance

Component	Choice
Block	Pre-norm decoder: x + Attn(RMSNorm(x)), then x + SwiGLU(RMSNorm(x))
Normalization	RMSNorm (no mean-centering, no bias), cached rstd for backward
Positions	RoPE (rotary), applied to q/k inside attention — no learned table
Attention	Causal multi-head self-attention, numerically-stable softmax
Feed-forward	SwiGLU: (silu(x·W₁) ⊙ x·W₃)·W₂, hidden width ⌊8/3·n_embd⌋
Output head	Untied lm_head projection (separate from the token embedding)
Optimizer	AdamW with decoupled weight decay + bias correction
Regularization	Inverted dropout (embeddings, attention weights, attn & MLP outputs)
Tokenizer	Byte-level BPE, trained from scratch on the corpus
Sampling	Temperature + top-k, sliding window of block_size real tokens
Export	GGUF (llama arch) → llama.cpp / Ollama

Quickstart

# 1. Provide a UTF-8 text corpus (data/ is git-ignored). tinyshakespeare works great:
mkdir -p data
curl -L -o data/input.txt \
  https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt

# 2. Run the full pipeline: train a BPE tokenizer, build a model, gradient-check it,
#    train on CPU, and sample from the trained checkpoint.
cargo run --release

The first run trains the tokenizer and caches it; later runs reuse it.

What you'll see

A ~1.05M-parameter model (4 layers, 4 heads, n_embd 128, vocab 1024, block 64) trains on tinyshakespeare on a laptop CPU in a few seconds:

model: 1049216 params
loss = 6.9847  (expected ~ln(1024) = 6.9315)        # untrained ≈ uniform guess

Phase 4 harness: softmax+CE analytic vs numerical, max rel error = 2.13e-6
Phase 5 gradient check (max rel error vs f64 numerical, < 1e-4 = correct):
  wte       5.48e-7    qkvw      1.99e-6    w1        1.36e-7
  lm_head   1.87e-8    ...       (every tensor checked)
worst tensor: 1.99e-6                                # backprop is correct

Phase 6 training (start loss ~ln(1024) = 6.93):
step    1: train 6.94  val 6.90
step  300: train 4.60  val 4.46
trained 300 steps in 6.72s (44.6 steps/s)

Phase 7 sample (temperature 0.8, top-k 40):
NERREO:
The have fual!
Ofning not his litii' the fay.
...

300 steps is just a smoke test — it's already producing Shakespeare-shaped text. Let it run longer for cleaner samples.

The bigger run, validated against a reference

The same from-scratch code trains a 15.74M-parameter Llama (8 layers, 8 heads, n_embd 384, vocab 2048, block 256) on a Greek corpus, with hyperparameters chosen to match an independent MLX reference training curve. The token stream and 90/10 split are kept identical so the loss trajectories are directly comparable:

PHASE8=1 PHASE8_STEPS=2000 cargo run --release
# MLX reference: iter250 t4.63/v5.04, iter500 t3.88/v4.46, val-min 3.97 @ 1500

This is the project's correctness claim taken all the way to the end: a hand-written CPU transformer tracking a mature GPU framework step for step.

The capstone: the whole loop from scratch

The bigger run above still borrowed one artifact: the reference implementation's pre-trained tokenizer, reused so both loss curves scored the exact same token stream. The capstone run drops that last crutch:

CAPSTONE=1 cargo run --release    # CAPSTONE_STEPS=n to override the default 2000

It trains a fresh byte-level BPE (vocab 2048) on the corpus with Cadmus — the from-scratch tokenizer this project links as its only tokenizer dependency — encodes a new token stream, trains the same 15.74M model on it, and exports the best-validation checkpoint as a llama-arch GGUF with the Cadmus tokenizer embedded. Raw text in, runnable model out, every stage hand-written.

Two results worth reporting:

• The from-scratch tokenizer out-compresses the reference's: the corpus encodes to 1.59M tokens vs 1.67M (≈5% fewer).
• Model quality lands within ~1% of the MLX reference. Best val loss reads 4.22 vs the reference's 3.97, but per-token losses aren't comparable across different tokenizations — fewer tokens must each carry more information. On the tokenization-independent metric, total corpus NLL, the runs are ≈1% apart: 4.22 × 1.59M ≈ 6.72M nats vs 3.97 × 1.67M ≈ 6.64M.

The exported GGUF runs in Talos, the companion from-scratch inference engine, which decodes it on the Apple GPU and independently measures perplexity 56 (≈4.03 nats/token) on held-out corpus text — trainer and inference engine agreeing through nothing but a GGUF file.

Run your model in llama.cpp / Ollama

EXPORT_GGUF=1 cargo run --release   # writes hephaistos.gguf (llama arch)

llama-cli -m hephaistos.gguf -p "ΜΗΝΙΝ ἄειδε"

The exporter maps the internal tensor names to GGUF's llama names, permutes the q/k weights from the rotate-half RoPE layout to GGUF's interleaved layout, and embeds the byte-level BPE tokenizer — so external tools load it as an ordinary Llama model.

Project layout

File	Phase	Responsibility
src/main.rs	0	Entry point, naive reference matmul, phase orchestration
src/data.rs	1	Byte-level BPE training, u16 token bins, random [B, T] batching
src/model.rs	2 · 5	Forward + backward for the whole Llama block (generic over f32/f64)
src/gradcheck.rs	4	The "truth machine": analytic vs numerical gradient verification
src/train.rs	6	AdamW training loop, validation, early stopping, best-checkpointing
src/sample.rs	7	Autoregressive generation with temperature + top-k
src/gguf.rs	11	GGUF writer + byte-level BPE tokenizer embedding

Phase roadmap

0 scaffold & reference matmul → 1 data & tokenizer → 2 forward pass → 3 sanity (untrained loss ≈ ln(vocab)) → 4 gradient-check harness → 5 full hand-written backward, every tensor checked → 6 training loop → 7 sampling → 8 scaled run vs MLX reference → 11 GGUF export.

Design notes

• Flat buffers, named offsets. Parameters and activations each live in one contiguous Vec, with a layout struct giving every tensor its (offset, len) — the same approach as Karpathy's llm.c, which makes the buffers easy to checkpoint and cross-check.
• Activation arena. The forward pass caches every intermediate the backward needs (attention softmax, RMSNorm rstd, pre-SiLU gate, …) so backprop is a straight reverse walk.
• Deterministic where it matters. Plain forward (eval, sampling, gradient check) is fully deterministic; dropout masks are only sampled by the training forward and reused by its paired backward.
• Parallelism without surprises. Threads only ever write disjoint regions, so no floating-point reduction is reordered across threads — results don't depend on the thread count.

Tests

cargo test

11 tests, including hand-checked matmuls, the softmax+CE harness validation, and every parameter tensor's gradient checked against f64 numerical differentiation (< 1e-4 required; actual worst case ~2e-6).

Note on clippy: the hand-written ops take wide argument lists by design (flat-buffer offsets passed explicitly), so clippy::too_many_arguments fires intentionally and is not enforced in CI. The correctness gate is cargo test.

Acknowledgements

Inspired by Andrej Karpathy's llm.c and nanoGPT, and standing on the architecture papers behind Llama: RoPE, RMSNorm, SwiGLU, and AdamW. The banner is an ASCII-style render of Luca Giordano's The Forge of Vulcan, generated with ren-ascii-sance.

License

MIT © 2026 Tim Suskov