Skip to content

Commit 2c74a0c

Browse files
committed
review: close what a hostile reader would find first
Adversarial pass over the tree. Nothing sensitive was in it, so this is about the things someone would pull out to argue the repo is not serious. Every unsafe block in olmo2.rs now carries a SAFETY justification. There were thirteen and none of them said why they were sound, which is the first thing a Rust reviewer looks for. Fourteen unit tests for the OLMo module's pure logic, so cargo test covers it rather than only the CLI selftest. They pin the parts that are easy to get quietly wrong: QK-norm spanning the projection rather than a head, the norm being sliced with its projection under tensor parallelism, and each rejection naming its reason. The docs contradicted the README, which is worse than either being wrong alone. The roadmap listed real full-depth checkpoint loading through the serving path as open when it now ships, hardware-support did not list the OLMo gates, and the sandbox doc said the answers are synthetic without the qualifier that they are only synthetic by default. All three now agree with what the code does, and the roadmap says plainly what M6 still lacks: batching, sampling, a prefill kernel, tensor parallelism, and performance. just ci claimed to be everything CI runs and had drifted, so it now includes the OLMo gate. CONTRIBUTING explains how to write a gate that fails when its claim is false, using the two OLMo gates as the worked example. README gains what a first-timer actually needs: requirements before they commit, the model's own licence and the fact that it is a base model rather than an instruction-tuned one, and a troubleshooting section for the failures they will really hit. Also checked and clean: no secrets, no internal paths, no AI attribution, no emails, no broken links or anchors, no TODO or debug leftovers in shipped paths, no unescaped model output reaching the DOM in either page, and JSON escaping that handles the newlines a real model emits.
1 parent 37a6869 commit 2c74a0c

7 files changed

Lines changed: 385 additions & 7 deletions

File tree

CONTRIBUTING.md

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,7 @@ just check # cargo check --workspace --all-targets
5858
just test # cargo test --workspace
5959
just check-model-api # the public model API contract gate
6060
just check-demo # the GPU-free demo sandbox contract gate
61+
just check-olmo # the OLMo 2 checkpoint contract gate
6162
```
6263

6364
For model API changes, the focused checks are:
@@ -69,6 +70,22 @@ python3 tools/check_model_api_public_examples.py
6970
cargo test --locked --manifest-path examples/model-api-plugin/Cargo.toml
7071
```
7172

73+
If you add or change a model architecture, the gates for it belong next to the
74+
code and should be written so they fail when the claim is false rather than when
75+
the arithmetic is merely off. The OLMo 2 gates are the worked example. The
76+
attention gate would pass a kernel that ignored head indices, so it perturbs KV
77+
heads and requires them to move. The QK-norm gate would pass Qwen3-style
78+
per-head normalisation, so it perturbs one head and requires a different head's
79+
output to change. Write the test that separates your architecture from the one
80+
it most resembles.
81+
82+
```bash
83+
mainarch olmo2-preflight-selftest # CPU-only
84+
mainarch gpu-mha-attention-equivalence-selftest # needs a GPU
85+
mainarch gpu-olmo2-qk-rope-selftest
86+
mainarch gpu-olmo2-post-norm-selftest
87+
```
88+
7289
`tools/check_model_api_public_examples.py` runs every public model API command
7390
and asserts the receipt lines, including the `launch_executable: ready=false`
7491
boundary. If you extend the API, extend that gate in the same change.

README.md

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,19 @@ of live kernel execution without a vendor runtime as you can ask for. Then it
6565
serves the demo from that same binary, with the decode lane running on a real
6666
GPU.
6767

68+
### What you need
69+
70+
The CPU lane wants a Linux x86-64 host with Docker or Podman and about 6 GB of
71+
disk for the build. No GPU, no ROCm, nothing else.
72+
73+
The GPU lane wants an AMD GPU reachable at `/dev/kfd` and `/dev/dri`, and your
74+
user in the `render` group. The kernels are compiled for **gfx950**, which is
75+
MI355X on CDNA4. Other architectures will load the wrong ISA, and
76+
`mainarch probe` will tell you what you actually have.
77+
78+
Running OLMo 2 on top of that wants roughly 6 GB of disk for the checkpoint and
79+
about 3 GB of VRAM once it is resident, which is nothing on a 288 GB card.
80+
6881
### Running the real model
6982

7083
`just demo` doesn't download six gigabytes behind your back, so the model is a
@@ -428,6 +441,20 @@ and it reuses every kernel that is already gated.
428441
It runs at about 125 ms/token for 16 layers with FP4 KV, unfused and untuned.
429442
Correctness came first and nothing here has been optimised.
430443

444+
One thing worth knowing before you type a question at it: `OLMo-2-0425-1B` is a
445+
**base** model, not an instruction-tuned one, so it completes text rather than
446+
holding a conversation. Give it `The capital of France is` and it does the right
447+
thing. Give it a chat-style instruction and it will drift into whatever corpus
448+
pattern looks likeliest. The Instruct variant is the same architecture and loads
449+
without any change here if you want the other behaviour.
450+
451+
The model is © the Allen Institute for AI and released under Apache 2.0, the
452+
same licence as this repository. `just olmo-fetch` downloads it from Hugging
453+
Face and nothing about it is redistributed here. See
454+
[allenai/OLMo-2-0425-1B](https://huggingface.co/allenai/OLMo-2-0425-1B) for the
455+
model card, and [the OLMo 2 paper](https://arxiv.org/abs/2501.00656) for how it
456+
was trained.
457+
431458
```bash
432459
just olmo-gates # the four gates, on hardware
433460
mainarch olmo2-preflight-selftest # CPU-only
@@ -562,6 +589,42 @@ Fuller detail and the validation tiers live in `docs/hardware-support.md`.
562589

563590
---
564591

592+
## When it doesn't work
593+
594+
**`just probe` says the device is unavailable.** Your user needs to be in the
595+
`render` group *before* the container starts, because `keep-groups` passes
596+
through the groups you already had. Add yourself, then fully reconnect rather
597+
than reopening a shell, since the group membership is captured at session start.
598+
599+
**`just demo` takes the CPU lane on a machine that has a GPU.** It looks for a
600+
KFD topology node with SIMDs and a real gfx target version, so if `/dev/kfd`
601+
isn't passed through it will quietly and correctly fall back. Reopen using the
602+
`.devcontainer/gpu` configuration, and check `ls -l /dev/kfd /dev/dri/renderD*`
603+
inside the container.
604+
605+
**`--node 2` doesn't exist on your machine.** That argument is a KFD topology
606+
node id, not an ordinal GPU index, and the numbering depends on your host.
607+
`mainarch probe` lists the real ones. `just demo` and `just olmo` work it out
608+
for you.
609+
610+
**The port is already taken.** `MAINARCH_DEMO_BIND=127.0.0.1:9090 just demo`.
611+
612+
**`olmo2-preflight` refuses your checkpoint.** That is the preflight working. It
613+
names the reason, and the three it will usually give are a `model_type` that
614+
isn't `olmo2`, a `head_dim` that isn't 128, or an `input_layernorm` tensor, which
615+
means the checkpoint is pre-norm and this path implements post-norm. None of
616+
those are things a flag can force, because forcing them would produce wrong
617+
numbers rather than an error.
618+
619+
**A kernel dispatch fails with a VA guard error.** That is also working. The
620+
guard checks every buffer span against what the kernel will actually address,
621+
and it fires before the GPU sees a bad pointer. The message names the kernel, the
622+
argument, and the span it expected.
623+
624+
**Generation is slow.** It is. About 125 ms/token, unfused, untuned, with FP4 KV
625+
and a host round trip per layer. Nothing in this path has been optimised, and the
626+
roadmap is honest about what would change that.
627+
565628
## Reading it
566629

567630
If you want to understand how this works rather than just run it, here's the

0 commit comments

Comments
 (0)