Skip to content

Commit 1fb0836

Browse files
committed
docs: fix non-claims the OLMo work made false
The scope section still said there was no OpenAI-compatible serving of real weights, which the opening of the same README now disproves. A reader would reasonably conclude nobody checked. It now says what is actually true. Real weights are served over an OpenAI-shaped endpoint, one request at a time, with no batching, no scheduling, no multi-tenancy and no auth, because the KV cache holds one sequence. The reduced-scale claim was also ambiguous once two model paths exist, so it now separates them. OLMo runs full depth and full vocabulary from a real checkpoint. The Qwen path is a synthetic-weight proof at reduced scale and says so. Adds two claims that were missing rather than wrong. FP4 KV output is coherent but unmeasured against an FP16 baseline. Performance is unoptimised, at about 125 ms/token with a host round trip per layer. Also replaces an 'as above' that no longer referred to anything.
1 parent 8ed0139 commit 1fb0836

1 file changed

Lines changed: 24 additions & 7 deletions

File tree

README.md

Lines changed: 24 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -534,8 +534,12 @@ just check-model-api
534534

535535
The numbers above are easy to over-read, so here's where the edges actually are.
536536

537-
**This isn't a production serving stack.** No OpenAI-compatible serving of real
538-
weights, no continuous batching, no multi-tenant scheduling, no auth or quota.
537+
**This isn't a production serving stack.** It does serve real weights over an
538+
OpenAI-shaped endpoint, which is the whole point of the OLMo work, but it serves
539+
exactly one request at a time. There's no batching, no continuous batching, no
540+
scheduling across sequences, no multi-tenancy, no auth and no quota. The KV cache
541+
holds one sequence, so a second concurrent request waits. That's honest for what
542+
this is and it is nowhere near what a production server does.
539543

540544
**`just demo` on its own is still synthetic.** It serves a real HTTP seam, and on
541545
hardware there's real GPU work behind it, but the text coming back is scripted.
@@ -544,11 +548,17 @@ endpoint serves real completions instead. `/v1/models` and `/api/health` both
544548
report `synthetic: false` when that's true, so a client can tell the two apart
545549
without reading the source.
546550

547-
**The model loop runs at reduced scale.** Real head geometry and real per-op
548-
math, with reduced depth, vocabulary and expert count. It isn't a 94-layer,
549-
full-vocabulary, real-checkpoint proof.
551+
**Two model paths, and only one of them is real.** OLMo 2 runs at full depth and
552+
full vocabulary from a real checkpoint. The Qwen3-235B-A22B path
553+
(`mainarch model-decode`) is a synthetic-weight proof at reduced depth,
554+
vocabulary and expert count, keeping the real head geometry and per-op math. It
555+
demonstrates that the primitives compose into a working token loop. It is not a
556+
94-layer real-checkpoint Qwen.
550557

551-
**The model API doesn't execute.** As above.
558+
**The model API doesn't execute.** `mainarch-core::model_api` compiles and
559+
validates model definitions and emits fingerprinted receipts. It does not lower
560+
a graph into AQL, allocate buffers or submit queues, and its own receipts assert
561+
`launch_executable: ready=false`. The OLMo path does not go through it.
552562

553563
**The numbers come from a single host.** One machine with 8 MI355X cards, one
554564
software version. They're reproducible with the commands given, on that hardware.
@@ -557,7 +567,14 @@ software version. They're reproducible with the commands given, on that hardware
557567
table.
558568

559569
**FP4 accuracy is an open question.** Around 15% rel-L2 raw on the primitive, and
560-
what that costs a real model isn't measured here.
570+
what that costs a real model isn't measured here. OLMo runs on FP4 KV today and
571+
its output is coherent, but coherent is not the same as measured, and no
572+
benchmark has been run against an FP16 baseline.
573+
574+
**Performance is unoptimised.** About 125 ms/token for a 1B model. Nothing on
575+
the OLMo path is fused, batched or tuned, there's a host round trip per layer,
576+
and prefill costs one decode step per prompt token. The roadmap lists what would
577+
change that.
561578

562579
**MI355X and gfx950 only.** See `docs/hardware-support.md`.
563580

0 commit comments

Comments
 (0)