mainarch: a Rust inference stack that talks straight to the AMD kernel driver

Running a large language model on an AMD GPU normally means going through ROCm:
the HIP runtime, the HSA runtime beneath it, the thunk library beneath that, and
only then the kernel driver that actually talks to the card. mainarch skips them.
It opens /dev/kfd, builds the dispatch packet by hand, rings the doorbell, and
waits on the signal the command processor decrements.

From there it goes up. GPU virtual memory, AQL queues, a code-object loader
written from scratch, all-reduce over XGMI, FlashDecoding attention with paged
FP4 KV, a decoder layer, and an autoregressive token loop, in one typed Rust
workspace with nothing borrowed from a vendor runtime.

At the top of it a real model runs. OLMo 2 from the Allen Institute for AI
serves over /v1/chat/completions on an MI355X, at full depth and full vocabulary
from a real checkpoint. OLMo 2 rather than a more familiar model because it is
open source rather than open weights: AI2 publishes the corpus, the training
code and every intermediate checkpoint, so nothing between the training data and
a token leaving a hand-built AQL packet is a black box.

  crates/mainarch-sys           hand-encoded amdkfd ioctls
  crates/mainarch-core          device, memory, queues, code objects, attention,
                                decoder layer, model loop, weights, OLMo 2
  crates/mainarch-collectives   collectives and the rccl-tests-shaped harness
  crates/mainarch-cli           the mainarch binary
  kernels/                      187 GPU kernels and the gfx950 cross-compile
  demo/sandbox/                 a GPU-free demo server behind an OpenAI-shaped seam
  examples/model-api-plugin/    a standalone external model package

`just demo` builds the workspace, proves what the machine can do, and serves the
demo page, taking the live raw-KFD/AQL lane when an AMD GPU is reachable and a
CPU-only lane when one is not. `just olmo` runs the real model.

Correctness is gated rather than asserted, and the gates are written to fail when
a claim is false rather than when arithmetic is merely imprecise. The multi-head
attention kernel is checked bitwise against the grouped-query kernel it derives
from, and then against a perturbation that a kernel ignoring head indices would
survive. The QK-norm kernel is checked against an f64 reference, and then against
a perturbation only a projection-wide RMS can propagate. The post-norm kernel is
checked against both orderings so pre-norm cannot pass.

This is a reference architecture and a working prototype, not a production
serving stack. It serves one request at a time, there is no batching, no
scheduling across sequences, no auth and no quota, and nothing on the OLMo path
is fused or tuned. The README's scope section says where every edge is.

Apache License 2.0.

Author: maincode-dave

.devcontainer/Dockerfile

@@ -0,0 +1,50 @@
+# mainarch dev image. A clean Rust toolchain that talks to AMD silicon at the
+# KERNEL ABI. Deliberately NO ROCm / HIP / HSA runtime: this project replaces
+# that stack rather than building on it. The only AMD-specific things needed are
+#   * the kernel UAPI headers (linux/kfd_ioctl.h) shipped by linux-libc-dev
+#   * libdrm's amdgpu_drm.h for the DRM render-node ioctls
+#   * LLVM/clang, whose upstream amdgpu backend can emit gfx950 ISA on its own
+# The GPU, when present, is reached purely via /dev/kfd + /dev/dri passthrough
+# (see .devcontainer/gpu/devcontainer.json). The image itself builds and runs
+# fine on a machine with no AMD GPU at all, and every CPU-only path still works.
+ARG BASE=docker.io/library/ubuntu:24.04
+FROM ${BASE}
+
+ENV DEBIAN_FRONTEND=noninteractive \
+    LANG=C.UTF-8
+
+# --- system: build toolchain, kernel/driver ABI headers, debug + dev ergonomics
+RUN apt-get update && apt-get install -y --no-install-recommends \
+      build-essential clang lld llvm libclang-dev \
+      cmake ninja-build pkg-config \
+      linux-libc-dev libdrm-dev libelf-dev libnuma-dev zlib1g-dev \
+      mold \
+      git curl ca-certificates gnupg openssh-client \
+      gdb strace ltrace \
+      vim tmux htop jq ripgrep bsdmainutils file less \
+      python3 \
+ && rm -rf /var/lib/apt/lists/*
+
+# --- Rust toolchain (system-wide so root and any user share it)
+ENV RUSTUP_HOME=/usr/local/rustup \
+    CARGO_HOME=/usr/local/cargo \
+    PATH=/usr/local/cargo/bin:$PATH
+# NB: rustup-init takes one component per -c flag. clippy + rustfmt already come
+# from `--profile default`; only rust-analyzer + rust-src need adding.
+RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \
+      | sh -s -- -y --default-toolchain stable --profile default \
+        -c rust-analyzer -c rust-src \
+ && rustc --version && cargo --version
+
+# --- iteration harness: `just` task runner, nextest, watch mode
+# Installed from crates.io rather than a prebuilt release, so this works on any
+# builder without needing BuildKit cache mounts or a static-musl downloader.
+RUN cargo install --locked just cargo-nextest cargo-watch \
+ && just --version && cargo-nextest --version && cargo-watch --version
+
+# mold keeps incremental links snappy on a workspace this size
+ENV RUSTFLAGS="-C link-arg=-fuse-ld=mold"
+
+WORKDIR /workspaces
+LABEL org.opencontainers.image.title="mainarch-dev" \
+      org.opencontainers.image.description="Rust + raw AMD kernel ABI dev harness (no ROCm)"

.devcontainer/devcontainer.json

@@ -0,0 +1,27 @@
+{
+  "name": "mainarch (CPU)",
+  "build": {
+    "dockerfile": "Dockerfile"
+  },
+  "mounts": [
+    "source=mainarch-cargo-registry,target=/usr/local/cargo/registry,type=volume"
+  ],
+  "customizations": {
+    "vscode": {
+      "settings": {
+        "editor.formatOnSave": true,
+        "rust-analyzer.check.command": "clippy",
+        "[rust]": { "editor.defaultFormatter": "rust-lang.rust-analyzer" },
+        "files.watcherExclude": { "**/target/**": true }
+      },
+      "extensions": [
+        "rust-lang.rust-analyzer",
+        "vadimcn.vscode-lldb",
+        "tamasfe.even-better-toml",
+        "skellock.just"
+      ]
+    }
+  },
+  "remoteUser": "root",
+  "postCreateCommand": "bash .devcontainer/post-create.sh"
+}

.devcontainer/gpu/devcontainer.json

@@ -0,0 +1,36 @@
+{
+  "name": "mainarch (AMD GPU passthrough)",
+  "build": {
+    "dockerfile": "../Dockerfile",
+    "context": ".."
+  },
+  "runArgs": [
+    "--device=/dev/kfd",
+    "--device=/dev/dri",
+    "--security-opt=seccomp=unconfined",
+    "--group-add=keep-groups",
+    "--ipc=host",
+    "--cap-add=SYS_PTRACE"
+  ],
+  "mounts": [
+    "source=mainarch-cargo-registry,target=/usr/local/cargo/registry,type=volume"
+  ],
+  "customizations": {
+    "vscode": {
+      "settings": {
+        "editor.formatOnSave": true,
+        "rust-analyzer.check.command": "clippy",
+        "[rust]": { "editor.defaultFormatter": "rust-lang.rust-analyzer" },
+        "files.watcherExclude": { "**/target/**": true }
+      },
+      "extensions": [
+        "rust-lang.rust-analyzer",
+        "vadimcn.vscode-lldb",
+        "tamasfe.even-better-toml",
+        "skellock.just"
+      ]
+    }
+  },
+  "remoteUser": "root",
+  "postCreateCommand": "bash .devcontainer/post-create.sh"
+}

.devcontainer/post-create.sh

@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+# Runs once when the devcontainer is created. Nothing here is required for the
+# build; it just tells you which lane you are in.
+set -u
+
+echo
+rustc --version
+just --version 2>/dev/null || true
+echo
+
+if [ -e /dev/kfd ] && [ -d /sys/class/kfd/kfd/topology/nodes ]; then
+  echo "AMD kernel driver visible: /dev/kfd is present."
+  ls -l /dev/kfd /dev/dri/renderD* 2>/dev/null | head -4
+  echo "GPU lane available. 'just demo' will run the live KFD/AQL path."
+else
+  echo "No /dev/kfd in this container: running the CPU-only lane."
+  echo "Everything in 'just tour' still works; the live GPU proofs are skipped."
+  echo "On an AMD GPU host, reopen using the '.devcontainer/gpu' configuration."
+fi
+
+echo
+echo "mainarch dev harness ready (no ROCm anywhere in this image)."
+echo "Next:  just demo"
+echo

.gitattributes

@@ -0,0 +1,25 @@
+* text=auto
+
+*.rs text eol=lf
+*.toml text eol=lf
+*.md text eol=lf
+*.yml text eol=lf
+*.yaml text eol=lf
+*.json text eol=lf
+*.sh text eol=lf
+*.py text eol=lf
+*.html text eol=lf
+*.css text eol=lf
+*.js text eol=lf
+*.cl text eol=lf
+*.txt text eol=lf
+justfile text eol=lf
+Dockerfile* text eol=lf
+
+# Byte-exact fixtures: the model API gate compares these verbatim, so they must
+# not pick up CRLF on a Windows checkout.
+*.receipt text eol=lf
+
+# The compiled gfx950 HSA code object is loaded by our own ELF reader and its
+# sha256 is asserted; never let it through a text filter.
+*.co binary

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,55 @@
+# Bug Report
+
+## Summary
+
+Describe the observed bug and the expected behavior.
+
+## Validation Tier
+
+- [ ] CPU-only
+- [ ] Single GPU
+- [ ] Multi-GPU / XGMI
+- [ ] Baseline comparison
+- [ ] Documentation
+
+## Environment
+
+- commit:
+- host kernel:
+- GPU model / gfx target:
+- container or host runtime:
+- relevant device access (`/dev/kfd`, `/dev/dri`):
+
+## Reproduction
+
+Command:
+
+```bash
+
+```
+
+Environment variables:
+
+```text
+
+```
+
+Observed output:
+
+```text
+
+```
+
+Expected output:
+
+```text
+
+```
+
+## Scope
+
+- Does this involve GPU execution?
+- Does this involve a performance claim?
+- Does this involve model API metadata/preflight only?
+- Does this involve secrets, private hardware logs, hostnames, or cluster
+  identifiers that should be redacted?

.github/ISSUE_TEMPLATE/model_api_gap.md

@@ -0,0 +1,51 @@
+# Model API Gap
+
+## Model Or Architecture
+
+Name the model family, architecture feature, or primitive pattern that is not
+represented cleanly by the current `mainarch-core::model_api` surface.
+
+## Exact Gap
+
+- missing primitive
+- missing tensor/cache shape contract
+- missing lowering route metadata
+- missing checkpoint binding metadata
+- missing runtime preflight/reporting surface
+- unclear public API ergonomics
+
+## Contract Tier
+
+Select the narrowest tier this gap affects:
+
+- [ ] public model authoring contract
+- [ ] static runtime metadata contract
+- [ ] experimental execution boundary
+
+## Minimal Graph Shape
+
+List the smallest tensor shapes, cache layout, stage boundary, or op sequence
+that demonstrates the gap.
+
+```text
+
+```
+
+## Current Workaround
+
+Describe whether the gap is currently represented as an explicit `Gap`, a custom
+runner path, a fused native route, or not represented at all.
+
+## Evidence
+
+Include the command, test, example, or docs section that demonstrates the gap.
+If the gap crosses into live graph execution, include the hardware correctness
+oracle and name the submission blockers that would need to be removed. Otherwise
+state that the gap is CPU-only metadata, preflight, or documentation scope.
+Do not claim GPU execution or performance unless the matching hardware evidence
+is included.
+
+## Negative Scope
+
+State what the issue does not prove or request: for example, no GPU allocation,
+no queue submission, no kernel execution, no throughput claim, or no API change.

.github/pull_request_template.md

@@ -0,0 +1,43 @@
+# Pull Request
+
+## Exact claim
+
+State the narrow claim this PR makes. Keep it at the exact scale and behavior
+the evidence validates.
+
+## Change type
+
+- [ ] Runtime / kernel ABI / GPU execution
+- [ ] Kernels
+- [ ] Model API metadata or preflight
+- [ ] Checkpoint / weight loading
+- [ ] Collectives / performance
+- [ ] Documentation
+- [ ] Tooling or tests
+
+## Validation
+
+- [ ] `just ci` passes
+- [ ] Focused model API checks, if `model_api` changed
+      (`python3 tools/check_model_api_public_examples.py`)
+- [ ] Hardware gate, if the claim involves GPU execution
+- [ ] Apples-to-apples baseline, if the claim involves performance.
+      Say which configuration the baseline ran in
+
+## Evidence
+
+Paste the commands and their output. For hardware claims include the GPU model,
+gfx target, and driver version.
+
+```
+
+```
+
+## Negative scope
+
+What this change does *not* prove.
+
+## Notes for reviewers
+
+Assumptions, unresolved blockers, unsupported hardware, or intentionally
+deferred follow-up work.