Skip to content

Add tiered ROCm build-correctness and GPU inference qualification, and pin each release to one upstream commit - #29

Draft
sreeram-11 wants to merge 3 commits into
lemonadefrom
sreeram/add-rocm-build-and-inference-qualification
Draft

sreeram-11 wants to merge 3 commits into
lemonadefrom
sreeram/add-rocm-build-and-inference-qualification

Conversation

@sreeram-11

Copy link
Copy Markdown
Collaborator

What this does

Adds build-correctness and GPU inference checks for the ROCm artifacts, and pins each release to a single upstream commit.

Today the only verification of a ROCm build is one step asking whether a file matching ggml-hip*.dll exists, on Windows only. That passes for a DLL containing zero GPU code, and there is no equivalent check on Linux at all.

This matters because a broken ROCm build does not look broken.

  • llama.cpp loads GPU support as a plug-in and treats a plug-in that fails to load as a non-event: it logs a line, skips it, and runs on the CPU.
  • The server still starts, still answers, still returns correct text — just ten times slower, with no error.

Changes

  1. Adds .github/qualify-rocm/: 11 static checks on the published archive, and 5 GPU inference checks on the gfx1151 runners.
  2. Verifies the GPU backend really contains code for every gfx target the build asked for (27 on Linux, 20 on Windows). This was previously unverified.
  3. Runs llama-server on a real GPU and asserts the GPU actually did the work, rather than just that the server replied.
  4. Splits ROCm into one family per version, so a 7.14 failure no longer costs us the 10.0 and 10.1 artifacts.
  5. Checks all three ROCm versions are present. The old glob was satisfied by any one of them, so a release missing an entire version would have passed.
  6. Pins one upstream commit per run. Each job used to clone upstream separately, so they could compile different source and then get labelled with one tag.
  7. Resets the library search path on the Linux ROCm build (patchelf), which the CUDA and OpenVINO builds already did.
  8. Attaches checksums.txt and a per-archive qualification record to each release.

Gating

  • A qualification failure drops only the affected ROCm version.
  • Every other ROCm version, and CUDA / OpenVINO / CPU, publish as normal.
  • The release job then exits non-zero so the gap is noticed.

New files

File What it does
tier0_static.py The 11 static checks, run against the archive we are about to publish
tier1_gpu.py Starts llama-server on a real GPU and exercises it over HTTP
binfmt.py Reads compiled binaries: gfx targets, library dependencies, C library version
install_rocm.py Installs a user-style ROCm runtime into a throwaway venv
check_post_merge.py Two checks that can only run after the release job merges the Windows CPU backend
aggregate.py Combines the reports, decides which ROCm versions may ship, writes the job summary
report.py Shared JSON report format
README.md Why each check exists and how the less obvious ones work

Per-check detail is in .github/qualify-rocm/README.md.

Important points to know

These came out of running the checks against a real published release:

Finding Status
The ROCm completeness check was a glob, so one archive out of six satisfied it fixed here
libggml-hip.so shipped with the CI runner's absolute path in its library search path fixed here
Jobs cloned upstream separately, so one release could contain artifacts built from different commits fixed here
The Linux archive needs glibc 2.38, so it does not run on Ubuntu 22.04 or Debian 12 expected — Ubuntu 24.04 is the supported floor; now checked
The Windows ROCm build is the only job that does not pass the shared CMAKE_ARGS, so its zip ships 27 upstream test-*.exe binaries not changed — reported as a warning

Cost

Tier 0 runs 6 legs on standard runners, including on PRs.
Tier 1 runs 6 legs on the two gfx1151 machines.

@sreeram-11
sreeram-11 requested a review from superm1 October 13, 2026 09:04
@sreeram-11
sreeram-11 marked this pull request as draft October 13, 2026 10:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant