Bring up vLLM on ROCm 10 and fix scheduler stalls found in the 6-leg run - #1605
Merged
Merged
Conversation
The SGLang docker stack moved to ROCm 10 in #1593, but vLLM stayed on vllm/vllm-openai-rocm:v0.29.0, so a single example told the reader to pair a ROCm 10 SGLang image with a ROCm 7.2.3 vLLM one, and the compatibility matrix still claimed every bare-metal patch level matched its container image. Move the docker route to rocm/vllm, the only repository publishing a ROCm 10 vLLM build, and say plainly that the bare-metal wheel stays on rocm723 rather than tracking it.
The ROCm wheel index publishes vLLM for ROCm 7.2 only, so a ROCm 10 host had no installable vLLM at all and setup failed with no usable route. Add a VLLM_INSTALL_METHOD that resolves to a source build from VLLM_REPO at VLLM_SOURCE_REF when the detected ROCm is 10 or newer, and keep the wheel path for 7.2 where it still resolves. The resolved VLLM_ROOT and its build target are persisted so the later run launches the interpreter that was actually built rather than rediscovering one.
The framework agent optimizes by patching vLLM source, but the image serves an installed wheel, so a patch applied to the image checkout changed nothing that ran. Create a git worktree from a synthetic baseline commit over the image checkout and put it ahead of site-packages on the server's PYTHONPATH, so a patch reaches the process being benchmarked and can be reverted cleanly.
vLLM writes its rank traces directly into the benchmark workspace rather than a torch_trace subdirectory, which was not among the probed directories. Discovery fell through to the graph-capture sidecars and roofline then failed with capture_only_fallback while a complete trace sat in the root: two runs of the ROCm 10 leg lost kernel attribution that way.
On one exact ROCm 10 Qwen3-14B-FP8 stack this lever raises throughput while GSM8K collapses to zero, so benchmarking it can only promote a configuration that is faster and wrong. Drop those variants before they run, keyed on the stack fingerprint so every other stack keeps the lever. The aiter component is matched through the git-describe dist version as well as the plain commit: the fingerprint degrades to that form when AITER_COMMIT is unset, and an unmatched pin would silently let the lever back in.
AITER root resolution only understood split aiter_meta wheels, so on a source checkout it returned nothing, AITER_ROOT_DIR went unset, and the GEMM tuner tuned a different AITER than the one serving. Resolve the source layout too, and make the tune/serve alignment preflight fail the run instead of warning: tuning results attributed to the wrong library are worse than no tuning. The fingerprint degraded to "unknown" for rocm wherever ROCm comes from wheels rather than /opt/rocm, which made any pin naming rocm structurally unmatchable on bare metal. Read it from rocm-sdk-core, and record torch as well, since the torch build selects Kineto's GPU backend and the ROCm 10 legs differ in nothing else. The backend itself is read from the trace metadata that names it.
An open Kineto bug on the rocprofiler-sdk backend loses every GPU event, so trace analysis fails identically on each attempt. Only the watermark path was bounded, at three retries; cycle_start, the KERNEL re-profile and CLOSE kept arming fresh work that could not succeed, spending the profile budget of an unattended run on it. Condemn the stack once, from a parsed trace that carries host ops beside zero kernels, and suppress the remaining enqueue paths with the reason recorded. The verdict demands evidence on both sides: an empty or unreadable capture is a transient failure, and treating it as permanent would give up kernel attribution for the rest of the session.
A serving round waited on its Ray actor with no wall-clock bound, so a lease that never acquired resources held the round open indefinitely. Probe the actor and cap that wait. A stalled source-patch specialist was re-dispatched every round even when its source root made the attempt impossible. Prune that family after the first deterministic preflight failure and skip a key already running or failed. A long attempt key produced a report filename past the filesystem limit, so the kernel lane's GEAK and tuning outcomes were never written. Shorten it with a digest and record the unvalidated bucket and per-lane totals the final report reads.
Hyperloom drives both the claude and the codex backend, and the CLI preflight already treats an OpenAI-only environment as a valid single provider, but bare-metal setup rejected it: the OpenAI side counted only as a rider on an Anthropic credential from the same host. A gateway configured for codex alone therefore failed setup with no usable endpoint. Accept the OpenAI pair as a credential in its own right and persist it with CODEX_MODEL. A base URL without a key stays an error, so a half-configured gateway still fails at setup rather than at the first agent call.
KERNEL phase entry could spend minutes inside synchronous reprofile and tuning work while the coordinator tick loop was frozen, so the idle guard never had a chance to advance. Run phase-entry hooks out of band and record hook failures from the background task instead. Conc-sweep boot rounds were treated as measured variants even though their only job is to get the persistent server ready. Run the boot as a server-only phase, then measure the same concurrency through the reuse path; a sweep that ran but produced no comparable pair now terminates as sweep_failed rather than sweep_done. Also stop deterministic source-patch failures from re-entering the scheduler: forced stalled specialists skip any existing idempotency row, and delegated source-patch specialists prune immediately when no framework Git root exists.
Several tests asserted provider selection and credential precedence while inheriting the operator shell's Anthropic/OpenAI credentials, so they answered for the machine that ran them rather than the fixture they constructed. Clear only the variables each test owns before asserting on those branches. Subprocess-based tests also inherited an ambient PYTHONPATH that could resolve a stale installed Hyperloom instead of this checkout. Put this tree's src first for child processes during pytest, matching CI's editable install behaviour.
lishuoshuo-amd
requested review from
a team,
devalshahamd and
haofrank
as code owners
September 22, 2026 09:15
lishuoshuo-amd
force-pushed
the
fix/rocm10-vllm-image-in-skill
branch
2 times, most recently
from
September 22, 2026 09:19
19a222b to
a7eb8ab
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
|
|
||
| def _functions(*names: str) -> str: | ||
| text = _INSTALL_SH.read_text() |
| assert lease._actor is None | ||
|
|
||
| next_actor = _FakeActor((0, "next", "")) | ||
| monkeypatch.setattr(rb, "get_ray_backend", lambda: _StubBackendP2()) |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
|
|
||
|
|
||
| def _functions(*names: str) -> str: | ||
| text = _INSTALL_SH.read_text() |
| assert lease._actor is None | ||
|
|
||
| next_actor = _FakeActor((0, "next", "")) | ||
| monkeypatch.setattr(rb, "get_ray_backend", lambda: _StubBackendP2()) |
The detached-launch snippet puts $(dirname "$PYTHON") at the front of PATH, but PYTHON is never set in that shell: the launch is its own process and kernel-agent.env.sh does not export it. The unset variable collapsed the entry to ".", leaving /usr/local/bin ahead of the virtualenv, so python3 resolved to an interpreter without the optimizer's dependencies and the CLI died on ModuleNotFoundError before a session dir existed.
On a ROCm 10 stack the installer routes vLLM to a source build, which never touches the wheel index that carries aiter, and the only AITER installation lived inside install_sglang_framework. A bare-metal vLLM host therefore ended up with no aiter at all, so kernelforge's fusion validation - which runs vLLM with VLLM_ROCM_USE_AITER=1 - had nothing to enable. Three gaps close together: - AITER installation moves into ensure_aiter_for_python so the vLLM source route installs it into the isolated overlay, on both the fresh-build and the reuse path. - VLLM_ROCM_USE_AITER lands in .env whenever the detected framework is vLLM and aiter actually imports. The SGLang images ship SGLANG_USE_AITER pre-set; vLLM defaults its gate off, so nothing turned it on. It stays unset when aiter is missing, because an enabled gate without a matching module fails at serve time rather than falling back. - The source checkout is a depth-1 fetch of a commit SHA and carries no tags, so setuptools_scm stamped builds 0.1.dev1 instead of the release. The build now declares its version the same way the SGLang source install does.
The per-distribution SETUPTOOLS_SCM_PRETEND_VERSION_FOR_VLLM never applied: the source route builds with setup.py develop, which leaves the distribution name unknown, and vcs-versioning only consults the per-distribution form once it knows that name. A bare-metal vLLM leg therefore still reported 0.1.dev1 on a build that had been told its version. Set the generic variable alongside it; this invocation builds vllm alone, so nothing else can pick it up. Extracting the AITER install into ensure_aiter_for_python also broke the bare-metal SGLang test, which sources install_sglang_framework on its own and so lost the helper that decides whether to install. It now sources both.
Magpie prepends its own profiler flags before EXTRA_VLLM_ARGS, so the torch_profiler_dir this layer injects is the one vLLM keeps, not the one the launcher computes. The injected value is the run's output dir, while discovery probed only the Magpie workspace and its children, so a complete rank trace sat one level above every probed path and roofline failed three attempts with no_trace_files before giving up. Pin Magpie's own trace dir to the same directory through the environment variable its launcher already reads, so both sides agree whichever flag wins, and probe the output dir as well. The comment claiming the injected path does not matter is what allowed the two to drift, so it goes too.
The model-level ceiling and the bypass roofline both resolve achievable peaks first and vendor theoretical peaks when a GPU has no measured entry. The TraceLens arch spec was left out of that rule, so on MI355X it returned nothing at all: with no bundled arch data and a microbenchmark that cannot run while the serving process holds the only visible GPU, the whole trace_analyze step failed and every leg continued without a roofline. Resolve it the same way the other two do. Measured entries still win, so nothing changes for a GPU that has one.
Profile executors record the workspace they profiled into, not the trace file, so opening that path raised IsADirectoryError, which the OSError guard swallowed into an empty result. The backend was therefore never recorded on any run, and the condemn path that cites it in its reason had nothing to cite. Resolve a directory to the trace inside it. This is the field that distinguishes a roctracer build from a rocprofiler-sdk one, which is exactly what a ROCm trace carrying no GPU kernels has to be judged against.
Moving every phase-entry hook onto a background task freed the tick loop, but it also detached the effects callers read straight after the transition returns: a caller advancing into CLOSE sees the sequencer settle as part of that call, and a hook that had not run yet left it looking at an unsettled phase. Keep the background task for KERNEL, whose reprofile and tuning work is the minutes-long case the idle guard needed protecting from, and await the rest. A hook that raises is still logged and recorded rather than failing the transition, since that hook also closes the phase it left. The affected tests stubbed _on_phase_entered with the two-argument signature it had before reason and evidence were added; awaiting the hook surfaced that as a TypeError the background task used to swallow.
Both lease.close() calls sat inside assert expressions, so running the suite under -O would drop the close along with the check and leave the second call asserting idempotency over a teardown that never happened.
The matrix still listed vllm/vllm-openai-rocm:v0.29.0 as the validated image and described bare-metal vLLM as the rocm723 wheel only, while the installer now builds vLLM from source on ROCm 10 and the docs elsewhere point at the rocm/vllm image. Record the source route, its prerequisites and the two new pins, and guard VLLM_SOURCE_REF the same way SGLANG_REF is guarded. Co-authored-by: Cursor <cursoragent@cursor.com>
lishuoshuo-amd
changed the title
Running the KERNEL entry hook as a background task freed the tick loop, but the phase machine kept evaluating KERNEL exits while GEAK was still running inside that hook. GEAK is not a queued KERNEL-lane task and stamps no inline heartbeat, so the idle guard fired about ten minutes in: on the 10h ROCm 10 run every GEAK leg left KERNEL 40-75 minutes into a 2-3 hour GEAK run, the conc_sweep then booted servers on the same GPUs, later KERNEL entries started further GEAK runs alongside the first, and GEAK was still writing after the session had closed. Keep the hook out of band, but hold every transition out of KERNEL and the idle streak while it runs, and let run() wait for it before teardown. Co-authored-by: Cursor <cursoragent@cursor.com>
xiaofei-zheng
approved these changes
Sep 23, 2026
xiaofei-zheng
left a comment
xiaofei-zheng
left a comment
Collaborator
There was a problem hiding this comment.
No blocking issues — LGTM.
Paths I checked:
- Every
_enqueue_internal_analysis_taskcall site handles the newNonereturn (close.py:98, kernel.py:304, kernel.py:4253, macro_cycle.py:318, prelude.py:2644). sweep_failedis a registered stop reason with a terminal CLOSE handler, and the new branch cannot fire on a sweep that produced pairs:was_skipped+budget_exhaustedonly co-occur whensuccessful_pairs == 0.- Out-of-band KERNEL entry hook: the phase machine holds transitions out of KERNEL and the idle streak while it is in flight, and
Coordinator.runawaits it on teardown; a failed or cancelled hook is logged through_record_phase_entry_task_result. GpuSpecialistLease.closeforcing the kill after an unconfirmed stop is safe:COOPERATIVE_REAP_BUDGET_SECis 8.5s (0.5 + 5.0 + 1.0 + 2.0) againstCLOSE_STOP_TIMEOUT_SEC10.0, so a cooperative reap fits inside the bounded stop. Same shape asServingLease.close.RESOURCE_ACQUIRE_TIMEOUT_SECbounds only the actor-scheduling phase; the round's ownray.get(ref)still carries no timeout, so the invariant the deletedtest_serving_lease_coordinator_no_timeoutprotected is preserved.gpu_trace_unsupported_reasonis an additive defaulted field, so noSCHEMA_VERSIONbump is needed; it is also added to the LLM-forbidden allowlist inpolicy/gate.py.- The source-patch preflight cannot prune a specialist the runner would have accepted: the router's root set is a superset of
SpecialistSubprocessConfig.framework_source_roots, and both resolvepreferredfrom the same session framework. - conc_sweep boot-only rung: excluded from measured points and re-measured as the first reuse point via
grid[boot_idx:];kernel_opt_outcomehas no programmatic in-tree consumer beyond the JSON andreport.py:1160, which only re-reads ints. - Image/pin move is consistent across
docs/compatibility.rst,docs/install/install.md, the four exampleSKILL.mdfiles,install.sh(VLLM_IMAGE_SOURCE_COMMIT) andinstall_baremetal.sh(VLLM_SOURCE_REF). - Title and description match the current diff, including the declared behaviour changes (schema v2 +
lane_totals,sweep_failed, non-zero exit on a hard aiter preflight mismatch).
Tests: no local suite run (CI is green at e29247d). I only ran a standalone repro against Recorder._stable_item_filename to check the long-fragment-name path; it behaves as intended for names over the filesystem limit.
6 tasks
lishuoshuo-amd
added a commit
that referenced
this pull request
Sep 23, 2026
KERNEL entry runs out of band since #1605 and the phase is held until the hook returns. In this walkthrough the hook launched geak_runner.py as a real subprocess, and KERNEL only reached a terminal geak_result once that process failed, so whether eleven ticks reached CLOSE depended on how fast the runner exited on the CI host. The runner path is made unresolvable, which takes the existing runner_not_found branch: GEAK records a terminal result in-process and the phase winds down to SWEEP without starting anything. The test now also pins that branch, so a routing change cannot quietly bring the subprocess back. Co-authored-by: Cursor <cursoragent@cursor.com>
zoroyihan7
added a commit
that referenced
this pull request
Sep 23, 2026
#1605 landed a bound on the same wait this branch bounds, and the two guard different windows rather than the same one. Its deadline covers the window before Ray places the actor, and kills the actor when it expires. That is safe there and only there: an actor that never acquired resources has spawned no server, so nothing survives the kill still mapping GPUs. Once it has been placed, #1605 returns to an unbounded wait, which is the window this branch's ceiling covers -- and where killing the actor is exactly what must not happen, because the round may still be serving. Keep both, in one loop: the acquire probe and its kill, then the ceiling and its quarantine. Dropped from #1605's version is the bare ``ray.get(ref)`` taken for an uncancellable round once placed; it carries no deadline, so it reopens the parked-thread failure for precisely the rounds no cancel will ever reach. Those rounds keep polling instead, which costs one wakeup per 0.25s and puts them under the ceiling. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description: Fix ROCm 10 validation gaps and the orchestrator stalls that made the 6-leg run hard to trust. The branch adds ROCm 10 vLLM source-install routing, makes the Docker vLLM image patchable, fixes trace discovery and unsafe SGLang lever filtering, records enough stack/profiler identity to explain ROCm10 profiler failures, stops repeated no-op analysis/profile retries, and fixes several Hyperloom scheduler/reporting issues found during the run.
Linked issue(s): none
Tests: added/updated? commands run? (what to test; if a test was replaced, where its assertions live now):
python3 -m pytest src/hyperloom/inference_optimizer/tests/test_setup_cli.py -q -p no:randomly— 85 passedpython3 -m pytest src/hyperloom/inference_optimizer/tests/test_install*.py src/hyperloom/inference_optimizer/tests/test_anthropic_credential_registry.py src/hyperloom/inference_optimizer/tests/test_llm_config.py src/hyperloom/inference_optimizer/tests/test_baremetal_doc_version_consistency.py -q -n auto— 290 passedpython3 -m pytest src/hyperloom/inference_optimizer/tests/test_roofline_executor.py src/hyperloom/inference_optimizer/tests/test_common_provenance.py src/hyperloom/inference_optimizer/tests/test_prelude_roofline.py src/hyperloom/inference_optimizer/tests/test_kernel_attempt_summary.py -q -p no:randomly— 189 passed, 1 skippedpython3 -m pytest src/hyperloom/inference_optimizer/tests/test_explore_executor.py -q -p no:randomly— 86 passedpython3 -m pytest src/hyperloom/inference_optimizer/tests/test_kernel_idle_streak.py src/hyperloom/inference_optimizer/tests/test_close_phase_sequencer.py src/hyperloom/inference_optimizer/tests/test_conc_sweep.py src/hyperloom/inference_optimizer/tests/test_conc_sweep_session_deadline.py src/hyperloom/inference_optimizer/tests/test_stop_reason_outcomes.py src/hyperloom/inference_optimizer/tests/test_ray_backend_unit.py::test_specialist_close_forces_actor_kill_after_stop_attempt -q -p no:randomly— 224 passed, 2 skippedpython3 -m pytest src/hyperloom/inference_optimizer/tests/test_pre_release_stall_liveness.py src/hyperloom/inference_optimizer/tests/test_delegate_denial_loop.py src/hyperloom/inference_optimizer/tests/test_specialist_lifecycle.py ... -q -p no:randomly— 292 passed, 2 skippedpython3 -m pytest src/hyperloom -q -n autostill has pre-existing failures on this workstation; the failure set was reproduced byte-identically with the branch changes stashed. One collection error was missing localhypothesis, which is already declared in the test extra.Size/complexity triggers crossed: yes. This is intentionally a consolidation branch for the ROCm10 bring-up and the scheduler bugs discovered while validating it. It is split into small commits by concern so it can still be reviewed by subsystem: vLLM source install, vLLM image source tree, trace discovery, SGLang compatibility filter, AITER/provenance, trace capability gating, scheduler/reporting fixes, codex credential setup, and follow-up scheduler/log fixes.
If this simplifies or refactors: which mechanism was removed, which contract was preserved, and the tests that show it:
Coordinator.runwaits for it before teardown; conc_sweep boot is no longer counted as a measured point.Observable effect: operators get a ROCm10 bare-metal vLLM install path instead of a dead wheel route, Docker vLLM patches affect the served source tree, vLLM roofline picks the real rank trace instead of capture sidecars, known-bad SGLang unified-attention variants are skipped before benchmark, zero-GPU traces suppress repeat roofline scheduling with a recorded reason, and final reports keep GEAK/GEMM outcomes even with long attempt keys. Also:
vllm/vllm-openai-rocm:v0.29.0(vLLM 0.29.0, ROCm 7.2.3) torocm/vllm:rocm10.0.0_ubuntu24.04_py3.14_pytorch_2.12.0_vllm_0.27.0(vLLM 0.27.0, ROCm 10.0);docs/compatibility.rstnow documents both vLLM routes and theVLLM_SOURCE_REF/VLLM_IMAGE_SOURCE_COMMITpins.VLLM_ROCM_USE_AITER=1to.envwhenaiterimports, and on the ROCm 10 source route also persistsVLLM_ROOT,FRAMEWORK_REPO_PATHandVLLM_TARGET_DEVICE.reports/kernel_optimization_summary.jsonmoves toschema_version2 with a newlane_totalsblock (source_level,geak,gemm_tuning).kernel_opt_outcomenow rolls up all lanes and can beunvalidated;totals.attemptedcounts attempts across all lanes, whiletotals.integrated/totals.rejectedstay source-level only.kernelforge.gemm_tuneruncommand now exits non-zero when the AITER tune/serve alignment preflight reports a hard mismatch (previously a warning).Breaking changes: no intended public API break. Bare-metal setup now accepts codex-only OpenAI credentials as a valid runtime backend; conc_sweep now treats a run with zero successful pairs as
sweep_failedinstead of a cleansweep_done. Bare-metal vLLM setup now refuses any ROCm stack other than ROCm 7.2.x (wheel) or ROCm 10 withtorch.version.hip7.15 (source build), andVLLM_INSTALL_METHOD=wheel|sourcemust match the detected route; the wheel route now fails when the ROCm torch version cannot be resolved fromVLLM_ROCM_INDEX(previously a warning that fell back to vLLM's own torch pin).PR addresses single concern: no. It addresses the ROCm10 validation campaign and the Hyperloom runtime blockers found by that campaign. The commit stack is separated by subsystem to make review tractable.
Root cause is upstream (Magpie/TraceLens/GEAK/IntelliKit/AgentKernelArena), ticket filed: ROCm10 SGLang CPU-only traces are rooted in upstream PyTorch/Kineto rocprofiler-sdk flush behaviour (see [ROCm] Fix teardown memory corruption and truncated traces on the rocprofiler-sdk backend pytorch/kineto#1564). This PR does not patch Kineto; it records/suppresses unsupported GPU traces and leaves image selection/custom torch build as the runtime workaround.