Drop the rotated-out internal ROCm apt repo before apt-get update - #548
Merged
Conversation
The rocm/pytorch base images ship /etc/apt/sources.list.d/rocm.list pinned to a specific internal build on compute-artifactory.amd.com — currently compute-rocm-rel-7.1 20. That build has been rotated out upstream, so the index returns 404, apt-get update exits 100, and because the Dockerfile chains update && install the whole layer fails. With no image, every downstream CI job fails within seconds; the failures on recent PRs are this, not the PR contents. ROCm and torch are already installed in the base image and nothing here installs further rocm-* packages, so the repo is unnecessary. Removing it is also a no-op on images that do not carry the file. Verified against rocm/pytorch:rocm7.1_ubuntu24.04_py3.13_pytorch_release_2.9.1: apt-get update as-is exit 100 rm rocm.list then apt-get update exit 0 rm rocm.list then the full install line exit 0 rocm and torch after removal intact, 2.9.1+rocm7.1.0 Applied to Dockerfile.ccl and apptainer/iris.def as well, which share the same base image family and the same update-and-install chain. Dockerfile.dev builds from ubuntu:24.04 and is unaffected.
github-actions
Bot
added
in-progress
iris/mem/utils.py imported memrealtime and smid under one try/except. Upstream
Triton at the pinned commit (bcbcabdd) exports memrealtime but not smid, so the
import failed as a pair and both helpers fell back to stubs that call
tl.static_assert(False). Any kernel recording a trace event then failed to
compile:
CompileTimeAssertionFailure: smid is unavailable in this Triton build
at device_utils.get_cu_id() in record_event_start
This was masked for months because the CI image is a fixed tag that was never
rebuilt; the first clean rebuild surfaced it.
Each intrinsic is now probed on its own, so a missing smid no longer disables
timestamps, and each fallback emits the instruction the intrinsic would rather
than refusing to compile. get_cu_id reads CU_ID from HW_REG_HW_ID exactly as
get_xcc_id below it already reads HW_REG_XCC_ID.
The CU_ID field is 4 bits, so it identifies the CU within its shader engine
rather than globally -- pair it with get_xcc_id. Verified on gfx950: a
256-workgroup launch reports CU_ID 0-8 with XCC_ID 0-7, the latter matching the
part's 8 XCDs. Architectures without a fallback report 0 instead of failing to
compile, since tracing is diagnostic and losing CU attribution beats breaking
every traced kernel.
tests/unittests/test_device_context.py: 41 passed on gfx950.
The specialized kernels launch num_sms workgroups and split them on pid < GEMM_SMS, giving the remainder to the communication path. Passing gemm_sms equal to the CU count leaves that remainder empty, so every workgroup takes the GEMM branch and the scatter never runs. Validation caught it as C=0.0, while the reported throughput went UP because the communication was simply skipped. The perf job hit this after moving to a 256-CU part, where its hardcoded --gemm_sms 256 matches the CU count exactly. The computed default has the same hole: 2**int(log2(cu_count)) equals cu_count whenever the CU count is a power of two, so it only ever worked on 304-CU parts by accident. Step the default down when it would leave nothing over, and reject an explicit value that does. Measured on 8x MI350X: the CI config now runs at 2904 TFLOPs against a 1440 threshold, and the default at 2530. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Technical Details
Test Plan
Test Result
Submission Checklist