Skip to content

feat: update Azl 3.0 HPC component versioning#517

Open
henryli001 wants to merge 1 commit into
masterfrom
lihl/update-azl-hpc-version
Open

feat: update Azl 3.0 HPC component versioning#517
henryli001 wants to merge 1 commit into
masterfrom
lihl/update-azl-hpc-version

Conversation

@henryli001

@henryli001 henryli001 commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

This PR updates the versioning of certain components in the Azure Linux 3.0 HPC image in terms of the following components:

  • hpcx (v2.24.1 -> v2.25.1)
  • hpcx_amd (v2.18 -> v2.23)
  • mvapich (v2.3.7-2 -> v4.1)
  • nccl (v2.28.3-1 -> v2.28.9-1)
  • DCGM (v4.4.2 -> v4.5.2)
  • nvshmem (v3.5.19 -> latest)

There are certain behavior changes when it comes to package installation in this change:

  • Bump nvbandwidth in general from v0.8 to v0.9. The latest v0.9 contains an upstream fix that was made to resolve compilation failure on Azure Linux so that no extra patching is needed
  • The current nvshmem RPMs used by Azure Linux HPC image are the RHEL packages built by NVIDIA. Thus, switch to directly consuming the latest RHEL RPMs from upstream NVIDIA RHEL cuda repo instead of the current one whose version is outdated.
  • Modify cuda-toolkit package installation to consume the latest version from upstream NVIDIA Azl3.0 cuda repo instead of the current one whose version is pinned

@henryli001 henryli001 force-pushed the lihl/update-azl-hpc-version branch from 4ca8b85 to 113dc20 Compare April 17, 2026 20:33
@henryli001 henryli001 requested a review from ljqqqqq April 17, 2026 20:33
Comment thread versions.json
"version": "2.18",
"sha256": "45276ff7bd676cc668d1cc6a1fe926d5e157646aaf06201415e0aadb048be16d",
"url": "https://content.mellanox.com/hpc/hpc-x/v2.18/hpcx-v2.18-gcc-mlnx_ofed-redhat8-cuda12-x86_64.tbz"
"version": "2.23",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was held back due to perf regression, do not bump yet

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we planning to roll back the version on other distros as well, say Ubuntu 24.04?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We observed azl3 specific regression, hence the holdback (and the decision to bifurcate hpcx_amd from hpcx in the first place)

Comment thread versions.json
"url": "http://mvapich.cse.ohio-state.edu/download/mvapich/mv2/mvapich2-2.3.7-2.tar.gz"
"version": "4.1",
"sha256": "25a53d3725b669e2c648158fb7c9fc5b1388953f3a2f949748586c447d0e43ee",
"url": "https://mvapich.cse.ohio-state.edu/download/mvapich/mv2/mvapich-4.1.tar.gz"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mvapich 4 has perf regressions, do not bump if 2.3.7 continues to build

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same question as the one for the other comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We observed azl3-specific regression

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants