feat: Add AMD accelerator detection and tracking in libfabric topology

[mpashkovskii]

What?

1. AMD GPU vendor ID detection (0x1002)
2. MI300X device ID range support (0x74a0-0x74af)
3. PCI class fallback detection for unknown AMD GPU models
4. num_amd_accel counter variable
5. isAmdAccel() helper function
6. getNumAmdAccel() public accessor
7. HMEM interface mapping returns FI_HMEM_ROCR for AMD GPUs
8. Topology-aware grouping includes AMD accelerators
9. Logging updated to report AMD GPUs separately

Jira:
AIRIXL-2
AIRIXL-10

[Copilot]

Pull request overview

Adds AMD GPU (ROCm) awareness to the libfabric topology discovery layer so AMD accelerators can be detected, counted, included in topology grouping, and mapped to an HMEM interface.

Changes:

• Introduces AMD accelerator detection (isAmdAccel) and tracking (num_amd_accel, getNumAmdAccel).
• Updates topology discovery/grouping to include AMD accelerators and updates logging to report NVIDIA/AMD/Neuron counts separately.
• Extends HMEM iface selection to return FI_HMEM_ROCR for AMD accelerators.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
src/utils/libfabric/libfabric_topology.h	Adds AMD accelerator counter/accessor; updates getMrAttrIface() to return FI_HMEM_ROCR for AMD.
src/utils/libfabric/libfabric_topology.cpp	Implements isAmdAccel(), updates accelerator discovery counts/logging, and includes AMD in topology-aware grouping.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

getMrAttrIface() can now return FI_HMEM_ROCR, but nixlLibfabricRail::registerMemory() currently only initializes mr_attr.device.* for FI_HMEM_CUDA and FI_HMEM_NEURON. With FI_HMEM_ROCR the device union will remain at its default value, which may violate libfabric’s expectations for ROCR registrations. Please add explicit handling for FI_HMEM_ROCR (including initializing the appropriate mr_attr.device.rocr field) where memory registration attributes are built.

Suggested change

	return FI_HMEM_ROCR;
	// NOTE: FI_HMEM_ROCR is not yet fully supported in the memory
	// registration path (mr_attr.device.rocr initialization). Until
	// that wiring is complete, fall back to system memory.
	return FI_HMEM_SYSTEM;

[Copilot]

getMrAttrIface() selects the HMEM interface by comparing the provided device_id against (num_nvidia_accel, num_amd_accel). In the current call chain, device_id is passed through from the backend as the runtime’s device ordinal (e.g., CUDA device index), not a global index across vendors; on systems with mixed accelerator vendors this can misclassify an AMD/Neuron device_id=0 as CUDA and register memory with the wrong iface. Consider deriving the iface from the device’s PCI vendor (e.g., via device_pci_bus_id + hwloc lookup) or from a single selected runtime, rather than from device_id ranges.

Suggested change

	if (device_id < num_nvidia_accel) {
	return FI_HMEM_CUDA;
	} else if (device_id < num_nvidia_accel + num_amd_accel) {
	return FI_HMEM_ROCR;
	} else {
	return FI_HMEM_NEURON;
	}
	// The device_id passed here is a runtime-specific ordinal (e.g., CUDA
	// device index) and is not guaranteed to be a global cross-vendor index.
	// To avoid misclassifying devices on mixed-vendor systems, only infer
	// the iface from vendor counts when exactly one accelerator vendor is
	// present. Otherwise, fall back to a generic system HMEM iface.
	(void)device_id; // currently unused; kept for API compatibility
	int vendor_count = 0;
	if (num_nvidia_accel > 0) {
	vendor_count++;
	}
	if (num_amd_accel > 0) {
	vendor_count++;
	}
	if (num_aws_accel > 0) {
	vendor_count++;
	}
	// Single-vendor configurations: preserve legacy behavior by selecting
	// the corresponding HMEM iface for all device_ids.
	if (vendor_count == 1) {
	if (num_nvidia_accel > 0) {
	return FI_HMEM_CUDA;
	}
	if (num_amd_accel > 0) {
	return FI_HMEM_ROCR;
	}
	// num_aws_accel > 0 and no other vendors
	return FI_HMEM_NEURON;
	}
	// Mixed-vendor or no-accelerator configuration: we cannot safely infer
	// a vendor-specific iface from device_id. Use a generic system iface
	// to avoid registering memory with the wrong accelerator runtime.
	return FI_HMEM_SYSTEM;

[Copilot]

The PR description calls out MI300X device ID range support (0x74a0–0x74af), but isAmdAccel() currently uses a small whitelist and otherwise relies on the PCI class fallback. To align behavior with the stated range support (and avoid depending on class_id), consider adding an explicit device_id range check for 0x74a0–0x74af (and any other intended ranges) before the class fallback.

[gaoikawa]

@mpashkovskii - Were you able to validate / test these changes? Thanks.

[mpashkovskii]

@gaoikawa

@mpashkovskii - Were you able to validate / test these changes? Thanks.

I'm waiting access to EFA machine but I replaced those lines with:

        (void)discoverAccelWithHwloc();
        num_numa_nodes = hwloc_get_nbobjs_by_type(hwloc_topology, HWLOC_OBJ_NUMANODE);
        if (num_numa_nodes == 0) {
            num_numa_nodes = 1;
        }

And got from libfabric_topology_test following:

root@z:/app/repos/RIXL# ./build/test/unit/utils/libfabric/libfabric_topology_test
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1772458212.825362  103897 libfabric_topology_test_hip.cpp:34] === Testing Libfabric Topology Implementation ===
I0000 00:00:1772458212.826088  103897 libfabric_topology_test_hip.cpp:37] 1. Testing topology discovery...
I0000 00:00:1772458218.125358  103897 libfabric_topology_hip.cpp:137] Discovered 1 socket devices (TCP fallback)
I0000 00:00:1772458218.125429  103897 libfabric_topology_hip.cpp:105] Using simplified topology for sockets devices (no topology mapping needed)
I0000 00:00:1772458218.125449  103897 libfabric_topology_hip.cpp:376] Discovered 0 NVIDIA, 8 AMD, and 0 Neuron accelerators via hwloc
I0000 00:00:1772458218.125460  103897 libfabric_topology_hip.cpp:117] TCP devices available globally - no accelerator-specific mapping required
I0000 00:00:1772458218.125474  103897 libfabric_topology_test_hip.cpp:40]    SUCCESS: Topology discovery completed successfully
I0000 00:00:1772458218.125489  103897 libfabric_topology_test_hip.cpp:43] 2. Topology Information:
I0000 00:00:1772458218.125498  103897 libfabric_topology_test_hip.cpp:79] 3. Skipping GPU-specific tests (no GPUs detected)
I0000 00:00:1772458218.126210  103897 libfabric_topology_test_hip.cpp:86] === Test completed successfully! ===

Once I get EFA machine, i'll rerun the test.

[ddurnov]

My recommendation is to start validation with OFI providers and configurations you have access to

[mpashkovskii]

Will start

[edgargabriel]

@mpashkovskii could you have a look a the clang-format failure? Otherwise I would say the PR is good to merge (and it looks like its a simple fix)

[mpashkovskii]

@edgargabriel

@mpashkovskii could you have a look a the clang-format failure? Otherwise I would say the PR is good to merge (and it looks like its a simple fix)

I'll do, however the PR is not fully ready. I'm now fixing it with verbs but proper testing with EFA is needed. I suggest to merge only after the EFA testing is over.

[edgargabriel]

@edgargabriel

@mpashkovskii could you have a look a the clang-format failure? Otherwise I would say the PR is good to merge (and it looks like its a simple fix)

I'll do, however the PR is not fully ready. I'm now fixing it with verbs but proper testing with EFA is needed. I suggest to merge only after the EFA testing is over.

I disagree. We can not wait for the efa before we have the first version of the code up and running. If you look at the plan that I uploaded to JIRA, we expect most of the work to be done on libfabric with the verbs provider, enabling efa is one of the last steps.

In addition, for such a relatively complex topic, I would prefer to have things done step by step, not just one gigantic PR where everything is expected to work.

[mpashkovskii]

@edgargabriel

I disagree. We can not wait for the efa before we have the first version of the code up and running. If you look at the plan that I uploaded to JIRA, we expect most of the work to be done on libfabric with the verbs provider, enabling efa is one of the last steps.

In addition, for such a relatively complex topic, I would prefer to have things done step by step, not just one gigantic PR where everything is expected to work.

We initially discussed that plan with @ddurnov but I'm OK to adjust it. I can finish testing with verbs and we can merge the PR.

@ddurnov thoughts?

[edgargabriel]

@mpashkovskii I resynced the RIXL repository with the upstream NIXL repo. Can you check whether you need to rebase your current PR?

[mpashkovskii]

@edgargabriel merged develop branch and resolved the conflict. Will double-check if there are any issues on Monday

[edgargabriel]

Otherwise looks good to me.

[edgargabriel]

I am ok with this here, but its probably something that we will have to address at one point. The order in which devices are presented on the HIP layer and on the HSA layer doesn't necessarily have to match. So the clean solution is to have somewhere some code that performs the matching using the BDF retrieved for a hip device, and identifying the HSA agent with the same BDF