feat: enable topology discovery and memory transfer with AMD GPUs with verbs;ofi_rxm provider on 8xMI300X machine

[mpashkovskii]

Extends the Libfabric plugin's topology discovery and rail selection to support AMD MI300X systems with InfiniBand/RoCE NICs (Broadcom bnxt_re, Mellanox ConnectX). Previously, topology-aware rail building only worked for AWS EFA devices. This PR adds:

1. InfiniBand device identification — new isInfiniBandDevice() method recognizes Mellanox HCAs (vendor 0x15b3, class 0x0c06) via hwloc
2. verbs;ofi_rxm provider support — topology discovery, PCIe mapping, and rail building now handle the verbs RDM provider alongside EFA/CXI
3. MI300X topology test — hwloc XML captured from real MI300X hardware, with 8 bandwidth scenarios validating NUMA-aware DRAM rail selection

Testing

Tested with libfabric_topology_test which executes 9 sub-tests: 1 topology query + 8 bandwidth scenarios. All pass. I run three tests:

• With the real hwloc information on the 8xMI300x machine: NIXL_LOG_LEVEL=TRACE ./build/test/unit/utils/libfabric/libfabric_topology_test mi300x Log file: libfabric_topology_test.log.zip Total: 364,536 TRACE-level log lines.

• With the mock hwloc XML colected from the same machine:

  cd test/unit/utils/libfabric/topo
  ../../../../../build/test/unit/utils/libfabric/libfabric_topology_test mi300x

• With the complete set of XML mocks from the repo:

  cd test/unit/utils/libfabric/topo
  ../../../../../build/test/unit/utils/libfabric/libfabric_topology_test

Real hwloc information on the 8xMI300x machine

Phase 1: Topology Query for DRAM_SEG Rail Count (lines 1–429)

Validates that the topology engine computes the correct number of rails per NUMA node.

Step	Key Log Messages
Init	Testing topology query for DRAM_SEG NUMA-aware rail selection on instance type mi300x
hwloc load	hwloc topology initialized successfully with IO device support
NIC discovery	[TEST] Found OS device: bnxt_re0..7, mlx5_0, mlx5_1 (10 OpenFabrics devices via hwloc)
fi_getinfo mock	Found device - domain: bnxt_re5, provider=verbs;ofi_rxm (×10 devices)
Provider selection	Discovered 10 InfiniBand (verbs;ofi_rxm) devices
PCIe→NIC mapping	Mapped PCIe 0:a3:00.0 → Libfabric bnxt_re5 (NIC link speed: 400000000000) (×10)
AMD GPU discovery	Found AMD accelerator: 0:05:00.0 (vendor=0x1002, device=0x74a1, class=0x1200) (×8 MI300X GPUs)
Accelerator summary	Discovered 0 NVIDIA, 8 AMD, and 0 Neuron accelerators via hwloc
NUMA mapping	NIC at PCIe address 0:23:00.0 is closest to NUMA node 0 — 6 NICs on NUMA 0, 4 on NUMA 1
PCIe capacity	NUMA node 0 capacity is 2020 Gbps / NUMA node 1 capacity is 2020 Gbps
Bandwidth uniformity	NIC bandwidth is uniform across all PCIe devices with bandwidth: 400 Gbps
Upstream warning	Non-uniform NIC mlx5_0 upstream link speed: 252 Gbps, expecting 504 Gbps (mlx5 behind narrower PCIe link)
GPU→NIC affinity	Accelerator-PCI 0:c7:00.0 → [bnxt_re6] (×8 GPU-to-NIC mappings)
Rail count result	SUCCESS: Topology calculated correct rail count 4 for DRAM_SEG NUMA-aware rail selection policy

Phase 2–9: Bandwidth Scenario Tests (lines 430–364,536)

Each scenario re-discovers topology, then runs single-threaded + multi-threaded rail selection validation at a specific bandwidth target. Bandwidth determines how many rails are needed.

Scenario	Bandwidth	Single-threaded	Multi-threaded	Line Range
1	0 Gbps	SUCCESS	SUCCESS	430–47,947
2	400 Gbps	SUCCESS	SUCCESS	47,948–95,464
3	800 Gbps	SUCCESS	SUCCESS	95,465–142,981
4	1,200 Gbps	SUCCESS	SUCCESS	142,982–190,498
5	2,400 Gbps	SUCCESS	SUCCESS	190,499–238,016
6	2,800 Gbps	SUCCESS	SUCCESS	238,017–285,534
7	3,200 Gbps	SUCCESS	SUCCESS	285,535–333,052
8	4,000 Gbps	SUCCESS	SUCCESS	333,053–364,536

Each scenario follows the same internal flow:

1. Topology rediscovery — full hwloc + fi_getinfo mock cycle
2. Single-threaded test — sequentially selects rails for each NUMA node, verifies correct NIC assignment and round-robin distribution (~31K lines of TRACE per scenario)
3. Multi-threaded test — concurrent rail selection from multiple threads, verifies thread safety and correctness (~16K lines)
4. Cleanup — rail resource teardown (Closing endpoint/CQ/AV/domain/fabric for rail N)

Final line: Test scenario [bandwidth=4000] on instance type mi300x SUCCESS / === Test completed successfully! ===

Topology file

Topology XML file was generated using following snippet:

#include <hwloc.h>
#include <stdio.h>
int main() {
    hwloc_topology_t t;
    hwloc_topology_init(&t);
    hwloc_topology_set_io_types_filter(t, HWLOC_TYPE_FILTER_KEEP_ALL);
    hwloc_topology_load(t);
    hwloc_topology_export_xml(t, "mi300x-topo.xml", 0);
    hwloc_topology_destroy(t);
    printf("Done\n");
    return 0;
}

And then gcc export_topo.c -lhwloc -o export_topo && ./export_topo

Probably the same can be achieved with lstopo --of xml mi300x-topo.xml

[mpashkovskii]

For some reasons unneeded changes got included. Closing the PR to create a clean one.

[edgargabriel]

For some reasons unneeded changes got included. Closing the PR to create a clean one.

@mpashkovskii please note that this repository is now deprecated, I added notes on the README_rocm.md page. We have added ROCm support into the upstream NIXL repository, any future work is expected to go directly there, not to this repo anymore.

[mpashkovskii]

@edgargabriel yes I noticed it right after the merge. I created a PR in NIXL: ai-dynamo/nixl#1750 would appreciate review