[Feature] Automatically Find the best NIC for each GPU

[zyksir]

For Now I am using mscclpp and I find that we have to explicitly name each devices for each GPU, while NCCL can find the best device for each GPU automatically. Most time we have to set MSCCLPP_HCA_DEVICES for different nodes to achieve the best performance. A feature about parsing the topo and auto-choosing the device will help a lot.

In addition, sometimes "libibverbs.so" cannot be found and only "libibverbs.so.1" can be found. For now, I use ln -sv /usr/lib/x86_64-linux-gnu/libibverbs.so.1 /usr/lib/x86_64-linux-gnu/libibverbs.so to fix this. But I think if you can search "libibverbs.so.1" in the mscclpp code, that will help a lot.

This is a related issue. sgl-project/sglang#6834

[chhwang]

@zyksir NCCL’s topology discovery doesn’t work correctly on Azure VMs, so we don’t bring it into MSCCL++. For now, it is our general recommendation to set MSCCLPP_HCA_DEVICES for your own system. If you think it would help, we may add an API interface instead so that you can set the devices by your own code logic. How do you think?

We will work on fixing the libibverbs issue you mentioned.

[chhwang]

@zyksir Please check if this PR resolves your libibverbs issue: #543