bf16 datatype supported?

[clockfly]

Do I need to set some flags to enable bfloat16?

Currently, it is not working. Report following error for nccl test:

llm011:2504887:2504887 [7] enqueue.cc:1175 NCCL WARN Error : no algorithm/protocol available
llm011:2504887:2504887 [7] NCCL INFO enqueue.cc:1273 -> 3
llm011:2504887:2504887 [7] NCCL INFO enqueue.cc:567 -> 3
llm011:2504887:2504887 [7] NCCL INFO enqueue.cc:936 -> 3

llm012:1731092:1731092 [7] enqueue.cc:1175 NCCL WARN Error : no algorithm/protocol available
llm012:1731092:1731092 [7] NCCL INFO enqueue.cc:1273 -> 3
llm012:1731092:1731092 [7] NCCL INFO enqueue.cc:567 -> 3
llm012:1731092:1731092 [7] NCCL INFO enqueue.cc:936 -> 3
llm012:1731092:1731092 [7] NCCL INFO group.cc:140 -> 3
llm012:1731092:1731092 [7] NCCL INFO group.cc:341 -> 3
llm012:1731092:1731092 [7] NCCL INFO group.cc:422 -> 3
llm012:1731092:1731092 [7] NCCL INFO group.cc:106 -> 3

[bureddy]

it is supported on NDR network fabric.

[cjld]

I have the same problem: SHARP int8,uint8,bfloat16 Datatypes not supported, I am using 200G HDR switch, do HDR support bf16?

[salanki]

No, you need NDR / Quantum2.

[QiuBiuBiu]

Hi @salanki @bureddy @cjld , Where did you get the info that BF16 can't be supported by HDR? I didn't get these datatype difference between HDR and NDR on the official website.
Quantum2/NDR: https://www.nvidia.com/en-us/networking/quantum2/
Quantum1/HDR: https://network.nvidia.com/files/doc-2020/pb-qm8790.pdf

[salanki]

@QiuBiuBiu you can see in the code that CX7 is required for bf16 initialization.