H100 performance down

[Lmywl]

When I test Gemm_RS with M=4096 N=12288 K=49152 on the H100, It can be run directly. But it seems that the performance is worse than the baseline method, what causes this happen ?

GEMM configs saved.
Bandwidth curve captured.
Start predictive searching.
Search process finished.
Searched latency: 11.6717
Best solution:  [1536]
Solution saved.

        Item                 Value
        -----                -----
        m                     4096
        n                    12288
        k                    49152
        tile_num              1536
        gemm_dur (ms)         11.2484
        comm_dur (ms)          0.2897
        baseline_dur (ms)          7.8775
        overlap_dur (ms)         11.6806
        speedup             0.6744

[Lmywl]

When I test Gemm_RS with M=4096 N=12288 K=49152 on the H100, It can be run directly. But it seems that the performance is worse than the baseline method, what causes this happen ?

GEMM configs saved.
Bandwidth curve captured.
Start predictive searching.
Search process finished.
Searched latency: 11.6717
Best solution:  [1536]
Solution saved.

        Item                 Value
        -----                -----
        m                     4096
        n                    12288
        k                    49152
        tile_num              1536
        gemm_dur (ms)         11.2484
        comm_dur (ms)          0.2897
        baseline_dur (ms)          7.8775
        overlap_dur (ms)         11.6806
        speedup             0.6744

Besides, your baseline method seems not implenmented with torch + Nccl or other known method. What's it based on ?

[hkeee21]

Hi, thanks for trying.

We haven't tuned the performance on H100 yet, which means the GEMM templates provided in the repo deliver performance far from the optimal performance of the Hopper GEMMs. Additionally, the code is set to "sm80" in the CUTLASS arguments, resulting in further degradation. The Hopper adaptation is in the roadmap list.

The baseline is simply cuBLAS + NCCL, which can be found in https://github.com/infinigence/FlashOverlap/blob/main/src/baseline_impl.cu.

[Lmywl]

Hi, thanks for trying.

We haven't tuned the performance on H100 yet, which means the GEMM templates provided in the repo deliver performance far from the optimal performance of the Hopper GEMMs. Additionally, the code is set to "sm80" in the CUTLASS arguments, resulting in further degradation. The Hopper adaptation is in the roadmap list.

The baseline is simply cuBLAS + NCCL, which can be found in https://github.com/infinigence/FlashOverlap/blob/main/src/baseline_impl.cu.

In fact, when I test flashoverlap in A100, the result seems not so good, do I need to change the code to adopt GPU A100?

Processing: M=1024 N=8192 K=7392
GEMM configs saved.
Bandwidth curve captured.
compute hint for iteration 0
Start predictive searching.
Search process finished.
Searched latency: 0.7287
Best solution:  [512]
Solution saved.
world_size is: 4

        Item                 Value
        -----                -----
        m                     1024
        n                     8192
        k                     7392
        tile_num               512
        gemm_dur (ms)          0.5079
        comm_dur (ms)          0.1333
        baseline_dur (ms)          0.7543
        overlap_dur (ms)          0.7295
        speedup             1.0339

Processing: M=8192 N=12288 K=49152
GEMM configs saved.
Bandwidth curve captured.
compute hint for iteration 0
Start predictive searching.
Search process finished.
Searched latency: 43.7651
Best solution:  [3072]
Solution saved.
world_size is: 4

        Item                 Value
        -----                -----
        m                     8192
        n                    12288
        k                    49152
        tile_num              3072
        gemm_dur (ms)         37.9066
        comm_dur (ms)          0.8724
        baseline_dur (ms)         44.1863
        overlap_dur (ms)         43.7673
        speedup             1.0096

[hkeee21]

Here are my results.

M = 1024, N=8192, K=7392
Best solution: [512]

    Item                 Value
    -----                -----
    m                     1024
    n                     8192
    k                     7392
    tile_num               512
    gemm_dur (ms)          0.5205
    comm_dur (ms)          0.1298
    baseline_dur (ms)          0.7291
    overlap_dur (ms)          0.7075
    speedup             1.0306

M = 8192, N=12288, K=49152
Best solution: [318, 318, 318, 318, 636, 318, 636, 210]

    Item                 Value
    -----                -----
    m                     8192
    n                    12288
    k                    49152
    tile_num              3072
    gemm_dur (ms)         39.7025
    comm_dur (ms)          0.8473
    baseline_dur (ms)         44.3041
    overlap_dur (ms)         40.9780
    speedup             1.0812

The first case seems normal. The speedup is low because of the under-utilized bandwidth with segmented communication when M and N are not large enough.
The performance of the second case differs, and it seems there is no overlap in your results. I guess there is something wrong with the tuning process. But don't expect a significant overlap benefit as well, as the computation latency is much higher than the communication latency.