[Contributions] Submitting performance benchmarks

[asaiacai]

I've been using torchtitan for some time to benchmark some multinode bare metal clusters (H200, B200, GB200). While I like my internal notes about torchtitan performance, many people have been asking me to share my results in a more public peer reviewed fashion. Should I just be making a PR to docs/performance.md or is there another process for review.

[tianyu-l]

Great idea! This is more than welcome.

I would propose creating a dedicated folder benchmarks under root folder, where each file containing

1. the person / entity / GH name
2. hardware setup including GPUs, interconnections, etc.
3. model / theme of benchmarking, e.g. llama3 / flux.1
4. performance report, with configs used to reproduce, either via toml file or via this option https://github.com/pytorch/torchtitan/blob/main/torchtitan/config_manager.py#L47
5. versions of torchtitan, torch, torchao, or any relevant dependency

Please give feedbacks, so that I can create the folder with a template / guidance to unblock your submission.

[tianyu-l]

@asaiacai Here's the PR #1296

[asaiacai]

this is great thank you :)

[tianyu-l]

hey @asaiacai , following up on this looking forward to seeing your benchmarking results

[asaiacai]

apologies for the delay here. just submitted #1361 . I'm still reviewing it with my team just to make sure we didn't miss anything in our infra for our submission that could give us better results. i'll mark the PR as ready once we want to submit. :)