V0.4 Release Plan

[cp5555]

Release Manager

Peng Cheng (@cp5555)

Endgame

• Code freeze: Feb. 9th, 2024
• Bug Bash date: Feb. 12th, 2024
• Release date: Feb. 23rd, 2024

Main Features

MS-AMP O3 Optimization

• 1. Support auto scaling factor tuning (ASFT) for FP8 collective communication (Related to Auto scaling factor tuning for FP8 collective communication #41, [Feature] Auto scaling factor tuning for FP8 collective communication #140)
• 2. Support PyTorch FSDP (Related to Support for MS-AMP in FSDP #122)

MS-AMP Improvement

• 1. Move extension installation from post install to setup.py (Related to Moving extension installation from post install to setup.py under project root folder #43)
• 2. Improve FP8 kernel performance in MS-AMP (Optimize performance by fuse adding high precision tensor to fp8 tensor #132)
• 3. MS-AMP support on different devices (Nvidia A100 and AMD MI300X)

MS-AMP Examples

• 1. Release the datapoints (Related to Training curve datapoints or smoothing #115)

Backlog

FP8 on Activation

[tocean]

Will move all the non-started work items to next release v0.5.