Greetings!
I have successfully run fp8 quant on Qwen-7B model and int8 quant on Qwen-0.5B model in docker "hiyouga/verl:ngc-th2.6.0-cu126-vllm0.8.4-flashinfer0.2.2-cxx11abi0", but I found a problem:
the reshard time of both is heavily higher than base bf16 model.
I found your example log of FP8 Rollout for Qwen2.5-0.5B-Instruct on DAPO has the same appearance:
Is that normal? Does quant operation give rise to that a heavy time increase during resharding? Or could you explain that in some other way?
Looking forward to your reply!
Greetings!
I have successfully run fp8 quant on Qwen-7B model and int8 quant on Qwen-0.5B model in docker "hiyouga/verl:ngc-th2.6.0-cu126-vllm0.8.4-flashinfer0.2.2-cxx11abi0", but I found a problem:
the reshard time of both is heavily higher than base bf16 model.
I found your example log of FP8 Rollout for Qwen2.5-0.5B-Instruct on DAPO has the same appearance:
Is that normal? Does quant operation give rise to that a heavy time increase during resharding? Or could you explain that in some other way?
Looking forward to your reply!