[1/N] add fp8 fp32 scale support for custom RL model

[yiakwy-xpu-ml-framework-team]

Background

We added fp8 RL+SFT version of Deepseek V4 in week 0 support and suppressed DeepSeek V4 baseline in all major dimensions from our internal evaluation.

Hence we want to add 2 bit support for DeepSeek V4 with our Expert Pruning technology:

Noted, in H100/H800, we usually don't use E8M0 for scale, since it will introduce runtime overhead. FP32 scale is the best.

[yiakwy-xpu-ml-framework-team]

@antirez could you have a look at it ?

[antirez]

Hi, the PR itself has a few quality issues but especially it is not clear why it would be useful for the proejct as a whole given that we convert from DS4 hugging face formats.

[yiakwy-xpu-ml-framework-team]

Quantization is successful.

@antirez Thank you for the quick response, let me explain.

• our sft/RL model of deepseek v4 has embedding layer (bf16 or int32), while deepseek model has embedding with type int64

• since we are running in Hopper platform , our expert weight stored with E4M3 FP8 weight and weight scale stored with FP32 for best performance (which can verified in SGLang):

  

  Customer DSV4 sglang fp8 serving in Hopper platform with identity injectioin, private/public knowledge injection and enhanced security shield module
  

• Huggingface model is not SGLang compatible version, while our version is; and huggingface does not consider convert SFT/RL model from Bf16 to FP8 variants

The model is tuned specifically to handle Candonese, Chinese madarin and English efficiently.

[yiakwy-xpu-ml-framework-team]

Hi @antirez I can make sure the modification can generate correct model checkpoint for ds4. Wish your attention.

GB10 (2 bit dsv4-sft-rl, 15 toks/sec) :

Serving with raw model (no system prompt):