Nemotron-H MoE: expert NVFP4 scale2 tensors not loaded — repetitive garbage output

[poad42]

Nemotron-H MoE: expert NVFP4 scale2 tensors not loaded — repetitive garbage output

Problem

When loading a ModelOpt NVFP4 GGUF for NemotronHForCausalLM (the MoE variant, LLM_ARCH_NEMOTRON_H_MOE), the model generates repetitive garbage:

Prompt: "The capital of France is"
Output: "the capital of the United States.. The capital of the United States is the capital..."

The root cause is that nemotron-h.cpp's load_arch_tensors() creates the expert weight tensors (ffn_down_exps, ffn_up_exps, ffn_down_shexp, ffn_up_shexp) but never creates the corresponding scale tensors (ffn_up_exps_s, ffn_down_exps_s, ffn_up_shexp_s, ffn_down_shexp_s).

The graph code at build_ffn_layer() correctly passes these to build_moe_ffn() and build_ffn():

build_moe_ffn(inp_latent,
    model.layers[il].ffn_gate_inp,
    model.layers[il].ffn_up_exps,
    ...
    model.layers[il].ffn_down_exps,
    ...
    model.layers[il].ffn_up_exps_s,    // <-- NULL (never created!)
    ...
    model.layers[il].ffn_down_exps_s); // <-- NULL (never created!)

Since the pointers are NULL, build_moe_ffn and build_ffn skip the ggml_mul(res, w_s) step, leaving the NVFP4 expert outputs unscaled by the per-tensor scale2 factor. This produces wrong logits and the model degrades to repetitive output.

Model

• nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 (ModelOpt W4A16_NVFP4, 128 routed experts + shared expert)
• Converted via convert_hf_to_gguf.py with PR NVFP4: native load + quantize support, and fix vision with draft-mtp #70 (NVFP4 support) + fixes for W4A16_NVFP4 detection
• GGUF has the scale tensors written (blk.N.ffn_up_exps.scale, blk.N.ffn_down_exps.scale, etc.)
• Server loads model without errors but scale tensors are silently missing (NULL)

Fix

Add create_tensor calls for the expert/shared-expert scale tensors in load_arch_tensors() in src/models/nemotron-h.cpp:

// After ffn_up_exps / ffn_down_exps creation:
layer.ffn_up_exps_s   = create_tensor(tn(LLM_TENSOR_FFN_UP_EXPS,   "scale", i), {1}, TENSOR_NOT_REQUIRED);
layer.ffn_down_exps_s = create_tensor(tn(LLM_TENSOR_FFN_DOWN_EXPS, "scale", i), {1}, TENSOR_NOT_REQUIRED);

// After ffn_up_shexp / ffn_down_shexp creation:
layer.ffn_up_shexp_s   = create_tensor(tn(LLM_TENSOR_FFN_UP_SHEXP,   "scale", i), {1}, TENSOR_NOT_REQUIRED);
layer.ffn_down_shexp_s = create_tensor(tn(LLM_TENSOR_FFN_DOWN_SHEXP, "scale", i), {1}, TENSOR_NOT_REQUIRED);

Workaround (converter-side)

I worked around this by baking the scale2 into the E4M3 per-sub-block scales during conversion:

1. Convert each UE4M3 scale byte → float32
2. Multiply by scale2 (the global per-tensor scale)
3. Convert back to nearest UE4M3 byte
4. Write scale2=1.0 as the separate scale tensor (so the server's ggml_mul is a no-op)

With this workaround, the model generates correctly:

Prompt: "The capital of France is"
Output: " Paris.  \nThe capital of Germany is Berlin"

Environment

• GPU: gfx1151 (AMD Ryzen AI MAX+ 395, Radeon 8060S, RDNA 3.5)
• Backend: HIP (not Vulkan)
• ROCmFPX build: latest main (includes PR NVFP4: native load + quantize support, and fix vision with draft-mtp #70 NVFP4 support)
• Model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 → GGUF

Additional converter fixes needed for Nemotron-H NVFP4

Two more bugs in convert_hf_to_gguf.py (already fixed locally):

1. NVFP4 detection: The code checks quant_algo == "NVFP4" but ModelOpt uses "W4A16_NVFP4" per-layer. Fixed with substring match: "NVFP4" in (v.get("quant_algo") or "").

2. FP8/NVFP4 routing in modelopt handler: The dequant_simple lambda was applied to ALL .weight_scale tensors, including NVFP4 expert tensors (uint8 weight with 2D E4M3 scale). This causes a broadcast error ([2688, 116] vs [2688, 928]). Fixed by checking w().dtype and skipping non-FP8 (uint8) weights.

[poad42]

Note: the proper fix is server-side

The scale-baking workaround in the converter works but has a small precision cost (float32 → UE4M3 → float32 round-trip loses a few bits per scale). The proper fix is in the server:

In src/models/nemotron-h.cpp, load_arch_tensors(), add after the expert weight tensor creation:

layer.ffn_up_exps_s   = create_tensor(tn(LLM_TENSOR_FFN_UP_EXPS,   "scale", i), {1}, TENSOR_NOT_REQUIRED);
layer.ffn_down_exps_s = create_tensor(tn(LLM_TENSOR_FFN_DOWN_EXPS, "scale", i), {1}, TENSOR_NOT_REQUIRED);
layer.ffn_up_shexp_s   = create_tensor(tn(LLM_TENSOR_FFN_UP_SHEXP,   "scale", i), {1}, TENSOR_NOT_REQUIRED);
layer.ffn_down_shexp_s = create_tensor(tn(LLM_TENSOR_FFN_DOWN_SHEXP, "scale", i), {1}, TENSOR_NOT_REQUIRED);

The graph code in build_ffn_layer() already passes these to build_moe_ffn() and build_ffn() — they just need to be loaded from the GGUF. Once this is done, the converter can stop baking the scale2 and write it normally.

I attempted to patch and rebuild but the build container kept OOM-ing. The converter-side workaround (scale-baking) is the pragmatic alternative.

[poad42]

Update: proper server-side fix confirmed

The proper fix is two changes, both server-side:

1. src/models/nemotron-h.cpp — load the expert scale tensors

In load_arch_tensors(), after creating the expert weight tensors, add:

// NVFP4 per-tensor scale2 for routed experts
layer.ffn_up_exps_s   = create_tensor(tn(LLM_TENSOR_FFN_UP_EXPS,   "scale", i), {n_expert}, TENSOR_NOT_REQUIRED);
layer.ffn_down_exps_s = create_tensor(tn(LLM_TENSOR_FFN_DOWN_EXPS, "scale", i), {n_expert}, TENSOR_NOT_REQUIRED);

// NVFP4 per-tensor scale2 for shared expert
layer.ffn_up_shexp_s   = create_tensor(tn(LLM_TENSOR_FFN_UP_SHEXP,   "scale", i), {1}, TENSOR_NOT_REQUIRED);
layer.ffn_down_shexp_s = create_tensor(tn(LLM_TENSOR_FFN_DOWN_SHEXP, "scale", i), {1}, TENSOR_NOT_REQUIRED);

Note: routed expert scales are {n_expert} (one per expert), shared expert scales are {1} (scalar).

2. src/llama-graph.cpp — remove double-scale application in build_moe_ffn

build_lora_mm_id already applies w_s via ggml_mul internally. But build_moe_ffn also had an explicit block that applied down_exps_s again:

// This block was applying scale2 a SECOND time (scale^2 bug):
if (down_exps_s) {
    ggml_tensor * s = ggml_reshape_3d(ctx0, down_exps_s, 1, n_expert, 1);
    s = ggml_repeat_4d(ctx0, s, 1, n_expert, n_tokens, 1);
    s = ggml_get_rows(ctx0, s, selected_experts);
    experts = ggml_mul(ctx0, experts, s);
    cb(experts, "ffn_moe_down_scaled", il);
}

Removing this block fixes the double-application. build_lora_mm_id handles it correctly once.

Verification

With both fixes, no converter workarounds needed. The model generates correctly:

Prompt: "The capital of France is"
Output: " Paris.  \nThe capital of Germany is Berlin"

Needle-in-haystack test (256K context, needle at 50% position):

Context	Needle found	Entropy	Top-1 prob
4K	YES	0.12	0.97
8K	YES	0.15	0.96
16K	YES	0.12	0.97
32K	YES	0.17	0.95
64K	YES	0.17	0.95
128K	YES	0.22	0.93
256K	NO	0.57	0.79

Needle found at 4K-128K. Entropy stable (0.12-0.22). No catastrophic degradation at any context length. Model is 19.8GB, runs at -ngl 999 -c 262144 on gfx1151.

Summary of all fixes needed for Nemotron-H NVFP4

File	Fix
src/models/nemotron-h.cpp	Add create_tensor for ffn_up_exps_s, ffn_down_exps_s (shape {n_expert}), ffn_up_shexp_s, ffn_down_shexp_s (shape {1})
src/llama-graph.cpp	Remove the explicit down_exps_s application block in build_moe_ffn (already handled by build_lora_mm_id)
convert_hf_to_gguf.py line ~955	quant_algo == "NVFP4" → substring match (ModelOpt uses "W4A16_NVFP4")
convert_hf_to_gguf.py line ~690	Skip non-FP8 (uint8) weights in modelopt handler (let _generate_nvfp4_tensors handle NVFP4)