Mix precision

[vegah]

Motivation

Base: this branch is ROCm/FLM_Q4NX_Converter@mix-precision
(038e67e break: q8nx standard) with one commit on top. That branch is the
starting point and does the larger half of the work; this PR adds the last
piece and targets main so the fix reaches users of the default branch.

main cannot currently produce a loadable model.q4nx for any published Qwen3.5
build. The converter runs to completion and writes a file that looks entirely
correct — valid safetensors, every expected tensor name, a size within 0.15 % of
the official build — and FastFlowLM then exits during model load without
printing anything, because the NPU kernels are compiled for a fixed tile
layout.

Two changes are needed to close the gap:

1. 038e67e break: q8nx standard — yours, from mix-precision, unmerged as
   of this writing. It routes Q8_0 through _pack_q8nx instead of
   _pack_q4nx_8b, which stops col_block_size being forced 256 → 128 and
   padded up to the Q4_1 block size. That alone moves the middle dimension from
   32 to 16, matching FastFlowLM. Carried here unchanged; all credit yours.
2. e5cfd3a — the only new commit in this PR. One leftover remained in that
   path: Q8_0 has no bias term, but _pack_q8nx still reserves space for one, so
   mix-precision on its own still produces a file that will not load.

If you would rather land these separately, merging mix-precision first and then
taking e5cfd3a on top gives the same result — the two are independent edits to
the same function.

Technical Details

_pack_q8nx emits a placeholder bias block when m is None:

# We pack to q8nx, if there is no bias, we provide a fake bias of all zero
zero_np = np.zeros_like(scales_np)
merged = np.concatenate([scales_np, zero_np, data_np], axis=-1)

FastFlowLM's own builds reserve no such space. For a 32 × 256 block:

data    32 rows x 256 cols x 1 byte           = 8192
scales  (256/32 groups) x 32 rows x 2 bytes   =  512
                                        total =  8704   ← every published build
with the zero bias block                      =  9216   ← what main produces

The change drops the placeholder, leaving [scales, data].

For configs/qwen3.5_9b.json this affects the four tensor families carrying
"default_tensor_type": "Q8_0" — linear_alpha_proj, linear_beta_proj,
linear_out_proj and lm_head. That is 24 GatedDeltaNet layers × 3 plus the
output head = 73 of 475 tensors. The other 402 (Q4_1) were already correct on
main, which is why the size difference is small enough to be easy to miss.

The 8704 figure is not specific to the 9B — it holds across the whole published
Qwen3.5 line (see Test Result).

Test Plan

Converted the stock Qwen/Qwen3.5-9B and compared the output against the
published FastFlowLM/Qwen3.5-9B-NPU2:

# HF -> F16 GGUF. --no-mtp because llama.cpp exports the multi-token-prediction
# head as a 33rd block, and the q4nx name_map has no entry for nextn.*
python llama.cpp/convert_hf_to_gguf.py Qwen3.5-9B --outfile model-f16.gguf \
  --outtype f16 --no-mtp

# F16 -> Q4_1. The per-tensor Q8_0 overrides matter: the converter packs each
# tensor at the type it actually finds, so a blanket Q4_1 source yields the
# wrong layout for exactly the 73 tensors above.
llama-quantize --tensor-type ssm_alpha=q8_0 --tensor-type ssm_beta=q8_0 \
               --tensor-type ssm_out=q8_0 --output-tensor-type q8_0 \
               model-f16.gguf model-q4_1.gguf Q4_1

# Q4_1 -> q4nx. Architecture forced: llama.cpp writes
# general.architecture = "qwen35", which substring-matches the plain "qwen3"
# entry and loads configs/qwen3.json.
python -c "from q4nx import create_converter; \
  create_converter('model-q4_1.gguf', 'qwen3.5-9B').convert( \
  q4nx_path='out', weights_type='language')"

model.q4nx is a safetensors file, so the header can be compared directly —
including against published builds over HTTP range requests, without downloading
7 GB:

import json, struct, urllib.request
url = "https://huggingface.co/FastFlowLM/Qwen3.5-9B-NPU2/resolve/main/model.q4nx"
n = struct.unpack('<Q', urllib.request.urlopen(
        urllib.request.Request(url, headers={"Range": "bytes=0-7"})).read(8))[0]
hdr = json.loads(urllib.request.urlopen(
        urllib.request.Request(url, headers={"Range": f"bytes=8-{8+n-1}"})).read(n))
print(hdr["lm_head.weight"]["shape"])     # [7760, 16, 8704]

Then loaded the result on hardware: Ryzen AI 9 HX 370 (XDNA2), FastFlowLM
0.9.4x, registered as a separate model alongside the stock one.

Test Result

Layout matches the published build exactly.

official FastFlowLM/Qwen3.5-9B-NPU2 : 7,632,668,880 bytes
this PR                             : 7,632,668,880 bytes
tensors                             : 475 / 475 matching name, shape and dtype

Before the fix, those 73 tensors were […, 16, 9216] against the expected
[…, 16, 8704]; on unmodified main they were […, 32, 5120].

The 8704 layout is consistent across every published Qwen3.5 build, read from
their safetensors headers:

build	lm_head (Q8_0)	ssm_out (Q8_0)	down_proj (Q4_1)
0.8B	[7760, 4, 8704]	[32, 8, 8704]	[32, 14, 5120]
2B	[7760, 8, 8704]	[64, 8, 8704]	[64, 24, 5120]
4B	[7760, 10, 8704]	[80, 16, 8704]	[80, 36, 5120]
9B	[7760, 16, 8704]	[128, 16, 8704]	[128, 48, 5120]

Runs on the NPU. Coherent multilingual generation, correct code-switching, no
repetition or token corruption:

>>> Kan du ikke snakke norsk, da?
Ja, jeg snakker norsk! Jeg kan hjelpe deg med å svare på spørsmål, skrive
tekster, oversette eller bare ha en samtale.

This last check matters beyond "it loads": matching shapes only prove the
container is right. A wrong element ordering inside each Q8 block would also
load and then emit fluent nonsense.

Also verified on modified weights. A LoRA fine-tune of the same base, merged
with llama-export-lora, converts to a model.q4nx of identical size with all
475 tensors matching — as expected, since a LoRA merge only touches Q4_1 tensors
and leaves the 73 Q8_0 ones untouched.

For reference, the one locally-installed model with the old 5120-wide Q8_0
layout is a Qwen3.5-4B at flm_version 0.9.36, which current FastFlowLM already
refuses:

[FLM] Local model qwen3.5:4b version: 0.9.36 < 0.9.45
[FLM] Model qwen3.5:4b is not compatible with the current FLM version.

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests

[Atomic-Germ]

Can confirm these fixes; because I did basically the same in my fork and it worked; https://huggingface.co/Atomic-Germ/NuExtract3-4B-NPU2 is a qwen3.5, as is https://huggingface.co/Atomic-Germ/Qwopus3.5-9B-Coder-NPU2 , etc.