Error while converting qwen-3.5 to rkllm

[phkrl]

python3 ./converter.py --quantization Q8_K_M --dtype float16 --output-dir /path/to/output/qwen-3.5-4B-rknpu /path/to/Qwen3.5-4B
2026-03-10 19:16:56,254 - src.converter - INFO - Starting conversion of Qwen3.5-4B
2026-03-10 19:16:56,255 - src.converter - INFO - Loading model and tokenizer...
Using `use_fast=True` but `torchvision` is not available. Falling back to the slow image processor.
2026-03-10 19:17:02,418 - src.converter - INFO - No multimodal processor found, using text-only mode
`torch_dtype` is deprecated! Use `dtype` instead!
2026-03-10 19:17:02,714 - src.converter - ERROR - Error loading model and tokenizer: 'Qwen3_5Config' object has no attribute 'vocab_size'
2026-03-10 19:17:02,714 - __main__ - ERROR - Error during conversion: 'Qwen3_5Config' object has no attribute 'vocab_size'

Tried on commit 49e1295020d3b04cd4bdee99bd167670d9464f13

[phkrl]

Made a bit of investigation on this. The following patch allows to bypass this error using special qwen 3.5 config:

diff --git a/converter/src/converter.py b/converter/src/converter.py
index 05ad86e..32b6541 100644
--- a/converter/src/converter.py
+++ b/converter/src/converter.py
@@ -13,6 +13,7 @@ from typing import Optional, Dict, Any
 from dataclasses import dataclass
 from tqdm import tqdm
 from transformers import AutoModelForCausalLM, AutoTokenizer, AutoProcessor
+from transformers import Qwen3_5TextConfig
 from .quantization import QuantizationConverter
 from .utils import ModelMetadata, save_model_metadata
 from .rkllm import RKLLMConverter
@@ -115,11 +116,13 @@ class HuggingFaceToRKLLMConverter:
                 logger.info("No multimodal processor found, using text-only mode")

             # Load model
+            model_config = Qwen3_5TextConfig()
             self.model = AutoModelForCausalLM.from_pretrained(
                 self.config.model_id,
                 token=self.config.token,
                 torch_dtype=torch.float16 if self.config.dtype == 'float16' else torch.float32,
-                device_map=self.config.device
+                device_map=self.config.device,
+                config=model_config
             )

             logger.info("Model and tokenizer loaded successfully")

The attempt to convert 4B model took >16GB RAM and failed with the following log:

2026-03-19 21:52:31,043 - src.converter - INFO - Starting conversion of Qwen3.5-4B
2026-03-19 21:52:31,043 - src.converter - INFO - Loading model and tokenizer...
Using `use_fast=True` but `torchvision` is not available. Falling back to the slow image processor.
2026-03-19 21:52:35,278 - src.converter - INFO - No multimodal processor found, using text-only mode
The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
Loading weights: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 426/426 [00:27<00:00, 15.47it/s]
Qwen3_5ForCausalLM LOAD REPORT from: /home/rps/build/neural/Qwen3.5-4B
Key                                                                 | Status   |

--------------------------------------------------------------------+----------+---------------------------------------------------------------------------------------------------
lm_head.weight                                                      | MISSING  |

model.layers.{0...31}.mlp.up_proj.weight                            | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([9216, 2560]) vs model:torch.Size([12288, 4096])
model.layers.{0...30}.linear_attn.in_proj_a.weight                  | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([32, 2560]) vs model:torch.Size([32, 4096])

model.layers.{0...31}.mlp.gate_proj.weight                          | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([9216, 2560]) vs model:torch.Size([12288, 4096])
model.layers.{0...31}.post_attention_layernorm.weight               | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([2560]) vs model:torch.Size([4096])

model.layers.{0...30}.linear_attn.in_proj_b.weight                  | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([32, 2560]) vs model:torch.Size([32, 4096])

model.layers.{0...30}.linear_attn.out_proj.weight                   | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([2560, 4096]) vs model:torch.Size([4096, 4096])
model.layers.{0...31}.input_layernorm.weight                        | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([2560]) vs model:torch.Size([4096])

model.layers.{0...30}.linear_attn.in_proj_z.weight                  | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([4096, 2560]) vs model:torch.Size([4096, 4096])
model.norm.weight                                                   | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([2560]) vs model:torch.Size([4096])

model.layers.{0...31}.mlp.down_proj.weight                          | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([2560, 9216]) vs model:torch.Size([4096, 12288])
model.layers.{3, 7, 11, 15, 19, 23, 27, 31}.self_attn.q_proj.weight | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([8192, 2560]) vs model:torch.Size([8192, 4096])
model.layers.{0...30}.linear_attn.in_proj_qkv.weight                | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([8192, 2560]) vs model:torch.Size([8192, 4096])
model.layers.{3, 7, 11, 15, 19, 23, 27, 31}.self_attn.o_proj.weight | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([2560, 4096]) vs model:torch.Size([4096, 4096])
model.layers.{3, 7, 11, 15, 19, 23, 27, 31}.self_attn.v_proj.weight | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([1024, 2560]) vs model:torch.Size([1024, 4096])
model.layers.{3, 7, 11, 15, 19, 23, 27, 31}.self_attn.k_proj.weight | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([1024, 2560]) vs model:torch.Size([1024, 4096])
model.embed_tokens.weight                                           | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([248320, 2560]) vs model:torch.Size([248320, 4096])

Notes:
- MISSING       :those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.
- MISMATCH      :ckpt weights were loaded, but they did not match the original empty weight shapes.
2026-03-19 21:57:04,870 - src.converter - ERROR - Error loading model and tokenizer: You set `ignore_mismatched_sizes` to `False`, thus raising an error. For details look at
the above report!
2026-03-19 21:57:04,870 - __main__ - ERROR - Error during conversion: You set `ignore_mismatched_sizes` to `False`, thus raising an error. For details look at the above report!

UPD: small change after installing torchvision

2026-03-19 22:03:27,399 - src.converter - INFO - Loading model and tokenizer...
2026-03-19 22:03:35,425 - src.converter - INFO - Loaded multimodal processor

[phkrl]

So, the patch actually missed one line

diff --git a/converter/src/converter.py b/converter/src/converter.py
index 05ad86e..6c3262c 100644
--- a/converter/src/converter.py
+++ b/converter/src/converter.py
@@ -13,6 +13,7 @@ from typing import Optional, Dict, Any
 from dataclasses import dataclass
 from tqdm import tqdm
 from transformers import AutoModelForCausalLM, AutoTokenizer, AutoProcessor
+from transformers import Qwen3_5TextConfig
 from .quantization import QuantizationConverter
 from .utils import ModelMetadata, save_model_metadata
 from .rkllm import RKLLMConverter
@@ -115,11 +116,14 @@ class HuggingFaceToRKLLMConverter:
                 logger.info("No multimodal processor found, using text-only mode")
             
             # Load model
+            model_config = Qwen3_5TextConfig()
             self.model = AutoModelForCausalLM.from_pretrained(
                 self.config.model_id,
                 token=self.config.token,
                 torch_dtype=torch.float16 if self.config.dtype == 'float16' else torch.float32,
-                device_map=self.config.device
+                device_map=self.config.device,
+                config=model_config,
+                ignore_mismatched_sizes=True
             )
             
             logger.info("Model and tokenizer loaded successfully")

--quantization Q8_K_M does not actually work (despite readme says otherwise), but Q8_0 does. It took up to 53Gb RAM to convert 4B model to rkllm.

UPD: Serving model does not work. rkllama_server shows following messages:

FROM: .rkllm
HuggingFace Path: /home/rps/build/neural/Qwen3.5-4B/
2026-03-25 10:27:45,535 - rkllama.rkllm - DEBUG - Initializing RKLLM model from /home/rps/build/neural/Qwen3.5-4B-Q8_0-rkllm/Qwen3.5-4B/.rkllm with options: {'temperature': '0.7', 'num_ctx': '4096', 'max_new_tokens': '1024', 'top_k': '7', 'top_p': '0.5', 'repeat_penalty': '1.1', 'frequency_penalty': '0.0', 'presence_penalty': '0.0', 'mirostat': '0', 'mirostat_tau': '3', 'mirostat_eta': '0.1', 'from': '".rkllm"', 'huggingface_path': '"/home/rps/build/neural/Qwen3.5-4B/"', 'system': '"You are a helpful AI assistant."'}
I rkllm: rkllm-runtime version: 1.2.3, rknpu driver version: 0.9.8, platform: RK3588
I rkllm: loading rkllm model from /home/rps/build/neural/Qwen3.5-4B-Q8_0-rkllm/Qwen3.5-4B/.rkllm
E rkllm: invalid rkllm model!
2026-03-25 10:27:45,544 - rkllama.worker - ERROR - Failed creating the worker for model 'Qwen3.5-4B': Failed to initialize RKLLM model: -1
2026-03-25 10:27:45,547 - werkzeug - INFO - 127.0.0.1 - - [25/Mar/2026 10:27:45] "POST /load_model HTTP/1.1" 400 -

I suppose, either the library does not support qwen 3.5, or conversion got wrong. There was this message:

rkllm.py 111 2026-03-25 09:49:41,784 - src.rkllm - INFO - Converting weights for layer: model.layers.0.input_layernorm.weight
rkllm.py 114 2026-03-25 09:49:41,784 - src.rkllm - INFO - Weight shape: (4096,)
/home/rps/build/other/rkllama/converter/src/rkllm.py:203: RuntimeWarning: invalid value encountered in cast
  q8_weights = np.clip(weight_array * 128, -128, 127).astype(np.int8)
rkllm.py 130 2026-03-25 09:49:41,934 - src.rkllm - INFO - Converted 4096 bytes

[phkrl]

As a sidenote: there is https://github.com/invisiofficial/rk-llama.cpp which somehow is capable to run new models like qwen 3.5 and gemma 4 on rknpu

[danielferr85]

@phkrl Just now released a PR with native integration with that repo from rkllama.

rkllama create workers for models in .GGUF format and fires llama-server instances per model.... More info check issue #154