Qwen3.6-35B-A3B RKLLM conversion for RK3588 — any plans?

[woodyst]

Hi @NotPunchnox,

I'm trying to run Qwen3.6-35B-A3B (Qwen/Qwen3.6-35B-A3B) on my
Orange Pi 5 Plus (16 GB RAM, RK3588) with rkllama.

The MoE architecture makes it viable memory-wise (~8 GB active RAM),
but I don't have an x86 machine with enough RAM (≥64 GB) to run the
conversion with rkllm-toolkit2 myself.

Do you have any plans to publish a W8A8 converted version for RK3588,
similar to the other models you've shared on HuggingFace?

Thanks for the great work on rkllama!

Edu.

[aruametello]

i might be mistaken, but an original model file of "24 gigabytes" and 35B total parameter already at 4bits per parameter is not very realistic due to the memory constraints

since its "24gb's on disk" at 4bit encoding, you would probably need to encode the parameters to 2bit or "1bit"(???) to make it fit entirely on the 16gb device ram! rk3588 npu's seems to be faster at 8bits (W8A8) per parameter so this conversion may make it slower too. Just converting to W8A8 (4bit->8bit?) would actually make it twice the original size?

i currently run "Qwen3.6-35B 4bit" on a 32gb vram gpu (5090) and it uses about 30gb's of memory at 128k context, would be about 25gb's with a tiny 4096 context.

on the other hand some really crazy stuff has been tried out to "cache and swap" individual experts from fast nvme storage to run MoE models with way less available work memory than the actual whole model. This "could" in theory make your 3B@4bit active parameters MoE work with modest speed on an 16GB edge device. (ssd throughput would be one of the bottlenecks)

Article/github: https://github.com/danveloper/flash-moe

i wouldn’t dare to demand of the maintainers some funky feature like this, but i would like them very very much if they do. 😃

[danielferr85]

Hi @woodyst

You can try another great branch repo: https://github.com/invisiofficial/rk-llama.cpp/tree/rknpu2/ggml/src/ggml-rknpu2

There you can use models like Gemma4 and Qwen3.5, Qwen3.6 without any model conversion.... It uses the gguf format models....

Run on NPU for rk3588....

Pay attention of the owner says:

More info and videos using Gemma4 26B A4B with Q4_0: https://www.reddit.com/r/RockchipNPU/comments/1sc8fir/deploy_the_newest_qwen35_and_gemma4_models_of_any/

[Nurb4000]

well, that is cool i was hoping someone would add RK support to llama.cpp

[danielferr85]

Hi @woodyst @Nurb4000 .... I'm currently working in the integration between rkllama with that fork of llama.cpp.... When a request for a model in GGUF format, rkllama will start a process (worker) with llama-server for inference... Still in development phase but with great results.....So, rkllama will be the main orchestation for inference: if the request is for a model in rkllm format it will continue with the current default behavior, and if the model is GGUF format it will launch the commented workflow.....

Llama.cpp only allows openai format, but with this feature you will also can use Ollama API format.

your main model repository will be rkllama, with rkllm, rknn and gguf formats.

[woodyst]

Hi @danielferr85: Thanks for your info. I am very interested in your progress. My needs are using Hermes Agent with good coder model, but Hermes needs 64K context because it has a lot of context because of the agents. I am trying to modify Hermes Agent sources for using it with the orange pi 5 plus NPU with a 32K context (the context of qwen-2.5-coder in rkllama format). If I success with my changes, I will tell you. If you think I can help you with something, testing, coding, etc, tell me. I work with claude code. Thanks. Edu.

[danielferr85]

Hi @Nurb4000 we are in the same page..... jajajja.. I'm doing this for the same....

With this feature you can use even more large context than the required by hermes (64k)..... I will plan to use gpt-oss-20b , or gemma 4 26b A4 or Qwen 3.6 35B A3. Already tested these 3 models.... (also qwen 3.5, ibm granite 4.0, LFM2...). All GGUF models that works on rk-llama.cpp will work also here....

Don't waste your time touching hermes agent.... Give me this week and I will push the experimental version in my repo... (you can try it).. If everything is ok i will make as always the PR to the main rkllama project.

[danielferr85]

[diegodurrod]

@danielferr85 your feature points to be very interesting, I'm really interested in testing your fork, in this case, I have configured 7 rk3588 nodes in a kubernetes cluster, so we can keep in touch for testing your fork :)

[danielferr85]

Hi @diegodurrod @Nurb4000 @woodyst

If you are going to run the rkllama server locally you have to build the llama-server first. Please go to that repo and follow the steps for that fork. For running GGUF models rkllama will ask for a new argument where the binary llama-server exists (--llama_cpp_dir)

If you are going to use docker, you dont have to do anything. I will include the build of the llama-server inside the same rkllama container.

Very Soon I will release my fork of rkllama..

[Nurb4000]

Hi @diegodurrod @Nurb4000 @woodyst

If you are going to run the rkllama server locally you have to build the llama-server first. Please go to that repo and follow the steps for that fork. For running GGUF models rkllama will ask for a new argument where the binary llama-server exists (--llama_cpp_dir)

If you are going to use docker, you dont have to do anything. I will include the build of the llama-server inside the same rkllama container.

Very Soon I will release my fork of rkllama..

Be a couple of days, pretty busy here but will give it a shot early in the week. Thanks a bunch for your effort.

[danielferr85]

Hi @woodyst @aruametello @diegodurrod @Nurb4000

Tomorrow I will upload the fork....

Tested so far for GGUF models:

• Ollama API call with Stream false and true
• OpenAI API call with Stream false and true
• Calls with tools calls
• LLM with reasoning content....
• Unload (on demand) models from OpenWebUi (also for rklllm models)
• Automatically unload GGUF models workers expired.
• Pass custom RKNPU* enviroment vars ( https://github.com/invisiofficial/rk-llama.cpp/tree/rknpu2/ggml/src/ggml-rknpu2) and Arguments (https://github.com/invisiofficial/rk-llama.cpp/blob/rknpu2/tools/server/README.md) to llama-server per model (throught config.ini file per model in specific format).

TODO (soon):

• Include projector for vision request
• Better error handling
• Clean the code
• Alllow download gguf models from rkllama pull workflow
• Any issues or improvements detected.....

Notes:
rkllama client was not tested.... Only focus to server.

[danielferr85]

Done beta version @woodyst @aruametello @diegodurrod @Nurb4000 ....

Pull request created...