[Hexagon:Feature] Support Qwen3-VL with symmetric and asymmetric block-wise quantization

[yydhYYDH]

Description

This change adds two main capabilities to the Android Hexagon backend:

• Qwen3-VL Vision model execution;
• symmetric and asymmetric block-wise INT4/INT8 quantization for Vision and language-model weights.

Performance

llm_bench -p N -n 128 -rep 3 on OnePlus 13 / Snapdragon 8 Elite / Hexagon v79, tokens/s:

Weight format	Prefill 128	Prefill 256	Prefill 512	Decode avg
INT4 asymmetric block64	893.07	1257.30	1417.40	18.06
INT8 symmetric block64	631.46	713.35	732.10	20.86
INT8 asymmetric block64	597.42	691.14	730.40	11.57
FP16	350.42	502.68	609.72	4.02

Module

Hexagon

Type

• Feature
• Bugfix
• Perf
• Refact
• Style
• Doc
• Test
• Chore

Checklist

• Commit message follows [Module:Type] Description format
• Code compiles without errors
• Tested on relevant platform(s)
• No unrelated format or style changes included

[wangzhaode]

Thanks for the substantial Hexagon work. The direction and performance results look promising, and the three host-side validation tests build and pass locally.

Before merging, could you please address the following issues:

1. In HexagonConvolution::create(), if reorderInt4WeightForHmx() fails after int4Weight is allocated, the buffer remains non-null and the resource is still marked as a successful INT4 path. Please free/reset it or return/fallback on reorder failure.
2. Vision Flash Attention reserves min(heads, maxThreads) worker workspaces, while the DSP worker pool may pass a physical worker_index up to g_max_num_workers - 1. Please reserve the full worker count, as the existing Attention path does, to avoid overwriting workspace or mask memory when heads < workers.
3. The Vision path accepts non-NC4 formats but unconditionally changes the output format to NCHW. Please preserve the inferred input/output format, or explicitly require NCHW.

The new W8 decode path dynamically quantizes FP16 activations to INT8 and is enabled by default. A lightweight validation is enough: please provide one representative logits/output comparison with MNN_HEXAGON_W8A16_GEMV_I8 enabled and disabled, plus one real Qwen3-VL image smoke test on Hexagon. No extensive benchmark or device matrix is needed.

Thanks!

[yydhYYDH]

Thanks for the detailed review. I addressed all four points in commit 4906a05a.

• INT4 reorder failure now frees and resets int4Weight, so the resource is no longer marked as a valid INT4 path after a failed reorder.
• Vision Flash Attention now reserves workspace for the full DSP worker count, matching the physical worker_index range.
• The Vision Attention path now explicitly requires NCHW for Q/K/V/output and no longer overwrites the inferred output format.
• I added a host regression covering the worker-workspace case, including heads < maxThreads.

logits/output comparison

I also completed the requested lightweight validation on OnePlus 13 / Snapdragon 8 Elite / Hexagon v79 using Qwen3-VL-2B-Instruct-MNN-INT8-SYM-Hexagon.

The text prompt was:

Hello. Reply with one short sentence introducing yourself.

This comparison toggles the W8 decode INT8 GEMV path with MNN_HEXAGON_W8A16_GEMV_I8; the model weights remain INT8 in both runs. No generation length was passed to llm_demo, and both runs terminated naturally at EOS.

W8 INT8 GEMV enabled:

Hello! I'm Qwen, a large-scale language model developed by Alibaba Group. I can answer questions, create text, and perform various tasks. I can help you with a wide range of tasks, including answering questions, writing, logical reasoning, and more. If you have any questions or need assistance, feel free to ask me!

W8 INT8 GEMV disabled with MNN_HEXAGON_W8A16_GEMV_I8=0:

Hello! I'm Qwen, a large-scale language model independently developed by the Tongyi Lab under Alibaba Group. I can assist you with answering questions, creating text such as stories, official documents, emails, scripts, and more, as well as performing logical reasoning, programming, and other tasks. I'm here to help you in any way I can!

Full first-token logits comparison:

count=151936
max_abs_error=3.00000000
mean_abs_error=0.49711962
rmse=0.61183923
cosine=0.9984495367
top10_overlap=9/10

I also ran a longer text-generation check with max_new_tokens=256. The input prompt tokenized to 117 tokens:

Read the following facts carefully. Fact 1: Mercury is closest to the Sun. Fact 2: Venus has a dense carbon dioxide atmosphere. Fact 3: Earth has liquid surface oceans. Fact 4: Mars is called the red planet. Fact 5: Jupiter is the largest planet. Fact 6: Saturn has prominent rings. Fact 7: Uranus rotates on its side. Fact 8: Neptune has very fast winds. In one short sentence, state which planet is largest and which planet is known for prominent rings.

With W8 GEMV enabled:

Based on the given facts:

- **Fact 5**: Jupiter is the largest planet.
- **Fact 6**: Saturn has prominent rings.

So, the largest planet is **Jupiter**, and **Saturn** is known for prominent rings. Therefore, the answer is:

- **Jupiter is the largest planet.**
- **Saturn has prominent rings.**

With W8 GEMV disabled:

Based on the facts provided:

- **Fact 5** states that **Jupiter** is the largest planet.
- **Fact 6** states that **Saturn** has prominent rings.

Therefore, the planet that is largest is **Jupiter**, and the planet known for prominent rings is **Saturn**.

Both runs answered correctly and terminated naturally at EOS before reaching the 256-token limit. Both selected Based as the first token with argmax 28715.

Full first-token logits comparison:

count=151936
max_abs_error=3.57812500
mean_abs_error=0.47298237
rmse=0.58905461
cosine=0.9974614623
top10_overlap=10/10

Qwen3-VL image smoke test

The image is the repository's 480x360 resource/images/cat.jpg.

The Vision prompt was:

<img>/path/to/resource/images/cat.jpg</img>Describe this image in one short sentence.

I ran the same prompt with greedy sampling on the CPU and Hexagon backends using the same INT8 symmetric block-64 model. No generation length was passed to llm_demo; both runs terminated naturally at EOS and returned zero.

CPU backend response:

A small, orange and white kitten with large, curious eyes sits on a stone in a grassy area.

Hexagon backend response:

A small, orange tabby kitten with white fur on its chest and paws sits on a stone, looking directly at the camera with a curious expression.

The three host-side validation tests, Android MNN/LLM build, and v79 skel unexpected-symbol check all pass. The phone logs contain no error, fallback, fatal signal, NaN, or Inf marker.

[wangzhaode]

Thanks for the update—the INT4 failure handling, NCHW validation, and requested lightweight checks look good.

One workspace issue remains on the DSP side. In htp_ops_vision_flash_attention_fp16(), workerSlots is still derived from sync_attention_pick_task_count(totalTasks). When heads < g_max_num_workers, maskOffset is therefore calculated before the full physical worker workspace range, and a worker with a higher worker_index can overlap the converted mask memory.

Please expand workerSlots to at least g_max_num_workers before calculating maskOffset and requiredWorkspaceBytes, matching the existing htp_ops_flash_attn() path below.

This small fix plus the existing build/unit test is sufficient; no additional performance or accuracy matrix is needed. Thanks!