Qwen/Qwen3.6-35B-A3B BF16 inference is failing on Intel Arc Pro B70 cards

[mayavijx]

Getting below error while inferencing the Qwen/Qwen3.6-35B-A3B model:
RuntimeError: expected self and mat2 to have the same dtype, but got: c10::Half != c10::BFloat16

Steps to reproduce:
docker pull intel/llm-scaler-vllm:0.14.0-b8.3.1 (As recommended in the readme. Got the same issue with latest image as well)

docker run -td --privileged --net=host --device=/dev/dri --name=cont_llm_scaler -v Qwen3.6-35B-A3B/:/llm/models/Qwen3.6-35B-A3B/ --shm-size="32g" --entrypoint /bin/bash intel/llm-scaler-vllm:0.14.0-b8.3.1

docker exec -it cont_llm_scaler bash
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export ZE_AFFINITY_MASK=0,1,2,3

vllm serve --port 8000 --host 0.0.0.0 --gpu-memory-util 0.9 --max-num-batched-tokens 8192 --max-model-len 40000 --block-size 64 --dtype bfloat16 --model /llm/models/Qwen3.6-35B-A3B/ --served-model-name Qwen3.6-35B-A3B --tensor-parallel-size 4 --enforce-eager --trust-remote-code --disable-log-requests

Application started successful but failed for the below curl request:
curl -X POST "http://localhost:8000/v1/chat/completions" -H "Content-Type: application/json" --data '{
"model": "Qwen3.6-35B-A3B",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe this image in one sentence."
},
{
"type": "image_url",
"image_url": {
"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
}
}
]
}
],
"max_tokens": 256
}'

Please find the detailed log attached

qwen_3.6_35b_a3b_bf16_error.log

.

[Wesley-Du]

Hi @mayavijx currently the supported dypte is fp16 (--dtype float16 )

[azhuvath]

@Wesley-Du This might be related to vllm-project/vllm#41663 which is happening in vllm also. I tried both bfloat16 and float16 through llm-scaler. The issue happens for both under load.

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
VLLM_WORKER_MULTIPROC_METHOD=spawn
vllm serve
--model Qwen/Qwen3.6-35B-A3B
--served-model-name Qwen/Qwen3.6-35B-A3B
--dtype=float16
--enforce-eager
--port 8000
--host 0.0.0.0
--trust-remote-code
--gpu-memory-util=0.9
-tp=4
2>&1 | tee /llm/vllm.log > /proc/1/fd/1 &