Repository navigation
comparison with qualcomm ai hub model #7411
Description
🐛 Describe the bug
I ran Llama-v3.2-3B-Chat(precision w4a16) from ai-hub-model on a Snapdragon 8 Gen 3 device, achieving 20 tokens/s.
For comparison, I ran inference for the Llama3.2-3B model quantized to W4A16 using executorch with the QNN backend on the same device. The performance I observed was 10 tokens/s.
Could you provide insights into what might be causing this performance difference? Are there issues with how executorch handles quantized models that could explain this gap?
Any guidance or suggestions would be greatly appreciated!
Activity
AndreaChiChengdu commentedon Dec 27, 2024
good question, i found the same issue on llama 3.1 8b via executorch qnn backend，which performance is below 1/2 compare with qualcomm qaihub chaimed
my device is xiaomi14pro with sm8650（v75 npu） 16GB ram
kimishpatel commentedon Jan 3, 2025
cc: @cccclai
cccclai commentedon Jan 3, 2025
Yeah we found out the model definition in
llama_transformer.pyisn't ideal for running llama model on qnn backend. We've started a new model definition in https://github.com/pytorch/executorch/tree/e66cdaf514e15242692073db1271aae4657f2033/examples/qualcomm/oss_scripts/llama3_2 which have better latency numberIt's still wip and please expect some burden if trying them out, or maybe wait till it's more settled.
Jack-Khuu commentedon Feb 4, 2025
Converting to discussion