[Feature]: Vulkan support

[ericcurtin]

🚀 The feature, motivation and pitch

I started hacking on vLLM recently for RamaLama and other reasons to see what the hardware support is like compared to llama.cpp. The one big noticeable gap is Vulkan support. It would solve a bunch of problems around vLLM not running great on commodity hardware:

containers/ramalama#1677

Alternatives

Use llama.cpp

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ericcurtin]

#21184

[frzifus]

Vulkan support would be really great!

[ericcurtin]

I think we need some work at the pytorch level first but not an expert

[delphiRo]

Llama.cpp is the best inference engine because of it is working with any hardware!

[ericcurtin]

Llama.cpp is the best inference engine because of it is working with any hardware!

Yes llama.cpp has an advantage here. It is one of the main reasons vLLM should implement Vulkan, opens up vLLM usage to a wide array of hardware.

[AlpinDale]

I have done some basic work on this here: dphnAI/sonar#1416. It's a lot of work to say the least, I don't think it's something that could be done in a timeframe less than a year.

[ericcurtin]

I have done some basic work on this here: aphrodite-engine/aphrodite-engine#1416. It's a lot of work to say the least, I don't think it's something that could be done in a timeframe less than a year.

All I'll say is... Keep going... This is needed

[sp82]

I think we need some work at the pytorch level first but not an expert

can this be possible already?
https://docs.pytorch.org/executorch/main/backends-vulkan.html

[lumina37]

When will this draft be merged? I'm really looking forward to the vulkan backend

[AlpinDale]

I think we need some work at the pytorch level first but not an expert

can this be possible already?
https://docs.pytorch.org/executorch/main/backends-vulkan.html

You're right, pytorch can already load Vulkan shaders and use them. All we need is to write the compute shaders for paged attention, flash attention, layernorm, activations, and rope. The rest torch can handle.

[Kreijstal]

can gemini vibe code this

[ericcurtin]

can gemini vibe code this

Can it help yes, will:

"Hey gemini get the vulkan stuff done"

work? No

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!