Add CPU build of vLLM to tox CI and local test environments

[frreiss]

It would be useful to have a CPU build of vLLM for CI and local testing. Unfortunately there are no pre-built wheels of vLLM with CPU inference support. Building a wheel requires compiling vLLM from source. vLLM has two slightly different sets of instructions for doing so here and here.

The instructions seem pretty straightforward, but someone needs to automate the creation and installation of these packages. Ideally there should be some caching of the packages, as the compilation step probably takes a while.

[rockenman1234]

@frreiss I can take this one!

It will probably be some time until I can get to it though, probably closer to the end of the July - is there anything else you'd like to see in the PR not currently listed? (Features, design cues, etc)

Thanks :)

[frreiss]

The main thing is to find a specific point release of vLLM where the following three things work at the same time:

• CPU inference
• LoRA adapters
• Constrained decoding with regular expressions
  Then pin that release in place.

The notebook notebooks/rag.ipynb is a good guinea pig for testing compatibility.

[rockenman1234]

Hey @frreiss - I hope you’re doing well!

I haven’t forgotten about this and have been continuing to work on it, but it turned out to be a bigger challenge than I expected. The main issue is that vLLM hasn’t been very stable on my MacBook - almost every attempt I’ve made has caused my system to lock up.

I’ll keep pushing forward (this time on my x86 machine), but it’s going to take me a bit longer than I originally planned.