llama-swap with multiple inference engines

[d-shehu]

I've been using llama-swap with llama-server (sans router) for a while without issues. As llama-server now has the router mode which allows model switching, I wanted to repurpose llama-swap as a higher level proxy to manage model switching and loading across inference engine.

For example, on my Mac, I would like to be able to switch between mlx server, llama.cpp and ollama. Ideally this would extend to vllm in the future.

Anyways, I'm running into an issue where I just don't see how llama-swap is supposed to handle model swapping when the "checkEndpoint" is a health check and not a "is model loaded". I started to work around this issue using Caddy and passing through args but this seemed hackish.

Use Case:

Given finite memory and a matrix config which indicates which sets of models can be loaded in parallel how is the config supposed to set? I'm NOT using Docker on my Mac, for obvious reasons, and I prefer to use the HTTP API to load/unload models with llama.cpp, ollama for example. The inference engine start up when the user logs in via plist without loading models.

Or is this unsupported and llama-swap must be able to explicitly control each inference engine? If so that seems like a limitation since it's more efficient to keep the engine running especially if I keep certain small models in memory (embeddings, ocr) while swapping between larger models.

Thanks in advance.

[mohammadrezwankhan]

[d-shehu]

Yes, I get that. The adapter route is where I was going. I used Caddy to avoid writing throwaway code as I'm not interested in building a proxy. If I did that there'd be little use for llama-swap as I can do my own algorithm for deciding which models to load/unload ...

But there is an issue with the readiness check ... possibly a bug? If you connect OpenWebUI with Llama-swap and llama-server in router mode, llama-swap does not wait for llama-server to actually load the model.

openwebui will error out until the model loads. Ollama on the other hand blocks requests until the model has loaded. This is independent of lifecycle of the engine.

What I was suggesting is there is a gap in llama-swap's usage given that many inference engine have already implemented their own load/unload mechanism/ web UI, etc. And I think the greater value-add is in being able to wrap those engines, gateway-like (i.e. LiteLLM that is actually lite).

What I'm gathering is that without the adapter approach llama-swap simply isn't flexible enough to handle swapping. Thanks for your quick reply.

[mostlygeek]

llama-swap wasn't design for the use case you described where it will call the other server's (llama-server, ollama, etc) proprietary APIs to control the state of their resources. It is designed to reliably swap servers on demand with proper queuing of requests, etc. You may be able to leverage a wrapper script like cmd/vllm-wrapper that makes use of cmd and cmdStop to send API calls to the other servers.

I try to keep things as simple as possible. I use llama-server in standalone mode for gguf models, comfui and vllm in docker containers, and audio.cpp/whisper.cpp for audio things. I don't run other servers because it's harder to manage the memory resource and keep a synchronised and consistent state.