Limit to one request at the same time?

[slu-010101]

Hi,

I have loaded multiple models at the same time and can only use the CPU.
This working fine as long there is a single request for one model, but as soon
another request came in at the same time both of them are very slow.

Is there any option to limit llama-swap to one request at the same time and
all models?

Sam

[mostlygeek]

llama-swap keeps a request queue per model so there isn't a global queue to limit
requests to one at a time.

You could do a few things:

• avoid using groups or matrix. this will force a swap and result in one request queue active at a time
• set --parallel to 1 if using llama-server
• set concurrencyLimit 1 on a model config to send 429 responses when it's busy

[slu-010101]

@mostlygeek thank you for the feedback.

Unfortunately I need groups to preload all my models on startup, the system has enough RAM.
Is there an other way to preload the models at the same time? Without groups the first model will be unloaded as soon the second model load...