concurrencyLimit: 1 block also the .css requests with 429

[slu-010101]

Hi,

I assumed that a "concurrencyLimit: 1" would allow only one (AI) request per model, but it blocks the .css requests as well. Is that intentional?

V228

Thanks.

Sam

[mostlygeek]

Hi,

The concurrencyLimit is handled at the HTTP level and not at the application level. To limit inference concurrency it is better to configure on the inference side. For llama-server that would be -np, --parallel N flag.

Maybe the docs can be improved for it so it's a bit clearer in the example config 🤔

[slu-010101]

The new in-flight request info is awesome, thank you!

Is it possible to allow only one running request at the same time?
--parallel help only per model, but if there are two request on different (loaded) models, both running at the same time and no one can be finished due to hardware limitations.

Maybe you have an idea.