concurrencyLimit: 1 block also the .css requests with 429

[slu-010101]

Hi,

I assumed that a "concurrencyLimit: 1" would allow only one (AI) request per model, but it blocks the .css requests as well. Is that intentional?

V228

Thanks.

Sam

[mostlygeek]

Hi,

The concurrencyLimit is handled at the HTTP level and not at the application level. To limit inference concurrency it is better to configure on the inference side. For llama-server that would be -np, --parallel N flag.

Maybe the docs can be improved for it so it's a bit clearer in the example config 🤔