Description
I have problems when using BLS to communicate with the VLLM backend. The python BLS model (which calls a VLLM model)
itself is async but decoupled=False and also make an async_exec(decoupled=True) call to the VLLM model.
This does not work for reasons I can not comprehend completely, but it has to do with the asyncio backend which gives me an "invalid argument" error when i try to await the requests.
I found this hint in the python backend:
https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/bls.html#limitation
For me this mean calling an decoupled python model with async_exec will not work, but this would mean your default implementation of vllm would not be usable with BLS calls.
But then I found also this:
https://github.com/triton-inference-server/python_backend#decoupled-mode
Is this limitation of BLS still valid?
Triton Information
26.06 NGC Container
Are you using the Triton container or did you build it yourself?
I use a NGC container.
Description
I have problems when using BLS to communicate with the VLLM backend. The python BLS model (which calls a VLLM model)
itself is async but decoupled=False and also make an async_exec(decoupled=True) call to the VLLM model.
This does not work for reasons I can not comprehend completely, but it has to do with the asyncio backend which gives me an "invalid argument" error when i try to await the requests.
I found this hint in the python backend:
https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/bls.html#limitation
For me this mean calling an decoupled python model with async_exec will not work, but this would mean your default implementation of vllm would not be usable with BLS calls.
But then I found also this:
https://github.com/triton-inference-server/python_backend#decoupled-mode
Is this limitation of BLS still valid?
Triton Information
26.06 NGC Container
Are you using the Triton container or did you build it yourself?
I use a NGC container.