Support for token‑streaming in chat responses

[Bastian-S]

The current version of the moodle-block_ai_chat plugin (or the underlying moodle-local_ai_manager) appears to return chat responses only after the full message has been generated. Token‑streaming (incremental output while the model is generating) does not seem to be supported.

For my use case, streaming would significantly improve the user experience:

• It reduces perceived waiting time
• It makes interactions feel more natural and conversational
• It aligns with how most LLM-APIs typically behave

Is token‑streaming currently supported or planned for future releases of the plugin?
Are there technical limitations in the plugin or backend that prevent streaming?
Would it be possible to add an option for streamed responses?

Thank you very much for your work on the plugin and for your help!

[PhMemmel]

Hi @Bastian-S ,
thank you for raising this issue.

This definitively is a valid feature request, especially because it would solve the issue of timeouting requests in case of models that are taking a lot of time to respond or if your request contains a lot of data in which case the processing even with fast models takes a lot of time. Also, as you mentioned, this would of course be an UX improvement and align with the expectation you have as user knowing how chatgpt etc. work.

However, implementing streaming is a lot to think of and technically pretty tough to implement. There would have to be some polling mechanism, some storing of the parts of the request and the constant rerendering of the message in the frontend. Also, there currently are already problems converting the LLM output to the proper HTML output in the frontend, receiving only chunks and try to render them, also from a security perspective, is definitely not an easy task, see https://developer.chrome.com/docs/ai/render-llm-responses for example.

Additionally, the local_ai_manager acts as a proxy between the external LLM API and the frontend chatbot. Passing through the stream is not an option, so you have to deal with it in the PHP layer: add another chunk, store it somewhere temporarily, let the frontend poll and return again the next chunk. Applying Moodle formatting via format_text to chunks probably will cause a lot of trouble as well, not even thinking about applying filters like mathjax etc. and the special frontend rendering in agent mode.

So overall: I really would also have this feature, but the technical effort is huge and probably somehow unreliable. If someone has capabilities to do a probe of concept, we're of course open to discuss, but in the near future we probably won't be able to focus on this issue.