You should be able to reply to generated text to insert new text into a generation. This should have an optional prefix for chat mode. It should keep a few sessions in memory in a LRU cache. Sessions not in memory are replayed with a progress indicator.
You should be able to reply to generated text to insert new text into a generation. This should have an optional prefix for chat mode.
It should keep a few sessions in memory in a LRU cache. Sessions not in memory are replayed with a progress indicator.