The backend server, when called with generate, should support different inference modes.
In addition to the current endpoints, there should be a new endpoint bidirectionalInference. Each JSON object returned in the websocket should adhere to:
{ tokensAnticausal: number[], tokensCausal: number[], done: boolean }.
This would support several new inference modes.
The backend server, when called with generate, should support different inference modes.
In addition to the current endpoints, there should be a new endpoint
bidirectionalInference. Each JSON object returned in the websocket should adhere to:{ tokensAnticausal: number[], tokensCausal: number[], done: boolean }.This would support several new inference modes.