feat: stream + stop at first sentence boundary past NumPredict (SoftStop) - #157
Open
Saiska wants to merge 1 commit into
Open
feat: stream + stop at first sentence boundary past NumPredict (SoftStop)#157Saiska wants to merge 1 commit into
Saiska wants to merge 1 commit into
Conversation
Bot replies were hard-cut mid-word when generation hit NumPredict. This streams the Ollama response and stops at the first complete-sentence boundary once NumPredict tokens have been generated, so replies end on a finished sentence near the target length. - OllamaHttpClient::PostStreaming: streaming POST via the low-level send() + Request::content_receiver API (portable across httplib versions), aborting cleanly when the receiver returns false (Error::Canceled treated as success). - OllamaStreamAccumulator: reassembles the streamed NDJSON across raw byte chunks, counts response-bearing chunks (~tokens), and stops on tokens >= NumPredict && sentence-boundary, or on done. - QueryOllamaAPI: streaming branch gated by SoftStopEnable && NumPredict>0 && !ThinkMode; sends a hard ceiling of NumPredict+32 so the in-progress sentence can finish; the legacy non-streaming path is preserved in else. - New OllamaChat.SoftStopEnable config key (default on); reuses NumPredict as the soft target (no new length key). - Debug-gated boot self-test (SoftStopSelfTest, 4 cases) for the accumulator. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes the Ollama call stream and stop at the first complete-sentence boundary once
NumPredicttokens have been generated, so bot replies end on a finished sentence instead of being hard-cut mid-word.Why
With a finite
NumPredict, replies get truncated mid-word (e.g."Oh man, I'm"). This stops generation at a sentence boundary near the target length instead.How
OllamaHttpClient::PostStreaming— streaming POST via the low-levelsend()+Request::content_receiverAPI (portable across bundled httplib versions), aborting cleanly when the receiver returnsfalse(Error::Canceledis treated as success).OllamaStreamAccumulator— reassembles the streamed NDJSON across raw byte chunks, countsresponse-bearing chunks (~tokens), and stops ontokens >= NumPredict && <sentence boundary>, or ondone.QueryOllamaAPI— a streaming branch gated bySoftStopEnable && NumPredict > 0 && !ThinkMode; it sends Ollama a hard ceiling ofNumPredict + 32so the in-progress sentence can finish, and preserves the legacy non-streaming path verbatim in theelse.OllamaChat.SoftStopEnableconfig key (default1); reusesNumPredictas the soft target — no new length key.SoftStopSelfTest, 4 cases) validates the accumulator with no network.Think-mode and
NumPredict = 0(unlimited) deliberately skip soft-stop and use the existing non-streaming path. Toggle off → byte-identical legacy behavior.Testing
SoftStop self-test: 4/4 passed(stop-at-period / wait-for-boundary / chunk-split / done-flag).done; no leaked connections.AI assistance
This change was developed with AI assistance (Claude / Claude Code).
🤖 Generated with Claude Code