Skip to content

feat(llm): enforce terminal stream contract for every route#36693

Open
rekram1-node wants to merge 2 commits into
llm-remove-provider-errorfrom
llm-terminal-contract
Open

feat(llm): enforce terminal stream contract for every route#36693
rekram1-node wants to merge 2 commits into
llm-remove-provider-errorfrom
llm-terminal-contract

Conversation

@rekram1-node

Copy link
Copy Markdown
Collaborator

Stacked 3/3 on #36692 (which stacks on #36691).

Summary

LLMClient.stream now guarantees, for native protocol routes and the synthetic AI SDK route alike:

  • A successful stream emits exactly one terminal finish event and nothing after it.
  • EOF before finish means the provider stream was truncated (proxy cut, silent drop) and fails as LLM.MalformedResponse instead of letting a partial response settle as complete.
  • Output after finish fails the stream.

Enforcement applies after protocol parsing so stream.onHalt flushes remain subject to it, with per-subscription state.

Previously only generate detected missing terminals (via response assembly); stream consumers could observe silent truncation as a clean end.

Testing

LLMClient.stream now guarantees, for native protocol routes and the
synthetic AI SDK route alike: a successful stream emits exactly one
terminal finish event and nothing after it. EOF before finish means the
provider stream was truncated (proxy cut, silent drop) and fails as
LLM.MalformedResponse instead of letting a partial response settle as
complete; output after finish also fails. Enforcement applies after
protocol parsing so stream.onHalt flushes remain subject to it, with
per-subscription state.
@rekram1-node rekram1-node force-pushed the llm-remove-provider-error branch from 3cf531b to 5d87c7a Compare July 13, 2026 17:25
@rekram1-node rekram1-node force-pushed the llm-terminal-contract branch from 7a49d31 to 49e86dd Compare July 13, 2026 17:25
@kitlangton

Copy link
Copy Markdown
Contributor

Heads up: #36881 just merged to v2 with essentially the same terminal guard, since this was biting production today (a truncated write tool-input stream settled as execution.succeeded, details in #36766).

Two deltas from this PR's version, both driven by it landing on current v2 rather than on the provider-error removal stack:

  • The guard accepts finish or provider-error as terminal. Once refactor(llm): remove the provider-error stream event #36692 lands, that predicate collapses to finish-only.
  • Enforcement lives inside streamPrepared in makeFromTransport rather than in streamRequestWith, so direct route.streamPrepared consumers are covered too, and it reuses the route label already computed there. There is also a provider-neutral regression in test/adapter.test.ts.

So this PR should rebase down to roughly the predicate simplification plus whatever the flat error union renames. Sorry for the churn, and happy to help with the rebase if useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants