Fix: abort degenerate/repeating generation instead of streaming it to the output cap (#1029) — port of @r0h1tb's #1099

[kaluli123123]

Credit upfront

This fix's detection algorithm, calibration data, and full test suite (including the end-to-end SSE-replay test) are @r0h1tb's, from #1099 (closed, not merged, via the maintainer's standard backlog-sweep note citing #1165 as "covering" this — the underlying gap is still live on current main, confirmed below by porting #1099's own tests and verifying they fail on pre-fix main). This is a port of that PR's diff against current main, not independent (re)work. If a maintainer picks this up, #1099 and @r0h1tb should be the credited source.

Summary

A thinking stream can settle into emitting the same short phrase back to back thousands of times and never stop on its own: one incident measured 79,222 characters of "The the the the the the the " before the user aborted by hand. No layer owns this failure class: provider streamers are faithful assemblers that never inspect content, request construction attaches no repetition penalties, and the agent loop treats a completed stream as progress. A runaway response can burn the whole 32k output cap before anything notices. This is #1029.

Fix

A new RepetitionGuard (packages/ai/src/utils/repetition-guard.ts) runs incrementally over the reasoning channel of openai-completions (the affected provider in the original incident; the text channel is deliberately left unguarded since a long legitimate answer can contain generated tables/code that look repetitive, and a false abort there destroys real work). Two independent detectors, because a loop rarely stays verbatim for long:

• Periodic tail: the tail is one short unit repeated many times, found via the KMP failure function (smallest period of a string in O(n), no scanning of candidate period lengths).
• Novelty stall: distinct word trigrams over total, for loops that drift enough that verbatim periodicity misses them. The 0.20 novelty floor is calibrated against a measured corpus (verbatim/drifting loops score 0.001/0.136; enumerated analysis, markdown tables, JSON, source code, numbered lists, and prose all score 0.28–1.0), not guessed — the calibration table is reproduced in the module docstring. Word-trigram Jaccard between paragraphs was tried first and rejected: it could not separate drifting loops from legitimate structurally-similar paragraphs at any threshold.

On a hit, streamOpenAICompletions throws a StreamFailureError classified as a new "degenerate_output" StreamFailureKind, which the session's existing auto-retry path treats as retryable — the session re-samples instead of committing the garbage to the transcript. Kill switch: PRIME_AGENT_NO_REPETITION_GUARD=1.

A second, separate gap closed along the way (also part of #1099's diff): openai-completions.ts was the only major provider streamer that never called recordStreamFailure — every other provider already does. Failures on this path (OpenRouter, llama.cpp, any OpenAI-compatible endpoint) reached the session with no classification at all; the new degenerate_output kind depends on this wiring anyway, and fixing it also fixes classification for every existing failure kind on that provider.

Tests (ported verbatim from #1099)

• repetition-guard.test.ts (21 tests): unit coverage for both detectors, plus an extensive false-positive corpus (source code, markdown tables, numbered lists, JSON, prose, enumerated analysis, whitespace padding, short sub-threshold stutters) that the guard must NOT fire on.
• openai-completions-repetition-guard.test.ts (4 tests): end-to-end against the real streamOpenAICompletions, driven by a local SSE server replaying the actual incident (2,366 reasoning deltas of the loop unit) — confirms the stream terminates as an error, classifies as degenerate_output, and stops after ~2,072 characters against the 79,222 the incident produced; and that the kill switch restores full pass-through.

Verified both test files fail against pre-fix main (repetition-guard.ts doesn't exist; the e2e test fails to even import) and pass after the port.

Validation

cd packages/ai && npx vitest --run test/repetition-guard.test.ts test/openai-completions-repetition-guard.test.ts \
  test/openai-completions-cache-control-format.test.ts test/openai-completions-empty-tools.test.ts \
  test/openai-completions-prompt-cache.test.ts test/openai-completions-reasoning-replay.test.ts \
  test/openai-completions-response-model.test.ts test/openai-completions-thinking-as-text.test.ts \
  test/openai-completions-tool-choice.test.ts test/openai-completions-tool-result-images.test.ts \
  test/stream-failure.test.ts
# 113 passed, 0 failed.

npx tsgo -p tsconfig.json --noEmit and the root npm run check (biome, tsgo, installer render, browser smoke) both pass. The only manual merge needed against current main was one relocated hunk in openai-completions.ts (context drift, no logic conflict). Added a packages/ai/.changes/ fragment, credited to #1099/@r0h1tb, per the changelog-fragment CI check.

Patch

Branch: https://github.com/kaluli123123/prime-agent/tree/fix/repetition-guard-degenerate-output

Diff: main...kaluli123123:prime-agent:fix/repetition-guard-degenerate-output