From 2f8dda0b4251810eba9ebe115fc29154d79bfe7b Mon Sep 17 00:00:00 2001 From: TechNomadCode Date: Mon, 2 Nov 2026 23:39:13 +0100 Subject: [PATCH] Ask the model again when its reply holds its own markup, then remove it Co-Authored-By: Claude Opus 5.5 --- docs/plan.md | 1 + src/server/AGENTS.md | 3 +- src/server/ai/cloudflare.ts | 76 ++++++++++ tests/unit/cloudflare.test.ts | 255 +++++++++++++++++++++++++++++++++- 4 files changed, 333 insertions(+), 2 deletions(-) diff --git a/docs/plan.md b/docs/plan.md index 4629a81..e076a84 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -12,6 +12,7 @@ 1. `feat/support-check` stays for a later try. 2. Tuning resumes only on the owner's word, from what staging shows. 3. Staging, 2026-10-26: after one search, "Add grocery delivery to the PRD." got a reply whose whole text was a JSON object with a `question` field, shown as is, nothing saved; one run, an observation (`docs/records/staging-2026-10-26-grocery-json.md`, local). The owner chose the display fix only: such a reply now shows the sentence inside (`unwrappingJsonReplies`, `src/server/AGENTS.md`), on `main` since 42fcca6; `verify` passed locally (600 unit tests); unverified on staging until such a reply comes again. The plain wording is measured only on their word. + 4. 2026-11-02, seen by the owner on "Draft the first section.": replies whose text held the model's own markup, three times, none in the 141 records before. The owner chose the display fix and one automatic retry, on `fix/hide-model-markup` (`retryingModelMarkup`, `src/server/AGENTS.md`); unverified on the real model until such a reply comes again, and the two warnings in the logs show how often it happens and how often the retry helps. No cause is isolated; temperature (Cloudflare's default, 0.6) is unmeasured, and measured only on their word. - **Lean notes** (the owner, 2026-10-22; this lane only): this section, the measurement's parts of `src/server/AGENTS.md` and the lane's decisions hold only what a later session needs to test, fix or decide: the current state, the lessons that still apply, the next steps. Replace what's outdated instead of adding to it; one line of evidence at most, with the details in `docs/records/` (local). - **Each batch:** `--plan` first, then real runs only, never a probe; a refused request costs nothing, and the planned batch is sent again on the owner's word or after 00:00 UTC. The agent decides the batches within the day (`docs/decisions.md`, 2026-10-21). Plan updates go on a documents-only branch from `main`. - **Before each batch after a merge of the build lane:** read that merge's diff of `src/server` and `src/lib`. If it changed what the model receives (the prompt, the tools' descriptions and schemas, the generation settings, how messages and earlier answers reach the model, the search) or the transport (`src/server/ai/cloudflare.ts`, which slice 8a may change), stop and tell the owner before sending anything. The 9d comparisons list such merges as code differences; read the file list before trusting that warning. diff --git a/src/server/AGENTS.md b/src/server/AGENTS.md index c84cdb2..18a7517 100644 --- a/src/server/AGENTS.md +++ b/src/server/AGENTS.md @@ -73,7 +73,7 @@ What shapes the model (2026-10-22): | Searches and saves per answer | `chat.ts`: `MAX_SEARCHES`, `MAX_SAVE_TRIES`, `MAX_STEPS` | three searches, each result saying how many are left; two tries to save | | How earlier answers look to the model | `chat.ts`: `forEarlierAnswers` | searches cut to passage ids and interviewees; saved sections whole; reasoning left out | | Its reasoning between tool calls in one answer | the AI SDK, `@ai-sdk/openai-compatible`, `ai/cloudflare.ts` | passed back as `reasoning`; whether Cloudflare reads it shows in the records, see below | -| The transport | `ai/cloudflare.ts` | whole replies; empty text instead of `null`; reasoning sent as `reasoning` and tool results named, as Cloudflare's provider does (since 2026-10-22); no retry at the quota, retry waits of at most 5 seconds (slice 8a); a reply of one repeated character refused (since 2026-10-22); a reply that is only a JSON object of one text field becomes that text (since 2026-10-26) | +| The transport | `ai/cloudflare.ts` | whole replies; empty text instead of `null`; reasoning sent as `reasoning` and tool results named, as Cloudflare's provider does (since 2026-10-22); no retry at the quota, retry waits of at most 5 seconds (slice 8a); a reply of one repeated character refused (since 2026-10-22); a reply that is only a JSON object of one text field becomes that text (since 2026-10-26); a reply's text holding the model's markup is asked for once more, then removed (since 2026-11-02) | | The search | `search.ts` and the embedder | fake word overlap here, bge-large-en-v1.5 on staging; the five closest (the reranker is off since 2026-10-22, *The reranker*) | | The app's own messages to the model | the check-in (`src/features/AGENTS.md`) and `checkReport` | fixed texts | | The visitor's wording | the visitor | free | @@ -145,6 +145,7 @@ The tool that measures the model's behaviour (`docs/plan.md`, slices 9a to 9c; d - **The quota and other 429s** (slice 8a): `refusingQuota` wraps `fetch` for chat and embeddings alike, under whatever `fetch` the settings pass (the measurement's meter and recorder see every reply first). A 429 with code 3036 or 4006 (`QUOTA_CODES`; 4006 is what `/ai/v1` sent on 2026-10-20) throws `QuotaExceededError` from `fetch`, which `handleFetchError` in `@ai-sdk/provider-utils` passes through and the SDK doesn't retry: nothing works before 00:00 UTC. Any other 429 (the per-minute limit, code 3040 out of capacity) goes on to the SDK's two retries, but a `retry-after-ms` or `retry-after` asking for more than `MAX_RETRY_WAIT_MS` (5 seconds) is dropped first, so the SDK's own 2 and 4 seconds apply instead of up to 60 each, inside the route's 60 seconds (`getRetryDelayInMs` in `ai/dist/index.js`, read 2026-10-20). The request bodies are untouched. The quota codes are also in `measurement/budget.ts` (`QUOTA_CODES`); keep the two lists the same. Replayed replies in `tests/unit/cloudflare.test.ts`; the 4006 one is Cloudflare's own of 2026-10-20. - **Degenerate replies** (`refusingDegenerateReplies`, 2026-10-22): Cloudflare has answered with a run of "!" up to the output limit, with no reasoning and no tool call (2026-10-21 and 2026-10-22). A reply with no tool call, finish reason `length` and text at least 95% one character throws `DegenerateReplyError`, so the visitor gets the generic error and "Try again", never the text; the AI SDK doesn't retry it, and there's no retry of our own. The `wrapGenerate` middleware comes after `simulateStreamingMiddleware` in the array, since the last one is wrapped directly around the model and so sees each whole reply before it becomes a stream (the installed `LanguageModelMiddleware` type in `node_modules/ai/dist/index.d.ts` and `wrapLanguageModel` in `dist/index.js`; https://ai-sdk.dev/docs/reference/ai-sdk-core/wrap-language-model, read 2026-10-22 through Context7). The recorded replies are replayed in `tests/unit/cloudflare.test.ts`; the measurement counts such an answer "not run". - **Replies written as JSON** (`unwrappingJsonReplies`, 2026-10-26): on staging, gpt-oss-120b's whole reply after a search was `{"question":"…"}`, shown to the visitor as is. A reply with no tool call whose whole text is a JSON object of one non-empty text field, bare or in a code fence, becomes that text; anything else, a save written as text included, passes as it came. A `wrapGenerate` middleware next to `refusingDegenerateReplies`, for the same reason; the reply is replayed in `tests/unit/cloudflare.test.ts`. It changes what earlier answers look like to the model too, since the browser sends the unwrapped text back. +- **Replies holding the model's markup** (`retryingModelMarkup`, 2026-11-02): the owner saw gpt-oss-120b write its control tokens (https://github.com/openai/harmony/blob/main/docs/format.md, "Special Tokens") as the reply's text, three times on "Draft the first section.": a search after a save, its reasoning and made-up searches, and a search in an answer's first call, so nothing ran or saved. None of the 141 records before it held such a text. The same request goes once more, at most once per model call, so a second call's cost and time only when it happens; the reply carries both calls' tokens, and the meter counts both HTTP requests while a record's model call shows only the second reply. If the second reply holds markup too, its whole text goes, since the words before the first token can be reasoning; a tool call in the same reply stays. The chat then treats it as a reply without words (*Failures*, above). Two warnings without the text show in the logs how often it happens and how often the retry helps. No cause is isolated, and whether the retry usually works is unmeasured. The replies are replayed in `tests/unit/cloudflare.test.ts`. - **`includeUsage: true`** makes streamed replies report tokens; kept for when streaming returns. - **No token count for embeddings:** on staging (2026-10-17) Cloudflare's `/embeddings` reply had no `usage`, and the AI SDK then reports `NaN` tokens. `cloudflare-embedder.ts` logs "Cloudflare reported no token count" instead. - **Embedding batches:** Cloudflare's bge models take at most 100 texts a call; the package would send 2,048, so `cloudflare-embedder.ts` splits them. diff --git a/src/server/ai/cloudflare.ts b/src/server/ai/cloudflare.ts index 167b375..0f260ad 100644 --- a/src/server/ai/cloudflare.ts +++ b/src/server/ai/cloudflare.ts @@ -296,6 +296,81 @@ const unwrappingJsonReplies: LanguageModelMiddleware = { }, }; +/** + * gpt-oss's own control tokens + * (https://github.com/openai/harmony/blob/main/docs/format.md, "Special Tokens"). + */ +const MODEL_MARKUP = + /<[|](?:start|end|message|channel|constrain|return|call)[|]>/; + +/** Whether a reply's text holds the model's markup. */ +function holdsMarkup(content: readonly { type: string; text?: string }[]) { + return content.some( + (part) => part.type === "text" && MODEL_MARKUP.test(part.text ?? ""), + ); +} + +/** Two calls' count of one kind of token; undefined if neither reported it. */ +function addedTokens(first: number | undefined, second: number | undefined) { + if (first === undefined && second === undefined) return undefined; + return (first ?? 0) + (second ?? 0); +} + +/** + * gpt-oss can write its own markup as the reply's text: a search after a save, + * its reasoning and made-up searches, or a search in an answer's first call, + * so nothing runs (seen by the owner, 2026-11-02; replayed in + * `tests/unit/cloudflare.test.ts`). The same request goes once more, and the + * reply carries both calls' tokens. If the second reply holds markup too, its + * whole text goes, since what comes before the first token can be reasoning. + * The chat then treats it as a reply without words: a saved section stands + * alone, and otherwise the visitor gets the usual error and "Try again". + */ +const retryingModelMarkup: LanguageModelMiddleware = { + specificationVersion: "v4", + async wrapGenerate({ doGenerate }) { + const first = await doGenerate(); + if (!holdsMarkup(first.content)) return first; + console.warn( + "The chat model wrote its own markup into a reply; asking it again.", + ); + const second = await doGenerate(); + const { inputTokens: input, outputTokens: output } = first.usage; + const usage = { + ...second.usage, + inputTokens: { + total: addedTokens(input.total, second.usage.inputTokens.total), + noCache: addedTokens(input.noCache, second.usage.inputTokens.noCache), + cacheRead: addedTokens( + input.cacheRead, + second.usage.inputTokens.cacheRead, + ), + cacheWrite: addedTokens( + input.cacheWrite, + second.usage.inputTokens.cacheWrite, + ), + }, + outputTokens: { + total: addedTokens(output.total, second.usage.outputTokens.total), + text: addedTokens(output.text, second.usage.outputTokens.text), + reasoning: addedTokens( + output.reasoning, + second.usage.outputTokens.reasoning, + ), + }, + }; + if (!holdsMarkup(second.content)) return { ...second, usage }; + console.warn( + "The chat model wrote its own markup again; its text was removed.", + ); + return { + ...second, + usage, + content: second.content.filter((part) => part.type !== "text"), + }; + }, +}; + /** * The chat model, asked for whole replies. Streamed, Cloudflare sends * gpt-oss's tool calls as reasoning text without the tool's name and ends @@ -314,6 +389,7 @@ export function createCloudflareChatModel( middleware: [ simulateStreamingMiddleware(), unwrappingJsonReplies, + retryingModelMarkup, refusingDegenerateReplies, ], }); diff --git a/tests/unit/cloudflare.test.ts b/tests/unit/cloudflare.test.ts index 1b6343c..747a888 100644 --- a/tests/unit/cloudflare.test.ts +++ b/tests/unit/cloudflare.test.ts @@ -1,5 +1,5 @@ import { APICallError, isStepCount, streamText, tool } from "ai"; -import { afterEach, describe, expect, it, vi } from "vitest"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { z } from "zod"; import { DEMO_BUSY, QUOTA_USED_UP } from "@/lib/chat"; import { @@ -829,6 +829,259 @@ describe("the chat when Cloudflare's reply is a bare JSON object", () => { }); }); +// gpt-oss-120b, seen by the owner (2026-11-02), asked "Draft the first +// section.": after the save, with no tools on offer, the reply's whole text +// was this, a search written out in the model's own markup. +const LEAKED_SEARCH = + '<|start|>assistant to=functions searchNotescommentary JSON{" query":" Daan children shared list remove what they add without a big discussion", "topn":5}<|call|>'; + +// The same day, the same request sent again: the model's reasoning and made-up +// searches as the reply's text, typed from the owner's screenshot. No search +// ran and nothing was saved. +const LEAKED_REASONING = `analysisWe need to draft Introduction/ Overview based on interviews with six households. we must search for relevant passages about overall description of shared grocery list app, motivations etc. We have three searches before we must draft section. + +We need to search for general statements: maybe " households want a shared grocery list that everyone can add items to" etc. + +Let's search.<|end|><|start|>assistantcommentary{ +{ +" query": " shared grocery list households want everyone to add items" +} +}<|end|><|start|>assistantanalysisWe got results with passages ids and interviewee names. Let's see.<|end|><|start|>assistantcommentary{ +}<|end|><|start|>assistantanalysis we need to actually execute searchNotes. Let's call function.<|end|><|start|>assistantcommentary{ +{ +"searchNotes": { +" query": " shared grocery list households want everyone to add items" +} +} +}<|end|><|start|>assistantanalysisWe need to wait for response.<|end|><|start|>assistantcommentary{ +}<|end|><|start|>assistantanalysis likely returned something but not displayed due we need to capture output. Let's try again with simpler query: " grocery list app household shared list"<|end|><|start|>assistantcommentary{ +{ +"search Notes": { +" query": " grocery list app household shared list" +} +} +}<|end|><|start|>assistantanalysisOops function name is searchNotes not search Notes. Let's correct.<|end|><|start|>assistantcommentary{ +{ +"searchNotes": { +passage: { query: " grocery list app household shared list" } +} +} +}<|call|>`; + +// The same day, the same request in the first call of an answer: a search +// written out as text, typed from the owner's screenshot. Nothing ran. +const LEAKED_CALL = `analysis<|end|>>We need to draft Introduction / Overview section based on interviews with six households. we must search for relevant passages that describe overall context of shared grocery list app. Let's search notes for " shared grocery list households overview purpose " etc.<|end|><|start|>assistantcommentary to=functions.searchNotes {" query": " shared grocery list app household interview overview purpose" }<|call|>`; + +describe("the chat when Cloudflare's reply holds the model's markup", () => { + beforeEach(() => { + vi.spyOn(console, "info").mockImplementation(() => {}); + vi.spyOn(console, "warn").mockImplementation(() => {}); + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + const save = chatReply( + toolCallMessage("call-1", "saveSection", { + section: "Introduction / Overview", + statements: [ + { + text: "Daan's household keeps a paper list on the fridge.", + citations: [ + { source: "daan-1", quote: "It's a notepad with a magnet." }, + ], + }, + ], + }), + "tool_calls", + ); + const leakedSearch = chatReply( + { role: "assistant", content: LEAKED_SEARCH }, + "stop", + ); + + it("asks again once, and shows the second reply's words", async () => { + const { sent, model } = replaying([ + save, + leakedSearch, + chatReply( + { role: "assistant", content: "Does this capture it?" }, + "stop", + ), + ]); + + expect(await answerTo(model)).toEqual({ + text: "Does this capture it?", + errors: [], + }); + expect(sent).toHaveLength(3); + expect(sent[2]?.body).toEqual(sent[1]?.body); + }); + + it("carries on with the search the second reply asks for", async () => { + const { sent, model } = replaying([ + chatReply({ role: "assistant", content: LEAKED_CALL }, "stop"), + chatReply( + toolCallMessage("call-1", "searchNotes", { query: "the paper list" }), + "tool_calls", + ), + chatReply({ role: "assistant", content: "Nobody asked for it." }, "stop"), + ]); + + expect(await answerTo(model)).toEqual({ + text: "Nobody asked for it.", + errors: [], + }); + expect(sent).toHaveLength(3); + }); + + it("ends on the saved section when the second reply holds markup too", async () => { + const { sent, model } = replaying([save, leakedSearch, leakedSearch]); + + expect(await answerTo(model)).toEqual({ text: "", errors: [] }); + expect(sent).toHaveLength(3); + }); + + it("shows the usual error, after one retry, when nothing was saved", async () => { + vi.spyOn(console, "error").mockImplementation(() => {}); + const { sent, model } = chatModel(leakedSearch); + + expect(await answerTo(model)).toEqual({ + text: "", + errors: ["The assistant ended without an answer."], + }); + expect(sent).toHaveLength(2); + }); + + it("shows the quota message when the retry finds the allocation used up", async () => { + vi.spyOn(console, "error").mockImplementation(() => {}); + const { sent, model } = replaying([leakedSearch, quota4006]); + + expect(await answerTo(model)).toEqual({ + text: "", + errors: [QUOTA_USED_UP], + }); + expect(sent).toHaveLength(2); + }); + + it("counts the tokens of both calls", async () => { + const { model } = replaying([ + leakedSearch, + chatReply({ role: "assistant", content: "Hello" }, "stop"), + ]); + + const result = streamText({ model, prompt: "Hi" }); + + expect(await result.text).toBe("Hello"); + expect(await result.usage).toMatchObject({ + inputTokens: 688, + outputTokens: 60, + totalTokens: 748, + }); + }); + + it("doesn't ask again after a reply without markup", async () => { + const { sent, model } = chatModel( + chatReply({ role: "assistant", content: "Hello" }, "stop"), + ); + + await streamText({ model, prompt: "Hi" }).text; + + expect(sent).toHaveLength(1); + }); + + it.each([ + "<|start|>", + "<|end|>", + "<|message|>", + "<|channel|>", + "<|constrain|>", + "<|return|>", + "<|call|>", + ])("removes the whole text of a reply with %s", async (token) => { + const { model } = chatModel( + chatReply( + { + role: "assistant", + content: `We need to draft the section. ${token}assistant to=functions`, + }, + "stop", + ), + ); + + const result = streamText({ model, prompt: "Hi" }); + + expect(await result.text).toBe(""); + }); + + it("removes reasoning written before the markup, as in the reply of 2026-11-02", async () => { + vi.spyOn(console, "error").mockImplementation(() => {}); + const { model } = chatModel( + chatReply({ role: "assistant", content: LEAKED_REASONING }, "stop"), + ); + + expect(await answerTo(model)).toEqual({ + text: "", + errors: ["The assistant ended without an answer."], + }); + }); + + it("still passes on a tool call next to the markup", async () => { + const { model } = chatModel( + chatReply( + { + ...toolCallMessage("call-1", "searchNotes", { query: "the list" }), + content: LEAKED_SEARCH, + }, + "tool_calls", + ), + ); + + const result = streamText({ + model, + prompt: "Hi", + tools: { + searchNotes: tool({ inputSchema: z.object({ query: z.string() }) }), + }, + }); + + expect(await result.text).toBe(""); + expect(await result.toolCalls).toEqual([ + expect.objectContaining({ + toolName: "searchNotes", + input: { query: "the list" }, + }), + ]); + }); + + it("says in the log what it did, without the reply's words", async () => { + const { model } = chatModel(leakedSearch); + + await streamText({ model, prompt: "Hi" }).text; + + expect(vi.mocked(console.warn).mock.calls).toEqual([ + ["The chat model wrote its own markup into a reply; asking it again."], + ["The chat model wrote its own markup again; its text was removed."], + ]); + }); + + it.each([ + ["plain words", "Nobody asked for grocery delivery."], + ["a look-alike", "Daan wrote and |call| in a note."], + ["an unknown token", "Daan wrote <|note|> in a note."], + ])("passes on %s as it came", async (_, content) => { + const { model } = chatModel( + chatReply({ role: "assistant", content }, "stop"), + ); + + const result = streamText({ model, prompt: "Hi" }); + + expect(await result.text).toBe(content); + expect(console.warn).not.toHaveBeenCalled(); + }); +}); + describe("the Cloudflare reranker", () => { /** Cloudflare's reply format for bge-reranker-base: each score by index. */ const rerankReply = (response: { id: number; score: number }[]) => () =>