Remind the model to save a section it wrote as text

docs/plan.md

@@ -31,7 +31,7 @@
   1. The rest of the owner's interface review, if any, as its next piece, and the measurement lane's staging try of the merged chat, whose screenshots may bring changes.
   2. On staging, the owner's check of the rate limit: the 11th message of a UTC day refused with the notice (about 10 answers of the day's allocation).
 - **Unverified:** on staging, the rate limit's refusal (the 11th message; Vercel's `x-forwarded-for` as one visitor's address; locally only against the fake's 429 and the database); the intro's carried-over arrival in Safari and Firefox; a reload keeping the PRD and the downloaded file (local Chromium only); a stop ending the answer on Cloudflare too, where the call in flight may still be finished and counted; the quota message against Cloudflare's real refusal (the tests replay its reply of 2026-10-20) and whether its per-minute 429 carries a retry header; a source name's pop-over, "Your decision, not from the interviews." under a claim of the visitor's, the table fix and the intro screen; Safari and Firefox (only Chromium ran; "Start over"'s pop-over under its button, on anchor positioning like the source's, else a centred box; the intro's arrival timed from the first paint, and its glimpse's fade at its foot; the chat's centred column and the PRD's narrower one; without anchor positioning the pop-over is a centred box, by design, and without `field-sizing`, before Safari 26.2 and Firefox 152, the message box keeps its three lines; the moving band, whose fallback is the still grey line; the PRD's highlight and rolling count; Jump to latest; the folded line's slide, which falls back to showing at once; the words' arrival, on inline words' relative offsets and a list marker's colour, and the searches drawing together; the pop-over growing in on `@starting-style`, before which it opens at once, and closing the same way only where `overlay` keeps it on top (Chrome), elsewhere at once, by design; the copy button, its label and the clipboard; Inter and its resized fallback, which needs Arial installed, so Linux and Android show the system's font until Inter arrives; a table's cells keeping their 144px and its box's sideways scroll; the Thinking line's band; what arrives, on `linear()` and `@property`, whose plain fallbacks are an ease-out and, without `@property`, a pane's rise and delay passing to what's in it). Motion on a phone and a slow machine (only headless Chromium on this one). On staging, how arrivals look at the real model's pace, a real reply's table, and the Thinking line through the real model's waits with the times it shows (the fake model's are about 0s). On staging, a reply that arrives whole coming in word by word, and a section's copy over HTTPS pasted into another app. Expect the Security Advisor's lint 0028 on `search_passages` on staging; it's intended (`supabase/AGENTS.md`). Production waits for launch and its Cloudflare token.
-- **Waiting on the owner:** pushing `docs/record-model-notice-merge` and its merge; running **Deploy production** for ee975d8; trying the 11th message on staging; the rest of the interface review, if any; whether `next dev` may keep adding its block to `AGENTS.md` (KI-8); the GitHub description and topics whenever they like.
+- **Waiting on the owner:** pushing `fix/say-when-a-section-was-not-saved` (a section written instead of saved gets up to two silent reminders: `src/server/AGENTS.md`, *A section written instead of saved*; the owner chose it over a notice, unmeasured, 2026-11-09), then trying "Draft the first section." on staging; pushing `docs/record-model-notice-merge` and its merge; running **Deploy production** for ee975d8; trying the 11th message on staging; the rest of the interface review, if any; whether `next dev` may keep adding its block to `AGENTS.md` (KI-8); the GitHub description and topics whenever they like.
 
 ### Shared rules
 - One branch per piece, from an up-to-date `main`; the owner pushes it, opens the pull request and merges it (`AGENTS.md`, Branches). After the other lane's merge, rebase an unpushed branch onto `main`, and merge `main` into a pushed one; never rewrite a pushed branch.

src/server/AGENTS.md

@@ -73,7 +73,7 @@ What shapes the model (2026-10-22):
 | Searches and saves per answer | `chat.ts`: `MAX_SEARCHES`, `MAX_SAVE_TRIES`, `MAX_STEPS` | three searches, each result saying how many are left; two tries to save |
 | How earlier answers look to the model | `chat.ts`: `forEarlierAnswers` | searches cut to passage ids and interviewees; saved sections whole; reasoning left out |
 | Its reasoning between tool calls in one answer | the AI SDK, `@ai-sdk/openai-compatible`, `ai/cloudflare.ts` | passed back as `reasoning`; whether Cloudflare reads it shows in the records, see below |
-| The transport | `ai/cloudflare.ts` | whole replies; empty text instead of `null`; reasoning sent as `reasoning` and tool results named, as Cloudflare's provider does (since 2026-10-22); no retry at the quota, retry waits of at most 5 seconds (slice 8a); a reply of one repeated character refused (since 2026-10-22); a reply that is only a JSON object of one text field becomes that text (since 2026-10-26); a reply's text holding the model's markup is asked for once more, then removed (since 2026-11-02) |
+| The transport | `ai/cloudflare.ts` | whole replies; empty text instead of `null`; reasoning sent as `reasoning` and tool results named, as Cloudflare's provider does (since 2026-10-22); no retry at the quota, retry waits of at most 5 seconds (slice 8a); a reply of one repeated character refused (since 2026-10-22); a reply that is only a JSON object of one text field becomes that text (since 2026-10-26); a reply's text holding the model's markup is asked for once more, then removed (since 2026-11-02); a section written instead of saved gets up to two reminders (since 2026-11-09) |
 | The search | `search.ts` and the embedder | fake word overlap here, bge-large-en-v1.5 on staging; the five closest (the reranker is off since 2026-10-22, *The reranker*) |
 | The app's own messages to the model | the check-in (`src/features/AGENTS.md`) and `checkReport` | fixed texts |
 | The visitor's wording | the visitor | free |
@@ -146,6 +146,7 @@ The tool that measures the model's behaviour (`docs/plan.md`, slices 9a to 9c; d
 - **Degenerate replies** (`refusingDegenerateReplies`, 2026-10-22): Cloudflare has answered with a run of "!" up to the output limit, with no reasoning and no tool call (2026-10-21 and 2026-10-22). A reply with no tool call, finish reason `length` and text at least 95% one character throws `DegenerateReplyError`, so the visitor gets the generic error and "Try again", never the text; the AI SDK doesn't retry it, and there's no retry of our own. The `wrapGenerate` middleware comes after `simulateStreamingMiddleware` in the array, since the last one is wrapped directly around the model and so sees each whole reply before it becomes a stream (the installed `LanguageModelMiddleware` type in `node_modules/ai/dist/index.d.ts` and `wrapLanguageModel` in `dist/index.js`; https://ai-sdk.dev/docs/reference/ai-sdk-core/wrap-language-model, read 2026-10-22 through Context7). The recorded replies are replayed in `tests/unit/cloudflare.test.ts`; the measurement counts such an answer "not run".
 - **Replies written as JSON** (`unwrappingJsonReplies`, 2026-10-26): on staging, gpt-oss-120b's whole reply after a search was `{"question":"…"}`, shown to the visitor as is. A reply with no tool call whose whole text is a JSON object of one non-empty text field, bare or in a code fence, becomes that text; anything else, a save written as text included, passes as it came. A `wrapGenerate` middleware next to `refusingDegenerateReplies`, for the same reason; the reply is replayed in `tests/unit/cloudflare.test.ts`. It changes what earlier answers look like to the model too, since the browser sends the unwrapped text back.
 - **Replies holding the model's markup** (`retryingModelMarkup`, 2026-11-02): the owner saw gpt-oss-120b write its control tokens (https://github.com/openai/harmony/blob/main/docs/format.md, "Special Tokens") as the reply's text, three times on "Draft the first section.": a search after a save, its reasoning and made-up searches, and a search in an answer's first call, so nothing ran or saved. None of the 141 records before it held such a text. The same request goes once more, at most once per model call, so a second call's cost and time only when it happens; the reply carries both calls' tokens, and the meter counts both HTTP requests while a record's model call shows only the second reply. If the second reply holds markup too, its whole text goes, since the words before the first token can be reasoning; a tool call in the same reply stays. The chat then treats it as a reply without words (*Failures*, above). Two warnings without the text show in the logs how often it happens and how often the retry helps. No cause is isolated, and whether the retry usually works is unmeasured. The replies are replayed in `tests/unit/cloudflare.test.ts`.
+- **A section written instead of saved** (`remindingToSave`, `ai/unsaved-draft.ts`, 2026-11-09): the owner saw gpt-oss-120b write "Draft the first section." as text, with nothing saved; told "you didnt save it", it saved. While `saveSection` is on offer, a reply with no tool call whose first line is only a section's title (or that is a save written as JSON) and that quotes at least two sources gets its draft and a reminder sent back, up to two times, and the visitor sees only what comes of it. A reply must call `saveSection` to count. No reminder starts 25 seconds or more after the model call began, since the route stops at 60. If it never saves, the first reply passes as written and the visitor is told nothing (the owner, 2026-11-09). It sits outside `retryingModelMarkup`, so each reminder gets that check too. The rule matched all 5 such replies among the 18 unsaved of 82 records and none of the other 13, but it was adjusted on those records; replies in `tests/unit/unsaved-draft.test.ts`. Unmeasured on the real model (the owner's choice, as with the markup retry): the warnings show how often it runs and helps.
 - **`includeUsage: true`** makes streamed replies report tokens; kept for when streaming returns.
 - **No token count for embeddings:** on staging (2026-10-17) Cloudflare's `/embeddings` reply had no `usage`, and the AI SDK then reports `NaN` tokens. `cloudflare-embedder.ts` logs "Cloudflare reported no token count" instead.
 - **Embedding batches:** Cloudflare's bge models take at most 100 texts a call; the package would send 2,048, so `cloudflare-embedder.ts` splits them.

src/server/ai/cloudflare.ts

@@ -6,6 +6,7 @@ import {
 } from "ai";
 import type { ChatModel } from "./chat-model.ts";
 import { CHAT_MODEL } from "./models.ts";
+import { looksLikeUnsavedDraft } from "./unsaved-draft.ts";
 
 // Cloudflare Workers AI through its OpenAI-compatible endpoint, used only on
 // staging and production. The account is on the free plan with no billing,
@@ -316,6 +317,27 @@ function addedTokens(first: number | undefined, second: number | undefined) {
   return (first ?? 0) + (second ?? 0);
 }
 
+type Usage = Awaited<ReturnType<ChatModel["doGenerate"]>>["usage"];
+
+/** What two calls used together, for one reply that took both. */
+function bothCalls(first: Usage, second: Usage): Usage {
+  const { inputTokens: input, outputTokens: output } = first;
+  return {
+    ...second,
+    inputTokens: {
+      total: addedTokens(input.total, second.inputTokens.total),
+      noCache: addedTokens(input.noCache, second.inputTokens.noCache),
+      cacheRead: addedTokens(input.cacheRead, second.inputTokens.cacheRead),
+      cacheWrite: addedTokens(input.cacheWrite, second.inputTokens.cacheWrite),
+    },
+    outputTokens: {
+      total: addedTokens(output.total, second.outputTokens.total),
+      text: addedTokens(output.text, second.outputTokens.text),
+      reasoning: addedTokens(output.reasoning, second.outputTokens.reasoning),
+    },
+  };
+}
+
 /**
  * gpt-oss can write its own markup as the reply's text: a search after a save,
  * its reasoning and made-up searches, or a search in an answer's first call,
@@ -335,30 +357,7 @@ const retryingModelMarkup: LanguageModelMiddleware = {
       "The chat model wrote its own markup into a reply; asking it again.",
     );
     const second = await doGenerate();
-    const { inputTokens: input, outputTokens: output } = first.usage;
-    const usage = {
-      ...second.usage,
-      inputTokens: {
-        total: addedTokens(input.total, second.usage.inputTokens.total),
-        noCache: addedTokens(input.noCache, second.usage.inputTokens.noCache),
-        cacheRead: addedTokens(
-          input.cacheRead,
-          second.usage.inputTokens.cacheRead,
-        ),
-        cacheWrite: addedTokens(
-          input.cacheWrite,
-          second.usage.inputTokens.cacheWrite,
-        ),
-      },
-      outputTokens: {
-        total: addedTokens(output.total, second.usage.outputTokens.total),
-        text: addedTokens(output.text, second.usage.outputTokens.text),
-        reasoning: addedTokens(
-          output.reasoning,
-          second.usage.outputTokens.reasoning,
-        ),
-      },
-    };
+    const usage = bothCalls(first.usage, second.usage);
     if (!holdsMarkup(second.content)) return { ...second, usage };
     console.warn(
       "The chat model wrote its own markup again; its text was removed.",
@@ -371,6 +370,82 @@ const retryingModelMarkup: LanguageModelMiddleware = {
   },
 };
 
+/** At most this many reminders to save, for one model call. */
+const MAX_SAVE_REMINDERS = 2;
+
+/**
+ * No reminder starts this long after the model call began. The chat route
+ * stops at 60 seconds, which would lose the whole answer; a recorded model
+ * call took up to 52 seconds and 20 at the 90th percentile.
+ */
+const SAVE_REMINDER_WINDOW_MS = 25_000;
+
+const SAVE_REMINDER =
+  "You wrote that section as text, so it was not saved. Call saveSection now with that section, and do not write it as text.";
+
+/**
+ * gpt-oss can write a section into its reply instead of saving it, with nothing
+ * saved or checked (seen by the owner, 2026-11-09, on "Draft the first
+ * section."). While saving is on offer, the model is shown its draft and told
+ * to save it, up to `MAX_SAVE_REMINDERS` times, and the visitor sees only what
+ * comes of that. If it never saves, the reply stays as the model first wrote
+ * it (the owner, 2026-11-09: the visitor gets no notice of this).
+ */
+const remindingToSave: LanguageModelMiddleware = {
+  specificationVersion: "v4",
+  async wrapGenerate({ doGenerate, params, model }) {
+    const started = Date.now();
+    const first = await doGenerate();
+    const offersSave = params.tools?.some(
+      (tool) => tool.name === "saveSection",
+    );
+    const text = first.content
+      .flatMap((part) => (part.type === "text" ? [part.text] : []))
+      .join("");
+    if (
+      !offersSave ||
+      first.content.some((part) => part.type === "tool-call") ||
+      !looksLikeUnsavedDraft(text)
+    ) {
+      return first;
+    }
+
+    let usage = first.usage;
+    for (let reminders = 0; reminders < MAX_SAVE_REMINDERS; reminders += 1) {
+      if (Date.now() - started >= SAVE_REMINDER_WINDOW_MS) break;
+      console.warn(
+        "The chat model wrote a section without saving it; reminding it.",
+      );
+      const reply = await model.doGenerate({
+        ...params,
+        prompt: [
+          ...params.prompt,
+          { role: "assistant", content: [{ type: "text", text }] },
+          {
+            role: "user",
+            content: [{ type: "text", text: SAVE_REMINDER }],
+          },
+        ],
+      });
+      usage = bothCalls(usage, reply.usage);
+      const saves = reply.content.some(
+        (part) => part.type === "tool-call" && part.toolName === "saveSection",
+      );
+      if (saves) {
+        console.warn("The reminder got the section saved.");
+        return {
+          ...reply,
+          usage,
+          // Words beside the save would show the draft after all.
+          content: reply.content.filter((part) => part.type !== "text"),
+        };
+      }
+    }
+    console.warn("The chat model never saved the section it wrote.");
+    return { ...first, usage };
+  },
+};
+
 /**
  * The chat model, asked for whole replies. Streamed, Cloudflare sends
  * gpt-oss's tool calls as reasoning text without the tool's name and ends
@@ -389,6 +464,7 @@ export function createCloudflareChatModel(
     middleware: [
       simulateStreamingMiddleware(),
       unwrappingJsonReplies,
+      remindingToSave,
       retryingModelMarkup,
       refusingDegenerateReplies,
     ],

src/server/ai/unsaved-draft.ts

@@ -0,0 +1,48 @@
+import { PRD_SECTIONS } from "../../lib/prd.ts";
+
+// Whether a reply is a PRD section written out as text, for a reply that
+// called no tool. A reply's quotes alone can't tell: the system prompt gives
+// answers about the interviews the same form. So the reply's first line has to
+// be nothing but a section's title, or the reply a save written as JSON, and
+// it has to quote at least two sources. Checked on the measurement's records
+// (2026-11-09: all 5 such replies of 18 without a save, and none of the other
+// 13); those were also what the rule was adjusted on.
+
+const MIN_QUOTES = 2;
+
+/** The dashes gpt-oss writes in passage ids, such as the non-breaking one. */
+const DASHES = /[‐-—]/g;
+
+const OPENING_QUOTE = `\\\\?["\\u201c]`;
+
+const QUOTED_SOURCES = [
+  // sofia-2: "…", also with the name around it or in a table's cell.
+  new RegExp(`\\b[a-z]+-\\d+\\b[^"\\u201c\\n]{0,12}${OPENING_QUOTE}`, "gi"),
+  // chat: "…", the product manager's own words.
+  new RegExp(`\\bchat\\s*:\\s*${OPENING_QUOTE}`, "gi"),
+  // A save written as JSON.
+  /"source"\s*:\s*"[a-z]+-\d+"/gi,
+];
+
+const lettersOf = (text: string) => text.toLowerCase().replace(/[^a-z]/g, "");
+
+const TITLES = new Set(PRD_SECTIONS.map(lettersOf));
+
+function opensWithSectionTitle(text: string): boolean {
+  if (/^\s*\{\s*"section"/.test(text)) return true;
+  const firstLine = text.trim().split("\n")[0] ?? "";
+  const title = lettersOf(firstLine.replace(/\((draft|updated|revised)\)/i, ""))
+    .replace(/^(section|draft)/, "")
+    .replace(/draft$/, "");
+  return TITLES.has(title);
+}
+
+export function looksLikeUnsavedDraft(reply: string): boolean {
+  const text = reply.replace(DASHES, "-");
+  if (!opensWithSectionTitle(text)) return false;
+  const quotes = QUOTED_SOURCES.reduce(
+    (sum, pattern) => sum + (text.match(pattern)?.length ?? 0),
+    0,
+  );
+  return quotes >= MIN_QUOTES;
+}