From 0769037707ad6cc35cb297bf42ee5087920bcf9c Mon Sep 17 00:00:00 2001 From: TechNomadCode Date: Mon, 9 Nov 2026 15:30:15 +0100 Subject: [PATCH] Give the chat's answers 180 seconds instead of 60 Co-Authored-By: Claude Opus 5.5 --- docs/plan.md | 2 +- src/app/api/chat/route.ts | 8 ++++++-- src/server/AGENTS.md | 4 ++-- src/server/ai/cloudflare.ts | 9 +++++---- tests/unit/unsaved-draft.test.ts | 13 ++++++++++++- 5 files changed, 26 insertions(+), 10 deletions(-) diff --git a/docs/plan.md b/docs/plan.md index a2b4253..4fe15f0 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -31,7 +31,7 @@ 1. The rest of the owner's interface review, if any, as its next piece, and the measurement lane's staging try of the merged chat, whose screenshots may bring changes. 2. On staging, the owner's check of the rate limit: the 11th message of a UTC day refused with the notice (about 10 answers of the day's allocation). - **Unverified:** on staging, the rate limit's refusal (the 11th message; Vercel's `x-forwarded-for` as one visitor's address; locally only against the fake's 429 and the database); the intro's carried-over arrival in Safari and Firefox; a reload keeping the PRD and the downloaded file (local Chromium only); a stop ending the answer on Cloudflare too, where the call in flight may still be finished and counted; the quota message against Cloudflare's real refusal (the tests replay its reply of 2026-10-20) and whether its per-minute 429 carries a retry header; a source name's pop-over, "Your decision, not from the interviews." under a claim of the visitor's, the table fix and the intro screen; Safari and Firefox (only Chromium ran; "Start over"'s pop-over under its button, on anchor positioning like the source's, else a centred box; the intro's arrival timed from the first paint, and its glimpse's fade at its foot; the chat's centred column and the PRD's narrower one; without anchor positioning the pop-over is a centred box, by design, and without `field-sizing`, before Safari 26.2 and Firefox 152, the message box keeps its three lines; the moving band, whose fallback is the still grey line; the PRD's highlight and rolling count; Jump to latest; the folded line's slide, which falls back to showing at once; the words' arrival, on inline words' relative offsets and a list marker's colour, and the searches drawing together; the pop-over growing in on `@starting-style`, before which it opens at once, and closing the same way only where `overlay` keeps it on top (Chrome), elsewhere at once, by design; the copy button, its label and the clipboard; Inter and its resized fallback, which needs Arial installed, so Linux and Android show the system's font until Inter arrives; a table's cells keeping their 144px and its box's sideways scroll; the Thinking line's band; what arrives, on `linear()` and `@property`, whose plain fallbacks are an ease-out and, without `@property`, a pane's rise and delay passing to what's in it). Motion on a phone and a slow machine (only headless Chromium on this one). On staging, how arrivals look at the real model's pace, a real reply's table, and the Thinking line through the real model's waits with the times it shows (the fake model's are about 0s). On staging, a reply that arrives whole coming in word by word, and a section's copy over HTTPS pasted into another app. Expect the Security Advisor's lint 0028 on `search_passages` on staging; it's intended (`supabase/AGENTS.md`). Production waits for launch and its Cloudflare token. -- **Waiting on the owner:** pushing `fix/say-when-a-section-was-not-saved` (a section written instead of saved gets up to two silent reminders: `src/server/AGENTS.md`, *A section written instead of saved*; the owner chose it over a notice, unmeasured, 2026-11-09), then trying "Draft the first section." on staging; pushing `docs/record-model-notice-merge` and its merge; running **Deploy production** for ee975d8; trying the 11th message on staging; the rest of the interface review, if any; whether `next dev` may keep adding its block to `AGENTS.md` (KI-8); the GitHub description and topics whenever they like. +- **Waiting on the owner:** pushing `fix/give-slow-answers-more-time` (the chat route's limit from 60 to 180 seconds: a 56-second model call on staging, 2026-11-09), then trying "Draft the first section." on staging, which also tries the merged reminders to save (987f8cb; unmeasured, the owner's choice); pushing `docs/record-model-notice-merge` and its merge; running **Deploy production** for ee975d8; trying the 11th message on staging; the rest of the interface review, if any; whether `next dev` may keep adding its block to `AGENTS.md` (KI-8); the GitHub description and topics whenever they like. ### Shared rules - One branch per piece, from an up-to-date `main`; the owner pushes it, opens the pull request and merges it (`AGENTS.md`, Branches). After the other lane's merge, rebase an unpushed branch onto `main`, and merge `main` into a pushed one; never rewrite a pushed branch. diff --git a/src/app/api/chat/route.ts b/src/app/api/chat/route.ts index 191a581..f817842 100644 --- a/src/app/api/chat/route.ts +++ b/src/app/api/chat/route.ts @@ -6,8 +6,12 @@ import { admitMessage } from "@/server/rate-limit"; // The chat's endpoint: the browser posts the conversation, and the answer // streams back as the AI SDK's UI message stream (server-sent events). -/** Seconds for one answer: its searches and its reply. Hobby allows 300. */ -export const maxDuration = 60; +/** + * Seconds for one answer: its searches, its reply and any retries. Hobby + * allows 300. At 60, one model call of 56 seconds used up an answer on + * staging (2026-11-09), and the visitor was left with a search and no reply. + */ +export const maxDuration = 180; /** * Requests longer than this are refused before they're parsed. A full PRD's diff --git a/src/server/AGENTS.md b/src/server/AGENTS.md index ff294ee..e4ba96b 100644 --- a/src/server/AGENTS.md +++ b/src/server/AGENTS.md @@ -142,11 +142,11 @@ The tool that measures the model's behaviour (`docs/plan.md`, slices 9a to 9c; d - **Text in every message:** Cloudflare's input schema requires each message's `content` to be a string or a list of text parts (`sync-input.json` of the model). The package sends `content: null` on an assistant message that only calls a tool, so the follow-up after a search got 400, code 5006 (staging, 2026-10-17). `withTextContent` in `transformRequestBody` sends `""` instead. - **256 output tokens by default:** the model's `max_tokens` defaults to 256 (`sync-input.json`), and gpt-oss's reasoning counts against it. On staging (2026-10-17), section drafts stopped with finish reason `length` before any text, so `chat.ts` asks for 4096. The schema states no maximum; the context window is 128,000 tokens. - **No tool choice:** the schema has `tools` but no `tool_choice`, so the chat stops the search by leaving the tools out of the request rather than sending `tool_choice: "none"`. The schema allows a request without tools, even after tool calls in the conversation. On staging (2026-10-18), the reply after a save, sent without tools, came back as text. A request without tools doesn't guarantee words, though: the same day, gpt-oss called `searchNotes` after it was switched off, even on the last call, which carried no tools, and the answer ended without words. -- **The quota and other 429s** (slice 8a): `refusingQuota` wraps `fetch` for chat and embeddings alike, under whatever `fetch` the settings pass (the measurement's meter and recorder see every reply first). A 429 with code 3036 or 4006 (`QUOTA_CODES`; 4006 is what `/ai/v1` sent on 2026-10-20) throws `QuotaExceededError` from `fetch`, which `handleFetchError` in `@ai-sdk/provider-utils` passes through and the SDK doesn't retry: nothing works before 00:00 UTC. Any other 429 (the per-minute limit, code 3040 out of capacity) goes on to the SDK's two retries, but a `retry-after-ms` or `retry-after` asking for more than `MAX_RETRY_WAIT_MS` (5 seconds) is dropped first, so the SDK's own 2 and 4 seconds apply instead of up to 60 each, inside the route's 60 seconds (`getRetryDelayInMs` in `ai/dist/index.js`, read 2026-10-20). The request bodies are untouched. The quota codes are also in `measurement/budget.ts` (`QUOTA_CODES`); keep the two lists the same. Replayed replies in `tests/unit/cloudflare.test.ts`; the 4006 one is Cloudflare's own of 2026-10-20. +- **The quota and other 429s** (slice 8a): `refusingQuota` wraps `fetch` for chat and embeddings alike, under whatever `fetch` the settings pass (the measurement's meter and recorder see every reply first). A 429 with code 3036 or 4006 (`QUOTA_CODES`; 4006 is what `/ai/v1` sent on 2026-10-20) throws `QuotaExceededError` from `fetch`, which `handleFetchError` in `@ai-sdk/provider-utils` passes through and the SDK doesn't retry: nothing works before 00:00 UTC. Any other 429 (the per-minute limit, code 3040 out of capacity) goes on to the SDK's two retries, but a `retry-after-ms` or `retry-after` asking for more than `MAX_RETRY_WAIT_MS` (5 seconds) is dropped first, so the SDK's own 2 and 4 seconds apply instead of up to 60 each, inside the route's time limit (`maxDuration`, 180 seconds) (`getRetryDelayInMs` in `ai/dist/index.js`, read 2026-10-20). The request bodies are untouched. The quota codes are also in `measurement/budget.ts` (`QUOTA_CODES`); keep the two lists the same. Replayed replies in `tests/unit/cloudflare.test.ts`; the 4006 one is Cloudflare's own of 2026-10-20. - **Degenerate replies** (`refusingDegenerateReplies`, 2026-10-22): Cloudflare has answered with a run of "!" up to the output limit, with no reasoning and no tool call (2026-10-21 and 2026-10-22). A reply with no tool call, finish reason `length` and text at least 95% one character throws `DegenerateReplyError`, so the visitor gets the generic error and "Try again", never the text; the AI SDK doesn't retry it, and there's no retry of our own. The `wrapGenerate` middleware comes after `simulateStreamingMiddleware` in the array, since the last one is wrapped directly around the model and so sees each whole reply before it becomes a stream (the installed `LanguageModelMiddleware` type in `node_modules/ai/dist/index.d.ts` and `wrapLanguageModel` in `dist/index.js`; https://ai-sdk.dev/docs/reference/ai-sdk-core/wrap-language-model, read 2026-10-22 through Context7). The recorded replies are replayed in `tests/unit/cloudflare.test.ts`; the measurement counts such an answer "not run". - **Replies written as JSON** (`unwrappingJsonReplies`, 2026-10-26): on staging, gpt-oss-120b's whole reply after a search was `{"question":"…"}`, shown to the visitor as is. A reply with no tool call whose whole text is a JSON object of one non-empty text field, bare or in a code fence, becomes that text; anything else, a save written as text included, passes as it came. A `wrapGenerate` middleware next to `refusingDegenerateReplies`, for the same reason; the reply is replayed in `tests/unit/cloudflare.test.ts`. It changes what earlier answers look like to the model too, since the browser sends the unwrapped text back. - **Replies holding the model's markup** (`retryingModelMarkup`, 2026-11-02): the owner saw gpt-oss-120b write its control tokens (https://github.com/openai/harmony/blob/main/docs/format.md, "Special Tokens") as the reply's text, three times on "Draft the first section.": a search after a save, its reasoning and made-up searches, and a search in an answer's first call, so nothing ran or saved. None of the 141 records before it held such a text. The same request goes once more, at most once per model call, so a second call's cost and time only when it happens; the reply carries both calls' tokens, and the meter counts both HTTP requests while a record's model call shows only the second reply. If the second reply holds markup too, its whole text goes, since the words before the first token can be reasoning; a tool call in the same reply stays. The chat then treats it as a reply without words (*Failures*, above). Two warnings without the text show in the logs how often it happens and how often the retry helps. No cause is isolated, and whether the retry usually works is unmeasured. The replies are replayed in `tests/unit/cloudflare.test.ts`. -- **A section written instead of saved** (`remindingToSave`, `ai/unsaved-draft.ts`, 2026-11-09): the owner saw gpt-oss-120b write "Draft the first section." as text, with nothing saved; told "you didnt save it", it saved. While `saveSection` is on offer, a reply with no tool call whose first line is only a section's title (or that is a save written as JSON) and that quotes at least two sources gets its draft and a reminder sent back, up to two times, and the visitor sees only what comes of it. A reply must call `saveSection` to count. No reminder starts 25 seconds or more after the model call began, since the route stops at 60. If it never saves, the first reply passes as written and the visitor is told nothing (the owner, 2026-11-09). It sits outside `retryingModelMarkup`, so each reminder gets that check too. The rule matched all 5 such replies among the 18 unsaved of 82 records and none of the other 13, but it was adjusted on those records; replies in `tests/unit/unsaved-draft.test.ts`. Unmeasured on the real model (the owner's choice, as with the markup retry): the warnings show how often it runs and helps. +- **A section written instead of saved** (`remindingToSave`, `ai/unsaved-draft.ts`, 2026-11-09): the owner saw gpt-oss-120b write "Draft the first section." as text, with nothing saved; told "you didnt save it", it saved. While `saveSection` is on offer, a reply with no tool call whose first line is only a section's title (or that is a save written as JSON) and that quotes at least two sources gets its draft and a reminder sent back, up to two times, and the visitor sees only what comes of it. A reply must call `saveSection` to count. No reminder starts 90 seconds or more after the model call began, since the route stops at 180. If it never saves, the first reply passes as written and the visitor is told nothing (the owner, 2026-11-09). It sits outside `retryingModelMarkup`, so each reminder gets that check too. The rule matched all 5 such replies among the 18 unsaved of 82 records and none of the other 13, but it was adjusted on those records; replies in `tests/unit/unsaved-draft.test.ts`. Unmeasured on the real model (the owner's choice, as with the markup retry): the warnings show how often it runs and helps. - **`includeUsage: true`** makes streamed replies report tokens; kept for when streaming returns. - **No token count for embeddings:** on staging (2026-10-17) Cloudflare's `/embeddings` reply had no `usage`, and the AI SDK then reports `NaN` tokens. `cloudflare-embedder.ts` logs "Cloudflare reported no token count" instead. - **Embedding batches:** Cloudflare's bge models take at most 100 texts a call; the package would send 2,048, so `cloudflare-embedder.ts` splits them. diff --git a/src/server/ai/cloudflare.ts b/src/server/ai/cloudflare.ts index 92a3436..294a96a 100644 --- a/src/server/ai/cloudflare.ts +++ b/src/server/ai/cloudflare.ts @@ -46,7 +46,7 @@ const QUOTA_CODES: readonly number[] = [3036, 4006]; /** * The longest wait between tries that Cloudflare's retry headers may ask of * the AI SDK. A longer one is dropped, so the SDK's own backoff (2 seconds, - * then 4) applies and an answer stays well inside the chat route's 60 seconds. + * then 4) applies and an answer stays inside the chat route's time limit. */ export const MAX_RETRY_WAIT_MS = 5000; @@ -375,10 +375,11 @@ const MAX_SAVE_REMINDERS = 2; /** * No reminder starts this long after the model call began. The chat route - * stops at 60 seconds, which would lose the whole answer; a recorded model - * call took up to 52 seconds and 20 at the 90th percentile. + * stops at 180 seconds, which would lose the whole answer; a recorded model + * call took up to 52 seconds and 20 at the 90th percentile, and one on staging + * 56. */ -const SAVE_REMINDER_WINDOW_MS = 25_000; +const SAVE_REMINDER_WINDOW_MS = 90_000; const SAVE_REMINDER = "You wrote that section as text, so it was not saved. Call saveSection now with that section, and do not write it as text."; diff --git a/tests/unit/unsaved-draft.test.ts b/tests/unit/unsaved-draft.test.ts index 2e535d4..2e2a628 100644 --- a/tests/unit/unsaved-draft.test.ts +++ b/tests/unit/unsaved-draft.test.ts @@ -254,13 +254,24 @@ describe("the chat when the model writes a section instead of saving it", () => let now = 1_000_000; vi.spyOn(Date, "now").mockImplementation(() => now); const { sent, model } = replaying([draft, save, checkedIn], () => { - now += 30_000; + now += 100_000; }); expect((await answerTo(model)).text).toBe(SCREENSHOT); expect(sent).toHaveLength(1); }); + it("still reminds after a slow first reply", async () => { + let now = 1_000_000; + vi.spyOn(Date, "now").mockImplementation(() => now); + const { sent, model } = replaying([draft, save, checkedIn], () => { + now += 30_000; + }); + + expect((await answerTo(model)).saved).toBe(true); + expect(sent).toHaveLength(3); + }); + it("logs what it did, without the reply's words", async () => { const { model } = replaying([draft, save, checkedIn]);