Replay the real model's recorded answers with npm run dev:replay

AGENTS.md

@@ -64,6 +64,8 @@ Use the npm scripts, not the tools underneath them: they run the environment gua
 | --- | --- |
 | `dev`, `build`, `start` | Next.js, behind the environment guard |
 | `dev:real` | The dev server with the chat on the real model, from this machine's token in `.env.development.local` |
+| `dev:replay` | The dev server with the chat replaying the real model's recorded answers, at their real pace; free, no token (`src/server/AGENTS.md`, *Replayed answers*) |
+| `replays:copy` | Copies a few counted runs of each measured request from a measurement's records into `src/server/ai/replays.json`; reads the records only |
 | `measure`, `measure:fake` | The measurement: fixed requests on the real model and the real search, with that token; the same on the fakes, free (`src/server/AGENTS.md`, *The measurement*) |
 | `measure:report` | A measurement's report and its comparison with the baseline, from the saved records; sends nothing |
 | `doctor` | Checks Node, Git, Docker, the pinned Supabase CLI, gitleaks and the GitHub CLI |

README.md

@@ -63,6 +63,8 @@ Then open http://localhost:3000. Everything runs on your machine: the database i
 | --- | --- |
 | `npm run dev` | The app in development mode, at http://localhost:3000 |
 | `npm run dev:real` | The same, with the chat on the real model and your own token (see below) |
+| `npm run dev:replay` | The same, with the chat replaying the real model's recorded answers: free, no token (see below) |
+| `npm run replays:copy -- --from <measurement folder>` | Copies a few recorded runs of each measured request into the project, for `dev:replay` |
 | `npm run measure -- --name <n> --purpose "<why>"` | Measures the AI's behaviour on the real model with your own token (see below); `--plan` only prints what it would send and cost |
 | `npm run measure:fake -- --name <n> --purpose "<why>"` | The same tool on the fakes, free |
 | `npm run build`, `npm start` | A production build, and serving it |
@@ -91,6 +93,10 @@ By default the chat runs on a scripted fake, so the tests are free and always gi
 
 The search still runs on the fake embedder, because the local passages carry its vectors, so which passages come back differs from staging. Every message spends neurons from the daily free allocation that staging and production share (10,000 a day; a first section costs about 310). Plain `npm run dev` reads the file too but keeps the fake chat; only `dev:real` switches the chat to the real model. Tests, builds and every other command except the measurement keep the fakes, and the environment guard refuses the token anywhere but that file.
 
+### Replayed real answers
+
+`npm run dev:replay` shows the interface with the real model's answers as they arrive on staging, for free and without a token. The chat replays answers the measurement recorded on gpt-oss-120b (a few runs of each measured request, in `src/server/ai/replays.json`): each reply whole, after the time the real model took. The intro's button, the grocery delivery suggestion and the other measured requests replay, each request's runs in turn; any other message gets the scripted fake's answer, paced the same way. The searches and saves still run on your machine, so the search lines show local passages, while the words come from the recording. The header says "Replaying recorded answers". To refresh the copy after a new measurement, run `npm run replays:copy -- --from docs/records/measurements/<name>` where the records are.
+
 ### Measuring the AI's behaviour
 
 The AI answers differently every time, so its behaviour is measured over several runs instead of judged from one. `npm run measure` sends five fixed requests (six answers: the first section, the disagreement about notifications, a gap in the interviews and the visitor's answer to it, and the grocery delivery and dark mode claims) to the real model, five runs each by default, with the search on the real embedder over the command's own copy of the passages. The app, the tests and the local database keep the fakes.

docs/plan.md

@@ -20,12 +20,11 @@
 
 ### Build lane
 - **Where:** its own worktree per piece, on a branch from an up-to-date `main`. If the session isn't in a worktree yet (`git worktree list` shows its folder as the main checkout, not under `.claude/worktrees/`), it creates one before touching anything, and never checks out a branch in the main folder, where the measurement lane works. Claude Code's worktree option puts it under `.claude/worktrees/`; run `npm ci` there first. A worktree made with `git worktree add` gets no `.env.local`: run `npm run env:local` there, or the end-to-end tests find no interviews. Its browser check uses `app-build` in `.claude/launch.json` (port 3200), never `app` or `app-real`, which are the measurement lane's on port 3000. It never runs `measure`, `measure:fake` or `dev:real`, never edits the measurement lane's files above, and changes code in `src/server` only where its piece needs it, never what the model receives (listed above); its pull request says which `src/server` files it changed and why.
-- **Doing:** nothing in progress. Slice 21 is merged (48101cd, with 82cde8b, 2026-10-21), and its deploy brought slices 20 and 21 to staging (run 39609303286), after slice 20's own deploy had stopped at `test:e2e`.
+- **Doing:** slice 22 is done on `feat/replay-real-answers` (`.claude/worktrees/slice-22-replay`), from `main` at e953bd0, waiting for the owner's push and merge. Local checks on 2026-10-21: `verify` (509 unit tests) and `test:e2e` (68, of which 1 new in `tests/e2e/replay.spec.ts`, on a second server on port 3101) passed in one full run each. The page checked at its real pace with `npm run dev:replay` from the worktree on port 3200: the intro's button replayed A-3 ("Used 4 searches", the save checked here with 14 of 14 quotes confirmed, "Thought 2s", the recorded words), the header's "Replaying recorded answers", no console errors. It adds `src/server/ai/replay-chat-model.ts`, `replays.ts` and `replays.json` (18 baseline runs, copied with `npm run replays:copy` from the main checkout's records, which it only read), lets `providers.ts` and `env.ts` take `AI_CHAT_PROVIDER=replay` and `REPLAY_SPEED`, and exports `lastVisitorMessage` from `fake-chat-model.ts`; nothing the real model receives changes, so the measurement lane needn't wait for it.
 - **Next:**
-  1. Slice 22, the replayed real answers, on its own branch from `main` once the owner says go (asked 2026-10-21).
-  2. Slice 8, the rate limit, once 9d is done and the owner has decided its legal basis and size (*Deferred*).
+  1. Slice 8, the rate limit, once 9d is done and the owner has decided its legal basis and size (*Deferred*).
 - **Unverified:** on staging, a reload keeping the PRD and the downloaded file (local Chromium only); a stop ending the answer on Cloudflare too, where the call in flight may still be finished and counted; the quota message against Cloudflare's real refusal (the tests replay its reply of 2026-10-20) and whether its per-minute 429 carries a retry header; a source name's pop-over, "Your decision, not from the interviews." under a claim of the visitor's, the table fix and the intro screen; Safari and Firefox (only Chromium ran; without anchor positioning the pop-over is a centred box, by design, and without `field-sizing`, before Safari 26.2 and Firefox 152, the message box keeps its three lines; the moving band, whose fallback is the still grey line; the PRD's highlight and rolling count; Jump to latest; the folded line's slide, which falls back to showing at once; the words' fade; the copy button, its label and the clipboard; Inter and its resized fallback, which needs Arial installed, so Linux and Android show the system's font until Inter arrives; a table's cells keeping their 144px and its box's sideways scroll; the Thinking line's band). On staging, a real reply's table, and the Thinking line through the real model's waits with the times it shows (the fake model's are about 0s). On staging, a reply that arrives whole fading in as one, and a section's copy over HTTPS pasted into another app. Expect the Security Advisor's lint 0028 on `search_passages` on staging; it's intended (`supabase/AGENTS.md`). Production waits for launch and its Cloudflare token.
-- **Waiting on the owner:** the go for slice 22; whether `next dev` may keep adding its block to `AGENTS.md` (KI-8); the GitHub description and topics whenever they like.
+- **Waiting on the owner:** pushing `feat/replay-real-answers`, its pull request and the merge; whether `next dev` may keep adding its block to `AGENTS.md` (KI-8); the GitHub description and topics whenever they like.
 
 ### Shared rules
 - One branch per piece, from an up-to-date `main`; the owner pushes it, opens the pull request and merges it (`AGENTS.md`, Branches). After the other lane's merge, rebase an unpushed branch onto `main`, and merge `main` into a pushed one; never rewrite a pushed branch.
@@ -119,11 +118,4 @@
 
 ### 21 · Steady end-to-end tests · risk: normal · status: done (48101cd, with 82cde8b)
 
-### 22 · Replayed real answers, to look at the interface here · risk: normal · status: todo
-- **Goal:** the owner and the build lane can watch the interface on this machine with the real model's answers as they arrive on staging (their words, tables, searches, saves and waits), for free and without the token. The fake model's answers are made up and arrive word by word every 15ms, with thoughts of about 0s; the real model's arrive whole, after calls of about 1.5 to 20 seconds each (the baseline's records, 2026-10-19).
-- **Acceptance:**
-  - A command such as `npm run dev:replay` runs the dev server with a chat model that replays recorded answers: each model call's parts as recorded, whole, after that call's recorded duration. It reads no token and sends nothing to Cloudflare; `dev:real` stays the only way to the real model here, and the tests stay on the scripted fake.
-  - The recordings are a few counted runs of each measured request, A to E, copied from `docs/records/measurements/` (ignored by Git, so only this machine's main checkout has them) into the project, trimmed to each call's parts and duration: no request bodies, headers, ids or logs. The copy only reads the measurement lane's records and never edits them.
-  - A visitor's message picks its recording by the request's words, so the intro's button and the try-to-break-it suggestion replay real answers (both take their words from `src/lib/suggestions.ts`), and a check-in after a replayed draft replays its continuation where one was recorded. Each recording's searches and saves still run here, on the local database and the fake embedder, so the search lines show local passages; what the recorded reply says comes from the recording, and the screen says that it's a replay.
-  - Any other message gets the scripted fake's answer, paced like the real model: whole replies, after waits in the range the recordings show, so the Thinking line, the folded searches and the fade can be watched in their real rhythm.
-  - Unit tests for picking a recording and for the pacing; an end-to-end check that one replayed answer, with its wait shortened for the test, shows its recorded words. Nothing in what the real model receives changes; the pull request names the `src/server` files it adds.
+### 22 · Replayed real answers, to look at the interface here · risk: normal · status: done (on `feat/replay-real-answers`; its id on `main` goes here after the merge)

package.json

@@ -10,6 +10,8 @@
   "scripts": {
     "dev": "node scripts/env-guard.ts && next dev",
     "dev:real": "node scripts/env-guard.ts --real-chat && node scripts/dev-real.ts",
+    "dev:replay": "node scripts/env-guard.ts && node scripts/dev-replay.ts",
+    "replays:copy": "node scripts/copy-replays.ts",
     "measure": "node scripts/env-guard.ts --measure && node scripts/measure.ts",
     "measure:fake": "node scripts/env-guard.ts && node scripts/measure.ts --fake",
     "measure:report": "node scripts/env-guard.ts && node scripts/measure.ts --report",

playwright.config.ts

@@ -4,6 +4,8 @@ import { defineConfig, devices } from "@playwright/test";
 // dev server, so they test what users get.
 const PORT = 3100;
 const baseURL = `http://127.0.0.1:${PORT}`;
+const REPLAY_PORT = 3101;
+const REPLAY_URL = `http://127.0.0.1:${REPLAY_PORT}`;
 
 export default defineConfig({
   testDir: "tests/e2e",
@@ -22,11 +24,27 @@ export default defineConfig({
     trace: "retain-on-failure",
   },
   projects: [{ name: "chromium", use: { ...devices["Desktop Chrome"] } }],
-  webServer: {
-    command: `npm run build && npm run start -- --port ${PORT}`,
-    url: baseURL,
-    reuseExistingServer: false,
-    timeout: 300_000,
-    stdout: "pipe",
-  },
+  // Playwright starts its servers one after another, each once the one
+  // before answers (`createPluginSetupTasks` in the installed runner, read
+  // 2026-10-21), so the second starts on the build the first made. It runs
+  // the chat on the replayed answers (`npm run dev:replay`'s model), a
+  // hundred times faster than recorded; only `tests/e2e/replay.spec.ts` uses
+  // it, and every other test stays on the scripted fake.
+  webServer: [
+    {
+      command: `npm run build && npm run start -- --port ${PORT}`,
+      url: baseURL,
+      reuseExistingServer: false,
+      timeout: 300_000,
+      stdout: "pipe",
+    },
+    {
+      command: `npm run start -- --port ${REPLAY_PORT}`,
+      url: REPLAY_URL,
+      reuseExistingServer: false,
+      timeout: 60_000,
+      stdout: "pipe",
+      env: { AI_CHAT_PROVIDER: "replay", REPLAY_SPEED: "100" },
+    },
+  ],
 });

scripts/copy-replays.ts

@@ -0,0 +1,88 @@
+// Copies a few counted runs of each measured request into the project, for
+// `npm run dev:replay` (src/server/ai/replays.ts). It only reads the
+// measurement's records, which are git-ignored, so it runs where they are:
+// the main checkout, or a worktree given `--from` with the main checkout's
+// measurement folder. It never writes there.
+//
+//   npm run replays:copy -- --from <folder of one measurement>
+//
+// Per request, runs whose last call wrote words come first, then the others,
+// each by run number, so the intro's button replays a worded answer first.
+
+import { readdirSync, readFileSync, writeFileSync } from "node:fs";
+import { basename, join } from "node:path";
+import { parseArgs } from "node:util";
+import {
+  endsWithWords,
+  isReplayable,
+  REPLAYS_FILE,
+  REPLAYS_PER_REQUEST,
+  toRecording,
+  type MeasuredRecord,
+  type Recording,
+  type Replays,
+} from "../src/server/ai/replays.ts";
+import { REQUESTS } from "../src/server/measurement/requests.ts";
+
+const { values } = parseArgs({
+  options: {
+    from: { type: "string", default: "docs/records/measurements/baseline" },
+  },
+});
+const from = values.from;
+
+const UTC_DAY = /^\d{4}-\d{2}-\d{2}$/;
+
+type Found = { record: MeasuredRecord; recordedOn: string };
+
+const found: Found[] = readdirSync(from)
+  .filter((entry) => UTC_DAY.test(entry))
+  .flatMap((day) =>
+    readdirSync(join(from, day))
+      .filter((file) => file.endsWith(".json"))
+      .map((file) => ({
+        record: JSON.parse(
+          readFileSync(join(from, day, file), "utf8"),
+        ) as MeasuredRecord,
+        recordedOn: day,
+      })),
+  );
+
+const recordings: Recording[] = REQUESTS.flatMap((request) =>
+  found
+    .filter(
+      ({ record }) =>
+        record.setup.request === request.name && isReplayable(record),
+    )
+    .sort(
+      (a, b) =>
+        Number(endsWithWords(b.record)) - Number(endsWithWords(a.record)) ||
+        a.record.setup.run - b.record.setup.run,
+    )
+    .slice(0, REPLAYS_PER_REQUEST)
+    .map(({ record, recordedOn }) =>
+      toRecording(record, { recordedOn, words: request.text }),
+    ),
+);
+
+const replays: Replays = {
+  format: 1,
+  measurement: basename(from),
+  recordings,
+};
+
+writeFileSync(REPLAYS_FILE, `${JSON.stringify(replays, null, 2)}\n`);
+
+console.log(
+  `Copied ${recordings.length} runs of "${replays.measurement}" into ${REPLAYS_FILE}: ${recordings.map((recording) => recording.id).join(", ")}.`,
+);
+for (const request of REQUESTS) {
+  const copied = recordings.filter((recording) =>
+    recording.id.startsWith(`${request.name}-`),
+  ).length;
+  if (copied < REPLAYS_PER_REQUEST) {
+    console.log(
+      `Request ${request.name}: ${copied} of ${REPLAYS_PER_REQUEST} wanted.`,
+    );
+  }
+}

scripts/dev-replay.ts

@@ -0,0 +1,43 @@
+// Starts the dev server with the chat on the real model's recorded answers
+// (src/server/ai/replay-chat-model.ts), for watching the interface with real
+// replies and their real waits, for free. It sends nothing to Cloudflare and
+// the replay reads no token. Like `dev:real`, it sets AI_CHAT_PROVIDER for the
+// dev server, which no env file can undo (Next.js, "Environment variable load
+// order", read 2026-10-19), and NEXT_PUBLIC_CHAT_REPLAY, which the header
+// shows; Next.js inlines a NEXT_PUBLIC_ variable from the environment when it
+// compiles (read 2026-10-21 through Context7). The search keeps the fake
+// embedder and the local database. `npm run dev:replay` runs the guard first.
+
+import { spawn } from "node:child_process";
+import { createRequire } from "node:module";
+
+const next = createRequire(import.meta.url).resolve("next/dist/bin/next");
+
+console.log(
+  [
+    "Dev server with the chat on recorded answers: the measurement's real gpt-oss-120b replies, replayed with their real waits.",
+    "The intro's button, the grocery delivery suggestion and the other measured requests replay; any other message gets the fake model, paced the same way.",
+    "Nothing is sent to Cloudflare. The searches and saves run here, on the local database and the fake embedder.",
+    "",
+  ].join("\n"),
+);
+
+const server = spawn(
+  process.execPath,
+  [next, "dev", ...process.argv.slice(2)],
+  {
+    stdio: "inherit",
+    env: {
+      ...process.env,
+      AI_CHAT_PROVIDER: "replay",
+      NEXT_PUBLIC_CHAT_REPLAY: "1",
+    },
+  },
+);
+server.on("error", (error) => {
+  console.error(`The dev server didn't start: ${error.message}`);
+  process.exitCode = 1;
+});
+server.on("exit", (code, signal) => {
+  process.exitCode = code ?? (signal ? 1 : 0);
+});

src/app/layout.tsx

@@ -18,6 +18,11 @@ export const metadata: Metadata = {
 // jump when Inter arrives (Next.js, `next/font`, read 2026-10-21).
 const inter = Inter({ subsets: ["latin"], variable: "--font-inter" });
 
+// `npm run dev:replay` sets it, so the screen says the answers are replayed
+// (src/server/ai/replay-chat-model.ts). Inlined when Next.js compiles, so no
+// build for staging or production has it.
+const REPLAY = process.env.NEXT_PUBLIC_CHAT_REPLAY === "1";
+
 export default function RootLayout({ children }: { children: ReactNode }) {
   return (
     <html lang="en" className={`${inter.variable} h-full antialiased`}>
@@ -33,14 +38,21 @@ export default function RootLayout({ children }: { children: ReactNode }) {
             <Link href="/" className="text-base font-semibold tracking-tight">
               {site.name}
             </Link>
-            <nav aria-label="Main">
-              <Link
-                href="/transcripts"
-                className="text-sm text-muted-foreground hover:text-foreground"
-              >
-                Transcripts
-              </Link>
-            </nav>
+            <div className="flex items-center gap-6">
+              {REPLAY && (
+                <p className="text-sm text-muted-foreground">
+                  Replaying recorded answers
+                </p>
+              )}
+              <nav aria-label="Main">
+                <Link
+                  href="/transcripts"
+                  className="text-sm text-muted-foreground hover:text-foreground"
+                >
+                  Transcripts
+                </Link>
+              </nav>
+            </div>
           </div>
         </header>
         <main