WP-10: Evals + cutover

docs/relaunch/CUTOVER-CHECKLIST.md

@@ -0,0 +1,28 @@
+# Cutover checklist (WP-10)
+
+Run when the owner has completed the key unlocks in `docs/FOR-ABID.md`
+(§2 AI keys, §6 Clerk + Neon, Blob token) and the eval gate has passed with
+live AI (`pnpm --dir web eval` — judge must not be BLOCKED).
+
+## Preconditions
+- [ ] `node web/scripts/verify-env.mjs production episodekit-web` — all required present
+- [ ] `pnpm --dir web eval` — judge = ok, faithfulness ≥4.3, usefulness ≥4.0
+- [ ] Owner reviewed ≥4 of 5 packages: "would publish with light edits"
+- [ ] `pnpm --dir web lint && pnpm --dir web typecheck && pnpm --dir web test` green on main
+- [ ] URL parity: legacy `site/sitemap.xml` paths all 200 on the new app (U17 test)
+
+## Execute
+- [ ] `./scripts/cutover.sh` (moves site/ api/ engine/ → legacy/, drops legacy vercel.json)
+- [ ] Vercel project `episodekit` (old live site): disconnect from repo or retarget `legacy/`
+- [ ] Vercel project `episodekit-web`: Root Directory = `web/`, Production branch = `main`, deploy
+- [ ] `pnpm --dir web exec drizzle-kit migrate` against production `DATABASE_URL`
+- [ ] Clerk webhook endpoint → `https://<prod-domain>/api/webhooks/clerk`
+- [ ] Lemon Squeezy webhook → `/api/webhooks/lemonsqueezy` (when LS keys land)
+
+## Post-cutover verification
+- [ ] `/`, `/tools/show-notes`, `/pricing`, `/privacy`, `/terms` → 200
+- [ ] Sign-up (email link + Google) → onboarding step 1 → workspace(plan=free)
+- [ ] Sample episode → full 7-artifact package in ≤5 min, quotes verified
+- [ ] Free-tool rate limit + cost cap live; `/admin` owner-only
+- [ ] Legacy URLs redirect or 200 (U17 parity)
+- [ ] `docs/FOR-ABID.md` updated: remaining owner actions checked off

docs/relaunch/DECISIONS.md

@@ -42,6 +42,9 @@ Format: `YYYY-MM-DD · WP-XX · decision · why · alternatives considered`.
 - 2026-10-27 · WP-06 · PGlite kept as one instance per dataDir via globalThis (Turbopack loads db modules per route chunk; a second instance on the same dir aborts the wasm) and `serverExternalPackages` so its wasm loader stays unbundled; `replaceSegments` falls back to sequential statements without neon's `.batch` · observed crashes; documented PGlite+bundler guidance · per-chunk instances
 - 2026-10-27 · WP-06 · `cacheComponents`/`partialPrefetching` disabled app-wide · the PPR runtime replay guard rejects `Date.now()` in streamed per-user pages (intermittent second-request failures); every page already uses `connection()` + Suspense which renders identically without it · per-route workarounds (fragile, guard fired from replayed shells)
 - 2026-10-27 · WP-06 · Stored segment ids namespaced `seg_<episodeId[:8]>_<idx>` — segments.id is the GLOBAL primary key (§7.1), so per-episode `seg_000000` collided across episodes (would break the 2nd episode in production) · found by the DB-mode e2e · uuid ids (breaks §7.3 prompt-shape familiarity)
+- 2026-10-27 · WP-10 · Eval transcripts: demo (real sample) + 7 clearly-labelled synthetic fixtures (interview/solo/panel/jargon/plain/narrative/short) standing in for the spec's "self-recorded/CC0" until the owner supplies real ones; drop-in replaceable, shapes documented in evals/transcripts/README.md · the gate must be runnable before owner recordings exist · skipping gate coverage (worse)
+- 2026-10-27 · WP-10 · Eval gate holds deterministic-fallback packages to limits + verbatim-quote rules but not AI count-minima (WP-03/04 honesty decision); AI-status content IS held to full §7.2 schemas; the LLM-judge reports BLOCKED (never faked) until valid keys exist · consistent with "never fake metrics" · padding fallback content to pass counts
+- 2026-10-27 · WP-10 · Cutover is scripted but NOT auto-executed: scripts/cutover.sh + docs/relaunch/CUTOVER-CHECKLIST.md gate the live flip on owner keys + live eval gate; moving site/ earlier would break the currently-serving deployment · immediate move · per-piece manual steps
 - 2026-10-27 · WP-01 · `/dev/components` ships in every build; "dev only" is enforced by noindex metadata + zero inbound links (no nav/sitemap entry) · WP-00 fixed `pnpm e2e` to test the production build via `next start`, so a NODE_ENV or env-var gate would break the axe e2e locally and in CI · gating the route on `NODE_ENV=development`; a `NEXT_PUBLIC_*` flag set in the CI e2e step
 - 2026-10-27 · WP-01 · Toast = shadcn `sonner` with the Toaster pinned bottom-right + 4 s (§5.3); dropped the generated `next-themes` wiring, sonner `theme="system"` follows prefers-color-scheme like every other component · the registry has no radix `toast` for the radix style ("only available for Base UI projects" per CLI) and sonner is shadcn's current default · hand-rolled wrapper on radix Toast primitives
 - 2026-10-27 · WP-01 · shadcn `--muted-foreground` maps to §5.2 `--body`, not `--faint`: faint on surface-2/3 sits at/below the 4.5:1 AA floor (4.51:1 / ~4.2:1) and muted-foreground renders on muted surfaces (dialog footers, segmented tabs); faint remains in use for text on bg/surface (4.83:1) via custom components · acceptance says fix the component mapping, not the spec tokens · mapping muted-foreground to faint and flagging the AA miss in the PR

web/evals/gate.ts

@@ -0,0 +1,186 @@
+// WP-10 eval gate (04-work-packages.md): runs the FULL generation pipeline
+// over every transcript in evals/transcripts/ and checks the deterministic
+// gates. The LLM-judge gate needs a live second provider (different from the
+// generator) — with no valid AI keys configured it is reported as BLOCKED,
+// never faked. Import from vitest (tests/eval/gate.test.ts) or via `pnpm eval`.
+import { readdirSync, readFileSync } from "node:fs";
+import { join } from "node:path";
+import { parseTranscript } from "@/lib/transcripts";
+import { generateArtifact } from "@/lib/ai/generate";
+import { ARTIFACT_KINDS, artifactContentSchema, type ArtifactKind } from "@/lib/artifacts/schemas";
+import { religionViolations } from "@/lib/artifacts/postcheck";
+import { DEFAULT_VOICE_PROFILE } from "@/lib/ai/voice";
+import type { Segment } from "@/lib/quotes/types";
+
+export interface EpisodeFixture {
+  name: string;
+  segments: Segment[];
+  hasTimestamps: boolean;
+}
+
+export function loadFixtures(dir: string): EpisodeFixture[] {
+  return readdirSync(dir)
+    .filter((f) => f.endsWith(".txt") && f !== "README.md")
+    .map((f) => {
+      const text = readFileSync(join(dir, f), "utf8");
+      const parsed = parseTranscript(text);
+      return {
+        name: f.replace(/\.txt$/, ""),
+        segments: parsed.segments.map((s, idx) => ({ id: `seg_${String(idx).padStart(6, "0")}`, idx, speaker: s.speaker ?? null, startMs: s.startMs ?? null, endMs: s.endMs ?? null, text: s.text })),
+        hasTimestamps: parsed.hasTimestamps,
+      };
+    });
+}
+
+export interface GateResult {
+  fixture: string;
+  kind: ArtifactKind;
+  ok: boolean;
+  status: "ai" | "fallback";
+  durationMs: number;
+  costUsd: number;
+  schemaValid: boolean;
+  quoteIntegrity: boolean;
+  bannedWords: string[];
+}
+
+/**
+ * Limit compliance for deterministic fallback packages: caps only (§7.2
+ * maxima). Fallback content is deliberately not padded to AI minima
+ * (DECISIONS.md WP-03/WP-04: honesty beats schema padding), so the gate holds
+ * it to the limits that matter for publishing, not the AI counts.
+ */
+const CAPS: Record<string, (c: Record<string, unknown>) => boolean> = {
+  show_notes: (c) => String(c.hook ?? "").length <= 600 && (Array.isArray(c.bullets) ? c.bullets.length <= 7 : true),
+  chapters: (c) => (Array.isArray(c.items) ? c.items.every((i) => String((i as { title: string }).title).length <= 50) : true),
+  summary: (c) => String(c.oneLiner ?? "").length <= 140,
+  titles: (c) => (Array.isArray(c.items) ? c.items.every((i) => String((i as { text: string }).text).length <= 80) : true),
+  blog: (c) => String(c.seoTitle ?? "").length <= 60 && String(c.metaDescription ?? "").length <= 155,
+  newsletter: (c) => String(c.previewText ?? "").length <= 90 && String(c.ctaText ?? "").length <= 30,
+  social: (c) =>
+    String(c.linkedin ?? "").length <= 1300 &&
+    (Array.isArray(c.thread) ? c.thread.every((p) => String(p).length <= 280) : true) &&
+    String((c.instagram as { caption?: string } | undefined)?.caption ?? "").length <= 2200,
+  quotes: () => true,
+};
+
+export interface GateSummary {
+  fixtures: number;
+  episodes: GateResult[];
+  schemaCompliance: number; // 0..1, gate ≥ 0.98
+  quoteIntegrity: number; // 0..1, gate = 1.0
+  bannedWordLeaks: number; // gate = 0
+  p90DurationMs: number; // gate ≤ 5 min
+  maxCostPerEpisode: number; // gate ≤ $0.50
+  judge: { status: "blocked" | "ok"; reason?: string };
+}
+
+const QUOTE_KINDS: Partial<Record<ArtifactKind, string>> = {
+  quotes: "items",
+  show_notes: "quoteRefs",
+  blog: "quoteRefs",
+  newsletter: "quoteRefs",
+};
+
+function quoteIntegrityOf(kind: ArtifactKind, content: unknown, segments: Segment[], requireQuotes: boolean): boolean {
+  const field = QUOTE_KINDS[kind];
+  if (!field) return true;
+  const refs = (content as Record<string, unknown>)?.[field];
+  // Fallback packages may ship without embedded quotes (no padding, WP-03/04);
+  // only the dedicated quotes artifact must always carry at least one. Every
+  // ref that IS present must be a verbatim 40–240-char transcript slice.
+  if (!Array.isArray(refs) || refs.length === 0) return !requireQuotes;
+  const source = segments.map((s) => s.text).join("\n");
+  return refs.every((ref) => {
+    const r = ref as { segmentId: string; start: number; end: number; endSegmentId?: string };
+    const seg = segments.find((s) => s.id === r.segmentId);
+    if (!seg) return false;
+    const text = seg.text.slice(r.start, r.end);
+    return text.length >= 40 && text.length <= 240 && source.includes(text);
+  });
+}
+
+function bannedWordsIn(content: unknown): string[] {
+  return religionViolations(JSON.stringify(content));
+}
+
+function percentile(nums: number[], p: number): number {
+  const sorted = [...nums].sort((a, b) => a - b);
+  return sorted[Math.min(sorted.length - 1, Math.floor((p / 100) * sorted.length))] ?? 0;
+}
+
+export async function runGate(fixtures: EpisodeFixture[], opts?: { timeoutMs?: number }): Promise<GateSummary> {
+  const episodes: GateResult[] = [];
+  const perEpisode: Map<string, GateResult[]> = new Map();
+
+  for (const fx of fixtures) {
+    const results: GateResult[] = [];
+    const started = Date.now();
+    for (const kind of ARTIFACT_KINDS) {
+      if (kind === "chapters" && !fx.segments.some((s) => s.startMs !== null)) {
+        continue; // chapters only when timestamps exist (M4)
+      }
+      const t0 = Date.now();
+      const res = await generateArtifact({
+        kind,
+        segments: fx.segments,
+        show: { name: `Eval Show — ${fx.name}` },
+        voice: DEFAULT_VOICE_PROFILE,
+        deps: { timeoutMs: opts?.timeoutMs },
+      });
+      const durationMs = Date.now() - t0;
+      const schemaValid =
+        res.status === "ai"
+          ? artifactContentSchema(kind).safeParse(res.content).success
+          : (CAPS[kind] ?? (() => true))(res.content as Record<string, unknown>);
+      results.push({
+        fixture: fx.name,
+        kind,
+        ok: schemaValid,
+        status: res.status,
+        durationMs,
+        costUsd: res.costUsd,
+        schemaValid,
+        quoteIntegrity: quoteIntegrityOf(kind, res.content, fx.segments, kind === "quotes"),
+        bannedWords: bannedWordsIn(res.content),
+      });
+    }
+    void started;
+    perEpisode.set(fx.name, results);
+    episodes.push(...results);
+  }
+
+  const schemaOk = episodes.filter((e) => e.schemaValid).length;
+  const quoteKinds = episodes.filter((e) => QUOTE_KINDS[e.kind]);
+  const quoteOk = quoteKinds.filter((e) => e.quoteIntegrity).length;
+
+  const episodeDurations = [...perEpisode.values()].map((rs) => rs.reduce((sum, r) => sum + r.durationMs, 0));
+  const episodeCosts = [...perEpisode.values()].map((rs) => rs.reduce((sum, r) => sum + r.costUsd, 0));
+
+  const anyAi = episodes.some((e) => e.status === "ai");
+  return {
+    fixtures: fixtures.length,
+    episodes,
+    schemaCompliance: episodes.length ? schemaOk / episodes.length : 0,
+    quoteIntegrity: quoteKinds.length ? quoteOk / quoteKinds.length : 1,
+    bannedWordLeaks: episodes.reduce((n, e) => n + e.bannedWords.length, 0),
+    p90DurationMs: percentile(episodeDurations, 90),
+    maxCostPerEpisode: Math.max(0, ...episodeCosts),
+    // The judge must run on a different provider than the generator (spec).
+    // With no valid keys every generation is deterministic-fallback, so a
+    // faithfulness score would be a fabrication — blocked, not faked.
+    judge: anyAi
+      ? { status: "blocked", reason: "judge provider keys not configured" }
+      : { status: "blocked", reason: "no valid AI keys — all episodes generated via deterministic fallback; owner must add GLM/Groq keys and re-run (FOR-ABID.md §2)" },
+  };
+}
+
+export function gatesPass(summary: GateSummary): { pass: boolean; failures: string[] } {
+  const failures: string[] = [];
+  if (summary.quoteIntegrity < 1) failures.push(`quote integrity ${(summary.quoteIntegrity * 100).toFixed(1)}% < 100%`);
+  if (summary.bannedWordLeaks > 0) failures.push(`banned-word leaks: ${summary.bannedWordLeaks}`);
+  if (summary.schemaCompliance < 0.98) failures.push(`limit compliance ${(summary.schemaCompliance * 100).toFixed(1)}% < 98%`);
+  if (summary.p90DurationMs > 5 * 60_000) failures.push(`p90 ${Math.round(summary.p90DurationMs / 1000)}s > 300s`);
+  if (summary.maxCostPerEpisode > 0.5) failures.push(`cost $${summary.maxCostPerEpisode.toFixed(2)} > $0.50`);
+  return { pass: failures.length === 0, failures };
+}

web/evals/transcripts/README.md

@@ -0,0 +1,8 @@
+# Eval transcripts (WP-10)
+
+Fixture transcripts for the eval gate. `demo.txt` is the real sample
+(docs/samples/demo-episode.txt). The other seven are SYNTHETIC fixtures
+written for this repo (plain invented speech, no real persons or shows) so
+the gate can run before the owner records/collects CC0 episodes; replace
+them 1:1 with real recordings by dropping files with the same shape
+(speakers/timestamps where noted) and re-running `pnpm eval`.

web/evals/transcripts/demo.txt

@@ -0,0 +1,58 @@
+Welcome back to the show. Today I want to talk about focus, because focus is the
+most expensive resource in modern work and almost nobody manages it like money.
+We budget our dollars, we calendar our hours, and then we let anyone with a
+notification spend our attention for free.
+
+Here is the uncomfortable truth about deep work: it is not a personality trait,
+it is an environment you build. Every one of us knows the version of ourselves
+that opens a document and actually finishes it, and we all know the version that
+checks one message and surfaces forty minutes later wondering where the morning
+went. The difference between those two people is rarely discipline; it is
+friction. Notice how the distracted version of you is never one decision deep —
+you made one small permissive choice and the environment did the rest.
+
+Because attention compounds, the cost of an interruption is not the five
+minutes it takes. The research on task switching says it can take over twenty
+minutes to get back to full depth, which means a single ping can quietly tax
+the entire morning. Remember that the next time someone says it will only take
+a second: seconds are how attention debt is issued, and the interest rate is
+brutal.
+
+So what actually works? The key is deciding before the day starts. Calendars
+beat to-do lists because a to-do item is a wish while a calendar block is a
+promise. Imagine looking at your Tuesday and seeing two ninety-minute blocks
+with your name on them, notifications off, one tab open; for most people that
+single change doubles their output before lunch, because output was never the
+problem — protected time was.
+
+And let us be honest about the tools, because this is the part nobody wants to
+hear. Every app that promises to organize your attention is also competing for
+it. Therefore the real upgrade is usually subtraction: fewer inputs, fewer
+tabs, fewer meetings that could have been a paragraph. The professional with
+three tools and a clear calendar will out-ship the one with thirty tools and a
+badge for organizing them all.
+
+There is also a social layer we skip. Focus is treated as a personal virtue,
+which makes it feel rude to defend; so we let the loudest agenda in the room
+spend everyone's attention. Boundaries are the fix, and boundaries sound like
+scripts: I am offline until noon; send it and I will get there this afternoon;
+let us make this a document instead of a meeting. Saying them out loud once
+costs less than resenting the interruption for a month.
+
+So here is the challenge for this week. Pick one ninety-minute block, put it on
+the calendar, silence everything, and work on the single thing that has been
+sitting on your list the longest. Notice how it feels; most people tell me it
+feels like getting a whole extra day back. Then keep the block. The version of
+your week with protected focus in it is the version that compounds.
+
+Because here is the takeaway I want you to carry out of this episode: your
+attention is the budget everything else is bought with. Spend it like it is
+yours, because it is.
+
+Thanks for listening; show notes and the full transcript are on the site, and
+if this helped, send it to the person who keeps pinging you at nine in the
+morning.
+
+---
+License: CC0 (no rights reserved). Original text written as a deterministic demo
+input for EpisodeKit — you may freely reuse, modify, and republish it.