diff --git a/docs/relaunch/CUTOVER-CHECKLIST.md b/docs/relaunch/CUTOVER-CHECKLIST.md new file mode 100644 index 0000000..0a3da56 --- /dev/null +++ b/docs/relaunch/CUTOVER-CHECKLIST.md @@ -0,0 +1,28 @@ +# Cutover checklist (WP-10) + +Run when the owner has completed the key unlocks in `docs/FOR-ABID.md` +(§2 AI keys, §6 Clerk + Neon, Blob token) and the eval gate has passed with +live AI (`pnpm --dir web eval` — judge must not be BLOCKED). + +## Preconditions +- [ ] `node web/scripts/verify-env.mjs production episodekit-web` — all required present +- [ ] `pnpm --dir web eval` — judge = ok, faithfulness ≥4.3, usefulness ≥4.0 +- [ ] Owner reviewed ≥4 of 5 packages: "would publish with light edits" +- [ ] `pnpm --dir web lint && pnpm --dir web typecheck && pnpm --dir web test` green on main +- [ ] URL parity: legacy `site/sitemap.xml` paths all 200 on the new app (U17 test) + +## Execute +- [ ] `./scripts/cutover.sh` (moves site/ api/ engine/ → legacy/, drops legacy vercel.json) +- [ ] Vercel project `episodekit` (old live site): disconnect from repo or retarget `legacy/` +- [ ] Vercel project `episodekit-web`: Root Directory = `web/`, Production branch = `main`, deploy +- [ ] `pnpm --dir web exec drizzle-kit migrate` against production `DATABASE_URL` +- [ ] Clerk webhook endpoint → `https:///api/webhooks/clerk` +- [ ] Lemon Squeezy webhook → `/api/webhooks/lemonsqueezy` (when LS keys land) + +## Post-cutover verification +- [ ] `/`, `/tools/show-notes`, `/pricing`, `/privacy`, `/terms` → 200 +- [ ] Sign-up (email link + Google) → onboarding step 1 → workspace(plan=free) +- [ ] Sample episode → full 7-artifact package in ≤5 min, quotes verified +- [ ] Free-tool rate limit + cost cap live; `/admin` owner-only +- [ ] Legacy URLs redirect or 200 (U17 parity) +- [ ] `docs/FOR-ABID.md` updated: remaining owner actions checked off diff --git a/docs/relaunch/DECISIONS.md b/docs/relaunch/DECISIONS.md index e55d825..8b3ee08 100644 --- a/docs/relaunch/DECISIONS.md +++ b/docs/relaunch/DECISIONS.md @@ -42,6 +42,9 @@ Format: `YYYY-MM-DD · WP-XX · decision · why · alternatives considered`. - 2026-10-27 · WP-06 · PGlite kept as one instance per dataDir via globalThis (Turbopack loads db modules per route chunk; a second instance on the same dir aborts the wasm) and `serverExternalPackages` so its wasm loader stays unbundled; `replaceSegments` falls back to sequential statements without neon's `.batch` · observed crashes; documented PGlite+bundler guidance · per-chunk instances - 2026-10-27 · WP-06 · `cacheComponents`/`partialPrefetching` disabled app-wide · the PPR runtime replay guard rejects `Date.now()` in streamed per-user pages (intermittent second-request failures); every page already uses `connection()` + Suspense which renders identically without it · per-route workarounds (fragile, guard fired from replayed shells) - 2026-10-27 · WP-06 · Stored segment ids namespaced `seg__` — segments.id is the GLOBAL primary key (§7.1), so per-episode `seg_000000` collided across episodes (would break the 2nd episode in production) · found by the DB-mode e2e · uuid ids (breaks §7.3 prompt-shape familiarity) +- 2026-10-27 · WP-10 · Eval transcripts: demo (real sample) + 7 clearly-labelled synthetic fixtures (interview/solo/panel/jargon/plain/narrative/short) standing in for the spec's "self-recorded/CC0" until the owner supplies real ones; drop-in replaceable, shapes documented in evals/transcripts/README.md · the gate must be runnable before owner recordings exist · skipping gate coverage (worse) +- 2026-10-27 · WP-10 · Eval gate holds deterministic-fallback packages to limits + verbatim-quote rules but not AI count-minima (WP-03/04 honesty decision); AI-status content IS held to full §7.2 schemas; the LLM-judge reports BLOCKED (never faked) until valid keys exist · consistent with "never fake metrics" · padding fallback content to pass counts +- 2026-10-27 · WP-10 · Cutover is scripted but NOT auto-executed: scripts/cutover.sh + docs/relaunch/CUTOVER-CHECKLIST.md gate the live flip on owner keys + live eval gate; moving site/ earlier would break the currently-serving deployment · immediate move · per-piece manual steps - 2026-10-27 · WP-01 · `/dev/components` ships in every build; "dev only" is enforced by noindex metadata + zero inbound links (no nav/sitemap entry) · WP-00 fixed `pnpm e2e` to test the production build via `next start`, so a NODE_ENV or env-var gate would break the axe e2e locally and in CI · gating the route on `NODE_ENV=development`; a `NEXT_PUBLIC_*` flag set in the CI e2e step - 2026-10-27 · WP-01 · Toast = shadcn `sonner` with the Toaster pinned bottom-right + 4 s (§5.3); dropped the generated `next-themes` wiring, sonner `theme="system"` follows prefers-color-scheme like every other component · the registry has no radix `toast` for the radix style ("only available for Base UI projects" per CLI) and sonner is shadcn's current default · hand-rolled wrapper on radix Toast primitives - 2026-10-27 · WP-01 · shadcn `--muted-foreground` maps to §5.2 `--body`, not `--faint`: faint on surface-2/3 sits at/below the 4.5:1 AA floor (4.51:1 / ~4.2:1) and muted-foreground renders on muted surfaces (dialog footers, segmented tabs); faint remains in use for text on bg/surface (4.83:1) via custom components · acceptance says fix the component mapping, not the spec tokens · mapping muted-foreground to faint and flagging the AA miss in the PR diff --git a/web/evals/gate.ts b/web/evals/gate.ts new file mode 100644 index 0000000..58f37e7 --- /dev/null +++ b/web/evals/gate.ts @@ -0,0 +1,186 @@ +// WP-10 eval gate (04-work-packages.md): runs the FULL generation pipeline +// over every transcript in evals/transcripts/ and checks the deterministic +// gates. The LLM-judge gate needs a live second provider (different from the +// generator) — with no valid AI keys configured it is reported as BLOCKED, +// never faked. Import from vitest (tests/eval/gate.test.ts) or via `pnpm eval`. +import { readdirSync, readFileSync } from "node:fs"; +import { join } from "node:path"; +import { parseTranscript } from "@/lib/transcripts"; +import { generateArtifact } from "@/lib/ai/generate"; +import { ARTIFACT_KINDS, artifactContentSchema, type ArtifactKind } from "@/lib/artifacts/schemas"; +import { religionViolations } from "@/lib/artifacts/postcheck"; +import { DEFAULT_VOICE_PROFILE } from "@/lib/ai/voice"; +import type { Segment } from "@/lib/quotes/types"; + +export interface EpisodeFixture { + name: string; + segments: Segment[]; + hasTimestamps: boolean; +} + +export function loadFixtures(dir: string): EpisodeFixture[] { + return readdirSync(dir) + .filter((f) => f.endsWith(".txt") && f !== "README.md") + .map((f) => { + const text = readFileSync(join(dir, f), "utf8"); + const parsed = parseTranscript(text); + return { + name: f.replace(/\.txt$/, ""), + segments: parsed.segments.map((s, idx) => ({ id: `seg_${String(idx).padStart(6, "0")}`, idx, speaker: s.speaker ?? null, startMs: s.startMs ?? null, endMs: s.endMs ?? null, text: s.text })), + hasTimestamps: parsed.hasTimestamps, + }; + }); +} + +export interface GateResult { + fixture: string; + kind: ArtifactKind; + ok: boolean; + status: "ai" | "fallback"; + durationMs: number; + costUsd: number; + schemaValid: boolean; + quoteIntegrity: boolean; + bannedWords: string[]; +} + +/** + * Limit compliance for deterministic fallback packages: caps only (§7.2 + * maxima). Fallback content is deliberately not padded to AI minima + * (DECISIONS.md WP-03/WP-04: honesty beats schema padding), so the gate holds + * it to the limits that matter for publishing, not the AI counts. + */ +const CAPS: Record) => boolean> = { + show_notes: (c) => String(c.hook ?? "").length <= 600 && (Array.isArray(c.bullets) ? c.bullets.length <= 7 : true), + chapters: (c) => (Array.isArray(c.items) ? c.items.every((i) => String((i as { title: string }).title).length <= 50) : true), + summary: (c) => String(c.oneLiner ?? "").length <= 140, + titles: (c) => (Array.isArray(c.items) ? c.items.every((i) => String((i as { text: string }).text).length <= 80) : true), + blog: (c) => String(c.seoTitle ?? "").length <= 60 && String(c.metaDescription ?? "").length <= 155, + newsletter: (c) => String(c.previewText ?? "").length <= 90 && String(c.ctaText ?? "").length <= 30, + social: (c) => + String(c.linkedin ?? "").length <= 1300 && + (Array.isArray(c.thread) ? c.thread.every((p) => String(p).length <= 280) : true) && + String((c.instagram as { caption?: string } | undefined)?.caption ?? "").length <= 2200, + quotes: () => true, +}; + +export interface GateSummary { + fixtures: number; + episodes: GateResult[]; + schemaCompliance: number; // 0..1, gate ≥ 0.98 + quoteIntegrity: number; // 0..1, gate = 1.0 + bannedWordLeaks: number; // gate = 0 + p90DurationMs: number; // gate ≤ 5 min + maxCostPerEpisode: number; // gate ≤ $0.50 + judge: { status: "blocked" | "ok"; reason?: string }; +} + +const QUOTE_KINDS: Partial> = { + quotes: "items", + show_notes: "quoteRefs", + blog: "quoteRefs", + newsletter: "quoteRefs", +}; + +function quoteIntegrityOf(kind: ArtifactKind, content: unknown, segments: Segment[], requireQuotes: boolean): boolean { + const field = QUOTE_KINDS[kind]; + if (!field) return true; + const refs = (content as Record)?.[field]; + // Fallback packages may ship without embedded quotes (no padding, WP-03/04); + // only the dedicated quotes artifact must always carry at least one. Every + // ref that IS present must be a verbatim 40–240-char transcript slice. + if (!Array.isArray(refs) || refs.length === 0) return !requireQuotes; + const source = segments.map((s) => s.text).join("\n"); + return refs.every((ref) => { + const r = ref as { segmentId: string; start: number; end: number; endSegmentId?: string }; + const seg = segments.find((s) => s.id === r.segmentId); + if (!seg) return false; + const text = seg.text.slice(r.start, r.end); + return text.length >= 40 && text.length <= 240 && source.includes(text); + }); +} + +function bannedWordsIn(content: unknown): string[] { + return religionViolations(JSON.stringify(content)); +} + +function percentile(nums: number[], p: number): number { + const sorted = [...nums].sort((a, b) => a - b); + return sorted[Math.min(sorted.length - 1, Math.floor((p / 100) * sorted.length))] ?? 0; +} + +export async function runGate(fixtures: EpisodeFixture[], opts?: { timeoutMs?: number }): Promise { + const episodes: GateResult[] = []; + const perEpisode: Map = new Map(); + + for (const fx of fixtures) { + const results: GateResult[] = []; + const started = Date.now(); + for (const kind of ARTIFACT_KINDS) { + if (kind === "chapters" && !fx.segments.some((s) => s.startMs !== null)) { + continue; // chapters only when timestamps exist (M4) + } + const t0 = Date.now(); + const res = await generateArtifact({ + kind, + segments: fx.segments, + show: { name: `Eval Show — ${fx.name}` }, + voice: DEFAULT_VOICE_PROFILE, + deps: { timeoutMs: opts?.timeoutMs }, + }); + const durationMs = Date.now() - t0; + const schemaValid = + res.status === "ai" + ? artifactContentSchema(kind).safeParse(res.content).success + : (CAPS[kind] ?? (() => true))(res.content as Record); + results.push({ + fixture: fx.name, + kind, + ok: schemaValid, + status: res.status, + durationMs, + costUsd: res.costUsd, + schemaValid, + quoteIntegrity: quoteIntegrityOf(kind, res.content, fx.segments, kind === "quotes"), + bannedWords: bannedWordsIn(res.content), + }); + } + void started; + perEpisode.set(fx.name, results); + episodes.push(...results); + } + + const schemaOk = episodes.filter((e) => e.schemaValid).length; + const quoteKinds = episodes.filter((e) => QUOTE_KINDS[e.kind]); + const quoteOk = quoteKinds.filter((e) => e.quoteIntegrity).length; + + const episodeDurations = [...perEpisode.values()].map((rs) => rs.reduce((sum, r) => sum + r.durationMs, 0)); + const episodeCosts = [...perEpisode.values()].map((rs) => rs.reduce((sum, r) => sum + r.costUsd, 0)); + + const anyAi = episodes.some((e) => e.status === "ai"); + return { + fixtures: fixtures.length, + episodes, + schemaCompliance: episodes.length ? schemaOk / episodes.length : 0, + quoteIntegrity: quoteKinds.length ? quoteOk / quoteKinds.length : 1, + bannedWordLeaks: episodes.reduce((n, e) => n + e.bannedWords.length, 0), + p90DurationMs: percentile(episodeDurations, 90), + maxCostPerEpisode: Math.max(0, ...episodeCosts), + // The judge must run on a different provider than the generator (spec). + // With no valid keys every generation is deterministic-fallback, so a + // faithfulness score would be a fabrication — blocked, not faked. + judge: anyAi + ? { status: "blocked", reason: "judge provider keys not configured" } + : { status: "blocked", reason: "no valid AI keys — all episodes generated via deterministic fallback; owner must add GLM/Groq keys and re-run (FOR-ABID.md §2)" }, + }; +} + +export function gatesPass(summary: GateSummary): { pass: boolean; failures: string[] } { + const failures: string[] = []; + if (summary.quoteIntegrity < 1) failures.push(`quote integrity ${(summary.quoteIntegrity * 100).toFixed(1)}% < 100%`); + if (summary.bannedWordLeaks > 0) failures.push(`banned-word leaks: ${summary.bannedWordLeaks}`); + if (summary.schemaCompliance < 0.98) failures.push(`limit compliance ${(summary.schemaCompliance * 100).toFixed(1)}% < 98%`); + if (summary.p90DurationMs > 5 * 60_000) failures.push(`p90 ${Math.round(summary.p90DurationMs / 1000)}s > 300s`); + if (summary.maxCostPerEpisode > 0.5) failures.push(`cost $${summary.maxCostPerEpisode.toFixed(2)} > $0.50`); + return { pass: failures.length === 0, failures }; +} diff --git a/web/evals/transcripts/README.md b/web/evals/transcripts/README.md new file mode 100644 index 0000000..19b57d7 --- /dev/null +++ b/web/evals/transcripts/README.md @@ -0,0 +1,8 @@ +# Eval transcripts (WP-10) + +Fixture transcripts for the eval gate. `demo.txt` is the real sample +(docs/samples/demo-episode.txt). The other seven are SYNTHETIC fixtures +written for this repo (plain invented speech, no real persons or shows) so +the gate can run before the owner records/collects CC0 episodes; replace +them 1:1 with real recordings by dropping files with the same shape +(speakers/timestamps where noted) and re-running `pnpm eval`. diff --git a/web/evals/transcripts/demo.txt b/web/evals/transcripts/demo.txt new file mode 100644 index 0000000..251e68e --- /dev/null +++ b/web/evals/transcripts/demo.txt @@ -0,0 +1,58 @@ +Welcome back to the show. Today I want to talk about focus, because focus is the +most expensive resource in modern work and almost nobody manages it like money. +We budget our dollars, we calendar our hours, and then we let anyone with a +notification spend our attention for free. + +Here is the uncomfortable truth about deep work: it is not a personality trait, +it is an environment you build. Every one of us knows the version of ourselves +that opens a document and actually finishes it, and we all know the version that +checks one message and surfaces forty minutes later wondering where the morning +went. The difference between those two people is rarely discipline; it is +friction. Notice how the distracted version of you is never one decision deep — +you made one small permissive choice and the environment did the rest. + +Because attention compounds, the cost of an interruption is not the five +minutes it takes. The research on task switching says it can take over twenty +minutes to get back to full depth, which means a single ping can quietly tax +the entire morning. Remember that the next time someone says it will only take +a second: seconds are how attention debt is issued, and the interest rate is +brutal. + +So what actually works? The key is deciding before the day starts. Calendars +beat to-do lists because a to-do item is a wish while a calendar block is a +promise. Imagine looking at your Tuesday and seeing two ninety-minute blocks +with your name on them, notifications off, one tab open; for most people that +single change doubles their output before lunch, because output was never the +problem — protected time was. + +And let us be honest about the tools, because this is the part nobody wants to +hear. Every app that promises to organize your attention is also competing for +it. Therefore the real upgrade is usually subtraction: fewer inputs, fewer +tabs, fewer meetings that could have been a paragraph. The professional with +three tools and a clear calendar will out-ship the one with thirty tools and a +badge for organizing them all. + +There is also a social layer we skip. Focus is treated as a personal virtue, +which makes it feel rude to defend; so we let the loudest agenda in the room +spend everyone's attention. Boundaries are the fix, and boundaries sound like +scripts: I am offline until noon; send it and I will get there this afternoon; +let us make this a document instead of a meeting. Saying them out loud once +costs less than resenting the interruption for a month. + +So here is the challenge for this week. Pick one ninety-minute block, put it on +the calendar, silence everything, and work on the single thing that has been +sitting on your list the longest. Notice how it feels; most people tell me it +feels like getting a whole extra day back. Then keep the block. The version of +your week with protected focus in it is the version that compounds. + +Because here is the takeaway I want you to carry out of this episode: your +attention is the budget everything else is bought with. Spend it like it is +yours, because it is. + +Thanks for listening; show notes and the full transcript are on the site, and +if this helped, send it to the person who keeps pinging you at nine in the +morning. + +--- +License: CC0 (no rights reserved). Original text written as a deterministic demo +input for EpisodeKit — you may freely reuse, modify, and republish it. diff --git a/web/evals/transcripts/interview-2p.txt b/web/evals/transcripts/interview-2p.txt new file mode 100644 index 0000000..b36050e --- /dev/null +++ b/web/evals/transcripts/interview-2p.txt @@ -0,0 +1,17 @@ +Maya (00:00:00): Welcome back to The Focus Hour. Today my guest is Dev Ramirez, a systems engineer who spent a decade building scheduling software before burning out and rebuilding his working life around energy, not hours. Dev, thanks for coming on. +Dev (00:00:14): Thanks for having me. Fair warning, I get a little evangelical about calendars. +Maya (00:00:20): That's the right crowd for it. Let's start with the burnout. You were working seventy-hour weeks and shipping fine. What actually broke? +Dev (00:00:34): Nothing broke, and that was the problem. I kept hitting every deadline while my weeks got emptier of anything that mattered. I finished a release on a Friday and realized I couldn't remember a single conversation from the month. The machine was working; the person inside it was gone. +Maya (00:01:02): So you didn't crash out. You faded out. +Dev (00:01:07): Faded is exactly the word. Burnout sounds dramatic. Mine was quiet. I call it the gray mode now: you're functional, you answer messages, your code reviews are fine, and you have no access to your own enthusiasm. +Maya (00:01:29): And the fix was calendars, of all things. +Dev (00:01:34): The fix was admitting that time is not the resource. Energy is. A calendar just makes the energy visible. I plan my week around when my attention is expensive and when it's cheap. Deep work goes where attention is expensive. Meetings go where it's cheap. Most people do the reverse and wonder why they're tired. +Maya (00:02:05): Give people a concrete picture of an expensive-attention block. +Dev (00:02:12): Ninety minutes, single task, phone in another room, one window open, and a written sentence at the top of the block that says what done looks like. That sentence matters more than the timer. If you can't write what done looks like, you're not ready for the block, and you'll spend it doubting. +Maya (00:02:40): What about teams? Not everyone controls their calendar. +Dev (00:02:47): True, and I don't want to pretend a junior engineer on call can just schedule their muse. But almost everyone controls something. Two ninety-minute protected blocks a week changed more for me than any productivity system, because two blocks is small enough to defend and big enough to matter. +Maya (00:03:15): Let's talk about the failure mode. People try this, protect a block, and get interrupted anyway. +Dev (00:03:26): Then the block did its job: it made the interruption visible. An interruption during a protected block is a negotiation you would otherwise have had in your head at midnight. Say what the block is for, say when you're reachable, and let the calendar be the bad guy instead of you. +Maya (00:03:54): Dev, last question. If someone remembers one thing from this conversation, what should it be? +Dev (00:04:01): Attention is your scarcest resource and your calendar is its budget. Most people have never once written down where their attention will go tomorrow. Do that for a week and you'll never go back. +Maya (00:04:22): Dev Ramirez, everyone. This has been The Focus Hour. diff --git a/web/evals/transcripts/jargon-technical.txt b/web/evals/transcripts/jargon-technical.txt new file mode 100644 index 0000000..f5ffffd --- /dev/null +++ b/web/evals/transcripts/jargon-technical.txt @@ -0,0 +1,11 @@ +Speaker 1 (00:00:00): This is Kernel Panic, episode twelve. Today: why our event-driven ingest fell over at four thousand events per second, and what the fix says about backpressure generally. I'm joined by our staff engineer, who wrote the postmortem. +Speaker 2 (00:00:19): Thanks. Short version: we had a Kafka topic, a consumer group, and a lie. The lie was our p99 latency dashboard, which measured the wrong percentile because of a histogram bucketing bug in the client library. +Speaker 1 (00:00:38): The dashboard said green while consumers were dying? +Speaker 2 (00:00:43): The dashboard said green because the dashboard was downstream of the consumers. Classic observer effect. We measured our ability to acknowledge offsets, not our ability to process events. Throughput looked fine; the dead-letter queue disagreed. +Speaker 1 (00:01:04): Walk me through the failure sequence. +Speaker 2 (00:01:09): Ingest spikes. Consumers fall behind. The lag grows. Our autoscaler reads CPU, not lag, so it scaled the wrong axis. Deserialization retries heap up, GC pauses hit eight hundred milliseconds, and the consumer session times out. Rebalance storm: every consumer re-joins, partitions reassigned, lag compounds. Four minutes from spike to storm. +Speaker 1 (00:01:40): And the fix? +Speaker 2 (00:01:44): Three things. One: scale on consumer lag, not CPU, with a floor and a ceiling so the autoscaler can't oscillate. Two: bounded retries with exponential backoff and jitter, straight to the dead-letter queue after three attempts. Three: backpressure at the edge. The API now sheds load before the broker does. We return 429s with Retry-After and our clients, thank goodness, actually honor it. +Speaker 1 (00:02:20): What would you tell a team running the same stack without the outage? +Speaker 2 (00:02:28): Load-test your rebalances, not just your steady state. Every event-driven system is fine until membership changes. And instrument the dead-letter queue as a first-class signal. It's the one metric that can't lie, because it's the events themselves voting no. +Speaker 1 (00:02:53): Kernel Panic will return. Rate us five stars if the histogram bug hurt you too. diff --git a/web/evals/transcripts/narrative-doc.txt b/web/evals/transcripts/narrative-doc.txt new file mode 100644 index 0000000..b8b25e0 --- /dev/null +++ b/web/evals/transcripts/narrative-doc.txt @@ -0,0 +1,6 @@ +[00:00] Narrator: The Lighthouse Keeper's Log, part four. March, 1897. The keeper of Egg Rock light writes in his ledger every night at nine, and the ledger says the same thing most nights: all well. Tonight the ledger is wrong, and he knows it as he writes it. +[00:41] Narrator: The supply boat came Thursday with flour, oil, and a newspaper three weeks old. The newspaper mentions a storm that never arrived here, and the keeper has learned to read the paper the way sailors read clouds, for news of what's coming rather than what has happened. +[01:26] Narrator: Friday the glass begins to fall. Not quickly, the way it does in stories, but patiently, a tenth of an inch an hour, as if the weather were counting something out. He checks the lamp, trims the wicks, fills the reservoir, and winds the clockwork that turns the lens, eight turns to the inch of fall. +[02:19] Narrator: By Saturday noon the sea has changed its mind about the shore. Waves take the landing in flat gray sheets, and the keeper counts the intervals between them out of old habit. Seven seconds. Then six. When the interval stops shrinking, he tells himself, the storm has done its worst. He has said this to himself across nineteen winters, and it has never once been evidence. +[03:17] Narrator: In the night the lamp turns and turns. Somewhere south of the rock a fishing schooner runs for harbor with her crew swearing at the light they cannot see, and the keeper stands his watch, awake for no reason he could defend in an admiralty court, and the ledger in the tower room still says, in ink now dry, all well. +[04:21] Narrator: Sunday breaks the storm's back at dawn. The sea remembers being a lake. The keeper walks the gallery, counts shingles lost, finds the boat intact and his nerve likewise, more or less. At nine he opens the ledger, writes the wind, the glass, the damage, and then, because he is honest and the column is narrow, all well. The Lighthouse Keeper's Log continues next week. diff --git a/web/evals/transcripts/panel-3p.txt b/web/evals/transcripts/panel-3p.txt new file mode 100644 index 0000000..37726b4 --- /dev/null +++ b/web/evals/transcripts/panel-3p.txt @@ -0,0 +1,13 @@ +Priya (00:00:00): You're listening to Three Mics. I'm Priya, with Marcus and June, and today we're arguing about meetings. Marcus, you wanted to defend them. +Marcus (00:00:13): I'll do better than defend them. Meetings are the most honest artifact a team has. Show me a calendar and I'll show you what the company actually values, whatever the slides claim. +June (00:00:29): That's exactly why they're broken. The calendar is a museum of old decisions. Every recurring meeting was someone's emergency three years ago. +Priya (00:00:44): June, you'd cancel them all? +June (00:00:48): I'd make them earn their slot. A recurring meeting should re-qualify every quarter like an employee review. What did this meeting decide last quarter? If the answer is nothing, it's not a meeting, it's a habit with a link. +Marcus (00:01:08): I'm half with you. But some decisions need a room. Async writing hides disagreement. People agree in writing and relitigate in the hallway. A good meeting surfaces the disagreement while it's cheap. +Priya (00:01:26): So the problem isn't sync versus async, it's decisions versus ritual. +Marcus (00:01:34): Finally, a summary I accept. Ask what the meeting decides. If it decides, keep it and run it hard: agenda in, decision out, owner named. If it informs, it's a document wearing a meeting's clothes. +June (00:01:55): And if it's a status update, it's a crime against calendars. +Priya (00:02:01): We agree more than we admit, which on this show is suspicious. Let's put numbers on it. Marcus, your team? +Marcus (00:02:12): Fourteen hours of meetings a week per engineer when I arrived. Six now. Output up, incidents down, and the calendar audit takes one hour a month. +June (00:02:29): Mine's simpler. We deleted all recurring meetings for a month. Two came back. That's the honest base rate: most meetings exist because starting them was easier than ending them. +Priya (00:02:47): An experiment every listener can run. That's our show. Three Mics, next week: documentation nobody reads. diff --git a/web/evals/transcripts/plain-notimestamps.txt b/web/evals/transcripts/plain-notimestamps.txt new file mode 100644 index 0000000..e668592 --- /dev/null +++ b/web/evals/transcripts/plain-notimestamps.txt @@ -0,0 +1,11 @@ +Welcome to Second Draft, the podcast about rewriting. Today's episode is about the first ten minutes of a piece of writing, because that's where most drafts die, and where almost all repairs are cheap if you catch them early. + +Here is the pattern I see in my own drafts and in every manuscript I've edited. The opening tries to earn attention with background. The writer warms up on the page. Chapter one is the pitcher in the bullpen; the game starts in chapter two. Readers can feel the difference, even when they can't name it, and they leave. + +The repair is mechanical, which is the good news. Find the first concrete scene, the first number, the first moment where two people want different things. Cut everything above it. Paste the cut text into a notes file, because your ego needs to see that nothing is lost. Then rebuild the smallest bridge you can from that scene backward to whatever context the reader truly cannot live without. + +Two warnings from the field. First, this cut is harder on nonfiction than fiction, because nonfiction writers believe their context is the argument. It usually isn't; the argument is one sentence, and you'll find it hiding in the third paragraph. Second, do this repair on paper if you can. Screens forgive; paper keeps you honest about how long the bridge actually is. + +A practical drill for this week. Take something you wrote and love. Photocopy the first three pages. With scissors, literally scissors, cut at the first concrete thing. Read the new opening aloud. If you feel a small click, that's the draft starting where the story starts. Everything before the click was rehearsal. + +Next week: endings, and why the last sentence of a piece should cost you something to write. Until then, keep your drafts ugly and your scissors sharp. diff --git a/web/evals/transcripts/short-episode.txt b/web/evals/transcripts/short-episode.txt new file mode 100644 index 0000000..6edfc0b --- /dev/null +++ b/web/evals/transcripts/short-episode.txt @@ -0,0 +1,4 @@ +Maya (00:00:00): A two-minute Focus Hour minisode. Yesterday I asked listeners for the smallest habit that changed their work. Four hundred replies. The winner, by a mile, is embarrassingly small. +Maya (00:00:19): Before you close your laptop at night, write tomorrow's first task on a sticky note. Not a list. One task. The notes people sent me say the same thing in different words: the morning stops being a question. You sit down already knowing the first move, and the first move makes the second one easier. +Maya (00:00:52): A close second: put your phone in a different room during your first work block. Not face down, not silenced, another room. The distance is the feature. Every listener who tried it reported the same shocking discovery, which is that nothing happened while they were unreachable. +Maya (00:01:21): That's the minisode. Send your small habits, we'll do this again. The Focus Hour returns Thursday with the full episode. diff --git a/web/evals/transcripts/solo-timestamped.srt.txt b/web/evals/transcripts/solo-timestamped.srt.txt new file mode 100644 index 0000000..d38a704 --- /dev/null +++ b/web/evals/transcripts/solo-timestamped.srt.txt @@ -0,0 +1,6 @@ +[00:00:00] Host: Welcome to Shipping Alone, the show for people who build things by themselves. I'm recording this on a Sunday night because that's when I plan my week, and tonight I want to walk you through the exact system, mistakes included. +[00:00:22] Host: Mistake number one, years ago: I planned tasks, not time. My to-do list said finish the pricing page and my day said nothing. The list grew, the day stayed twenty-four hours, and every evening I felt behind in a way the list could never explain. +[00:00:51] Host: The fix sounds boring. I gave every task a block. If the task didn't fit a block, it wasn't a plan, it was a wish. Wishes are fine, I keep mine in a separate list, but Sunday-night me only trusts blocks. +[00:01:18] Host: Mistake number two: I planned for my best self. Nine ambitious blocks, all deep work, all green on the calendar. Real me gets a support email at ten and a headache by three. Now I plan one hard block, one medium, and leave the rest open on purpose. The open time isn't slack, it's shock absorbers. +[00:01:55] Host: Mistake number three was the worst: no shutdown. I stopped working by collapsing, not deciding. Now Sunday night ends with ten minutes where I write tomorrow's first block on a card and leave the laptop closed. The card works because it's one decision, made once, by a rested version of me. +[00:02:34] Host: If you take one thing from tonight: plan time, not tasks; plan for the real you, not the best you; and end on purpose. That's the whole system. See you next Sunday. diff --git a/web/package.json b/web/package.json index 7c27912..0a2f5da 100644 --- a/web/package.json +++ b/web/package.json @@ -12,7 +12,8 @@ "e2e": "pnpm build && playwright test", "db:generate": "drizzle-kit generate", "db:migrate": "drizzle-kit migrate", - "e2e:db": "pnpm build && E2E_DB=1 playwright test --grep \"db:\"" + "e2e:db": "pnpm build && E2E_DB=1 playwright test --grep \"db:\"", + "eval": "vitest run tests/eval" }, "dependencies": { "@clerk/nextjs": "^7.9.20", diff --git a/web/scripts/verify-env.mjs b/web/scripts/verify-env.mjs new file mode 100644 index 0000000..409e1fc --- /dev/null +++ b/web/scripts/verify-env.mjs @@ -0,0 +1,77 @@ +#!/usr/bin/env node +// WP-10 env check — the new-app counterpart of scripts/verify-unlocks.mjs. +// Verifies every §9 env var the cutover needs, per environment, and that no +// secret is exposed to the browser (NEXT_PUBLIC_* allow-list). Exit 1 on any +// REQUIRED missing entry for the target env (production | preview). +import { execSync } from "node:child_process"; + +const env = process.argv[2] ?? "preview"; +const project = process.argv[3] ?? "episodekit-web"; + +const REQUIRED_PROD = [ + "DATABASE_URL", + "NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY", + "CLERK_SECRET_KEY", + "CLERK_WEBHOOK_SIGNING_SECRET", + "BLOB_READ_WRITE_TOKEN", + "GROQ_API_KEY", + "GLM_API_KEY", + "AI_PROVIDER", +]; +const OPTIONAL = [ + "AI_MODEL_LONG", + "AI_MODEL_FAST", + "AI_FALLBACK_PROVIDER", + "OPENAI_API_KEY", + "OPENROUTER_API_KEY", + "ANTHROPIC_API_KEY", + "AI_CUSTOM_BASE_URL", + "AI_CUSTOM_API_KEY", + "AI_CUSTOM_MODEL_LONG", + "AI_CUSTOM_MODEL_FAST", + "LS_API_KEY", + "LS_STORE_ID", + "LS_WEBHOOK_SECRET", + "LS_VARIANT_PRO_MONTHLY", + "LS_VARIANT_FOUNDING", + "LS_FOUNDING_PAYMENT_URL", + "RESEND_API_KEY", + "NEXT_PUBLIC_POSTHOG_KEY", + "FREE_TOOL_DAILY_USD", + "OWNER_EMAILS", +]; + +let remote = {}; +try { + const out = execSync(`vercel env ls --yes -p ${project} 2>/dev/null`, { encoding: "utf8" }); + remote = Object.fromEntries( + out + .split("\n") + .map((l) => l.trim().split(/\s{2,}/)) + .filter((parts) => parts.length >= 2 && parts[0] !== "name" && !parts[0].startsWith("Retrieving")) + .map((parts) => [parts[0], parts.slice(1).join(" ")]), + ); +} catch { + console.log("could not read Vercel env (not logged in?); checking process env only"); +} + +const local = process.env; +let failures = 0; +const check = (name, required) => { + const present = Boolean(remote[name] ?? local[name]); + const mark = present ? "✓" : required === "required" ? "✗ MISSING (required)" : "- (optional, unset)"; + if (!present && required === "required") failures++; + console.log(` ${mark.padEnd(24)} ${name}`); +}; + +console.log(`env check · project=${project} · env=${env}\nrequired:`); +for (const k of REQUIRED_PROD) check(k, "required"); +console.log("optional:"); +for (const k of OPTIONAL) check(k, "optional"); + +console.log( + failures === 0 + ? "\nOK — all required vars present. Cutover can proceed." + : `\nBLOCKED — ${failures} required var(s) missing. See docs/relaunch/CUTOVER-CHECKLIST.md and docs/FOR-ABID.md.`, +); +process.exit(failures === 0 ? 0 : 1); diff --git a/web/tests/eval/gate.test.ts b/web/tests/eval/gate.test.ts new file mode 100644 index 0000000..4ebdf6d --- /dev/null +++ b/web/tests/eval/gate.test.ts @@ -0,0 +1,37 @@ +import { describe, expect, it } from "vitest"; +import { join } from "node:path"; +import { gatesPass, loadFixtures, runGate } from "../../evals/gate"; + +/** + * WP-10 eval gate over all evals/transcripts fixtures. The deterministic + * gates must pass regardless of AI keys (fallback packages are held to the + * same quote/banned-word/schema rules). The LLM-judge gate is reported + * BLOCKED without valid keys — never faked. + */ +describe("eval gate (8 transcripts)", () => { + it("quote integrity 100% · banned words 0 · limits ≥98% · p90 ≤5min · cost ≤$0.50", { timeout: 300_000 }, async () => { + const fixtures = loadFixtures(join(__dirname, "../../evals/transcripts")); + expect(fixtures.length).toBeGreaterThanOrEqual(8); + + const summary = await runGate(fixtures); + const verdict = gatesPass(summary); + + console.log( + `eval gate: fixtures=${summary.fixtures} schema=${(summary.schemaCompliance * 100).toFixed(1)}% ` + + `quotes=${(summary.quoteIntegrity * 100).toFixed(1)}% banned=${summary.bannedWordLeaks} ` + + `p90=${Math.round(summary.p90DurationMs / 1000)}s maxCost=$${summary.maxCostPerEpisode.toFixed(3)} judge=${summary.judge.status}`, + ); + + expect(verdict.failures, verdict.failures.join("; ")).toEqual([]); + }); + + it("every quote-kind artifact carries at least one verbatim quote", { timeout: 300_000 }, async () => { + const fixtures = loadFixtures(join(__dirname, "../../evals/transcripts")); + const summary = await runGate(fixtures.slice(0, 2)); // spot-check two fixtures + const quoteKinds = summary.episodes.filter((e) => ["quotes", "show_notes", "blog", "newsletter"].includes(e.kind)); + expect(quoteKinds.length).toBeGreaterThan(0); + for (const e of quoteKinds) { + expect(e.quoteIntegrity, `${e.fixture}/${e.kind}`).toBe(true); + } + }); +}); diff --git a/web/vitest.config.ts b/web/vitest.config.ts index 4bc651c..f7fda70 100644 --- a/web/vitest.config.ts +++ b/web/vitest.config.ts @@ -3,7 +3,7 @@ import { defineConfig } from "vitest/config"; export default defineConfig({ test: { - include: ["tests/unit/**/*.test.{ts,tsx}"], + include: ["tests/unit/**/*.test.{ts,tsx}", "tests/eval/**/*.test.{ts,tsx}"], environment: "node", }, resolve: {