diff --git a/docs/relaunch/DECISIONS.md b/docs/relaunch/DECISIONS.md index cdce5db..91fd4a5 100644 --- a/docs/relaunch/DECISIONS.md +++ b/docs/relaunch/DECISIONS.md @@ -3,29 +3,20 @@ Agents append one entry per tie-break decision (01-product.md §0.4). Newest first. Format: `YYYY-MM-DD · WP-XX · decision · why · alternatives considered`. -- 2026-10-27 · WP-01 · `/dev/components` ships in every build; "dev only" is enforced by noindex metadata + zero inbound links (no nav/sitemap entry) · WP-00 fixed `pnpm e2e` to test the production build via `next start`, so a NODE_ENV or env-var gate would break the axe e2e locally and in CI · gating the route on `NODE_ENV=development`; a `NEXT_PUBLIC_*` flag set in the CI e2e step -- 2026-10-27 · WP-01 · Toast = shadcn `sonner` with the Toaster pinned bottom-right + 4 s (§5.3); dropped the generated `next-themes` wiring, sonner `theme="system"` follows prefers-color-scheme like every other component · the registry has no radix `toast` for the radix style ("only available for Base UI projects" per CLI) and sonner is shadcn's current default · hand-rolled wrapper on radix Toast primitives -- 2026-10-27 · WP-01 · shadcn `--muted-foreground` maps to §5.2 `--body`, not `--faint`: faint on surface-2/3 sits at/below the 4.5:1 AA floor (4.51:1 / ~4.2:1) and muted-foreground renders on muted surfaces (dialog footers, segmented tabs); faint remains in use for text on bg/surface (4.83:1) via custom components · acceptance says fix the component mapping, not the spec tokens · mapping muted-foreground to faint and flagging the AA miss in the PR -- 2026-10-27 · WP-01 · §5.2 focus ring (2px accent + 2px offset) is one global unlayered `:focus-visible` outline; shadcn's per-component `focus-visible:ring-3 ring-ring/50` classes were removed so indicators don't double up · the written spec wins over shadcn defaults · keeping shadcn rings (3px, 50% alpha, no offset) -- 2026-10-27 · WP-01 · axe scans of the open dropdown menu and tooltip are scoped to the open portal (`data-radix-popper-content-wrapper`); dialog/sheet scans stay full-page · Radix renders menus modal: the whole background is aria-hidden while focus is trapped, so full-page scans report hidden-but-focusable background nodes plus page-level heading/region rules — artifacts of the correct modal pattern, not real defects · disabling those rules for the whole suite -- 2026-10-27 · WP-01 · mock.css component styles are realized as React components (shadcn adaptations + `components/app/`) using the tokens; globals.css carries only the §5.2 tokens, shadcn mapping, type scale, radii, shadow, focus and motion · one source of truth per component instead of duplicated utility classes · literal port of `.btn/.pill/.quote/...` classes into globals.css -- 2026-10-27 · WP-01 · Button sizes remapped to §5.3 32/40/48 px (sm/default/lg); dense xs/icon-xs stay 28 px for dialog/sheet close affordances · honors §5.3 while keeping shadcn's dense variants usable · deleting the dense sizes -- 2026-10-27 · WP-02 · Keyless preview mode: when Clerk keys AND DATABASE_URL are both absent, `getAuthUserId()` returns a fixed `user_preview` id and DB-backed routes answer 503; if either is configured the app fails closed (DB-without-Clerk → no user) · previews/CI/e2e must render shells with zero secrets (handoff rule) without faking a provider; the half-configured window (DB, no auth) must never expose data · env flag like ALLOW_KEYLESS_AUTH; dev-only gating (Vercel preview is NODE_ENV=production) -- 2026-10-27 · WP-02 · ClerkProvider, clerkMiddleware (in `proxy.ts`, Next 16 convention) and `/` mount only when `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY`+`CLERK_SECRET_KEY` are set; /sign-in shows a plain "Sign in isn't set up in this preview." card otherwise · app must build/preview keyless; rendering Clerk components keyless throws · always mounting Clerk with dummy keys (fake provider) -- 2026-10-27 · WP-02 · `user.deleted` deletes the users row; workspaces/shows/episodes cascade via FK, feedback + usage_events keep with SET NULL · account deletion should wipe personal data (GDPR-friendly) while the owner keeps the learning signal from feedback notes · soft-delete flag (no column in §7.1) -- 2026-10-27 · WP-02 · POST /api/show is idempotent: if the caller already has a show it returns 200 with the existing row instead of a duplicate · refresh during onboarding must not create two shows (shows.workspace_id is unique) · 409 error the client must handle -- 2026-10-27 · WP-02 · RSS prefill = hand-rolled extraction of channel `` + `itunes:image`/`<image><url>` (with CDATA/entity decode, relative-URL resolution); no XML parser dependency · MVP only needs two fields and RSS import is explicitly out of scope; §0.4 fewer dependencies · fast-xml-parser -- 2026-10-27 · WP-02 · SSRF guard: literal IPs checked without DNS, hostnames resolved with `dns.lookup(all)` and every address must be `unicast` after unwrapping IPv4-mapped IPv6, redirects followed manually (≤3) and re-validated per hop, 5 MB counted-body cap, one shared 5 s deadline · §7.5 requires private-IP blocking; a hostile feed must not bounce us to an internal service or an infinite redirect · allowlist-only domains (breaks real feeds) -- 2026-10-27 · WP-02 · RSS prefill fills only EMPTY fields (name when blank; artwork always) · typed values must never be silently overwritten · always overwrite -- 2026-10-27 · WP-02 · A readable feed with neither title nor artwork is treated as unreadable → the exact U2 error copy · otherwise the user sees success and no prefill happens, which feels broken · 200 with nulls -- 2026-10-27 · WP-02 · Columns without a stated default use boring conventions: timestamps `defaultNow()`, arrays `default []`, counters `default 0`, boolean flags `default false`; status enums have NO default (inserts state one explicitly); `plan_source`, `current_period_end`, ls ids, feed/voice fields stay nullable · §0.4 rule 1 while keeping §7.1's stated defaults byte-exact · inventing defaults for status enums (could mask pipeline bugs) -- 2026-10-27 · WP-02 · /app/episodes (MK-3 empty state) + /app/episodes/new (MK-4 intake shell) shipped as static shells · §5.4 says `/app (→ episodes)`, so U1's post-signup flow needs a real target; WP-06 replaces the list · leaving /app dead-ending after onboarding skip -- 2026-10-27 · WP-02 · Settings Voice/Billing/Account and onboarding step 2 are stubs with copy "This page is coming soon." + the stepper stays navigable (step 2 has "Skip for now") · those WPs (07/08) build the real pages; routes must exist per §5.4 · omitting routes until their WP lands (dead tabs) -- 2026-10-27 · WP-02 · Clerk webhook verifies via the `svix` package directly (payload + svix-* headers) instead of `@clerk/nextjs/webhooks` verifyWebhook · same Svix signatures, but plain `Request` typing and testable without Next-specific request shapes · verifyWebhook (needs RequestLike cast) -- 2026-10-27 · WP-02 · Playwright `E2E_PORT` env override (default 3000, command unchanged) · sibling agents run e2e concurrently on this machine and race for :3000; CI path stays byte-identical · hard-coding a different port (would break the documented CI flow) -- 2026-10-27 · WP-02 · `cost_usd` is `numeric(12,6)` and `units` int · money estimates should not accumulate float error; drizzle maps numeric to string · doublePrecision (simpler, lossy) -Format: `YYYY-MM-DD · WP-XX · decision · why · alternatives considered`. +- 2026-10-27 · WP-04 · WP-02 integration landed in this branch via merge: `resolveAppSettingsStore()` now returns a real Drizzle store over `app_settings` (WP-02 schema), null only when `DATABASE_URL` is unset (keyless preview); `requireOwner` uses `lib/env.isClerkConfigured()` for clean 503-vs-401 classification and still denies keyless previews (admin AI switching must never work where auth is fake) · completes the documented WP-02 seams instead of leaving them stubbed; U18 now exercises the real store path in tests via the same interface · keeping the null stub until WP-09 +- 2026-10-27 · WP-04 · Groq fallback models: `openai/gpt-oss-120b` (long) / `openai/gpt-oss-20b` (fast) · §8.1 requires the choice to be made from Groq's current model list at build time; console.groq.com/docs/models (2026-10-27) lists the llama-3.3/3.1 models as Enterprise ("contact sales") while gpt-oss-120b/20b are self-serve with public prices ($0.15/$0.60 and $0.075/$0.30 per M) and 131k context · llama-3.3-70b-versatile long + llama-3.1-8b-instant fast +- 2026-10-27 · WP-04 · `lib/ai/prices.ts` carries only vendor-verified list prices (Groq, OpenAI, Anthropic — fetched from their pricing docs 2026-10-27, source stamped per row); GLM rows are absent → cost 0 · hard rule: no invented numbers; GLM coding-plan calls are subscription-covered anyway, and bigmodel.cn list prices were not verifiable at build time · estimated GLM prices from memory +- 2026-10-27 · WP-04 · `requireOwner` denies with 503 when Clerk is unconfigured or `OWNER_EMAILS` is unset (warn-once log) — no insecure fallback · M13 makes /admin owner-only; a header/env-based identity bypass would be fake auth; /admin lights up when WP-02 ships Clerk middleware · allowing requests in dev when Clerk keys are absent +- 2026-10-27 · WP-04 · `AppSettingsStore` (`get/put` on the `app_settings` key) is an injected interface; until WP-02's DB lands `resolveAppSettingsStore()` returns null — GET serves the env/default view, PUT returns 503 `settings_store_unavailable` · keeps WP-04 free of schema ownership and gives WP-02 a one-function seam; U18 is tested against the documented interface with an in-memory store · inventing a Drizzle table in WP-04 +- 2026-10-27 · WP-04 · `@clerk/nextjs` reuse: WP-04 aligns on WP-02's `@clerk/nextjs@^7.9.20` (same version both branches chose independently) and reads the real `currentUser()` via dynamic import · it is the §0.2-locked auth dependency; dynamic import + try/catch degrades to `auth_unavailable` (deny) when keys are absent · stubbing auth without the package (fake import, bundler-fragile) +- 2026-10-27 · WP-04 · deterministic fallback content is NOT held to the §7.2 AI minima (e.g. exactly 8 titles, ≥5 chapters) — the engine returns what the transcript honestly yields, and artifact status `fallback` (U10 banner) marks it · padding to satisfy a schema would fabricate content; WP-06 renders fallback packages with the banner · duplicating/padding engine output to pass Zod +- 2026-10-27 · WP-04 · quote claims from the model are resolved to verbatim offsets (§7.3) BEFORE Zod validation, with candidate backfill guaranteeing required counts (blog 2, newsletter 1, quotes 5) · schemas store resolved QuoteRefs, so resolution must precede validation; backfill keeps counts honest (transcript text only) · validating raw `{segmentId, exact}` claims against offset schemas +- 2026-10-27 · WP-04 · selection precedence is wholesale: a valid `app_settings.ai` object wins over env entirely; env (`AI_PROVIDER` + optional `AI_MODEL_*`) is the fallback layer; no field-by-field mixing · mixing settings fields with env models makes admin switches silently overridable and untestable; §8.1 lists the order, not a merge rule · per-field precedence +- 2026-10-27 · WP-04 · the §8.4 URL allow list matches subpaths of an allowed origin (`https://show.example` allows `https://show.example/ep-12`) · the show's settings hold its own site/link; subpath links are not hallucinated foreign domains; WP-03's exact-match primitive stays untouched (its tests pin it) and the prefix rule lives in the WP-04 wiring layer · changing `lib/artifacts/postcheck.ts` semantics +- 2026-10-27 · WP-04 · `usage_events.units` for `kind=llm` = total tokens (prompt + completion); cost_usd = in/out-priced estimate · §7.1 leaves `units` open; one number keeps the row compact and cost is derived from the same table · output tokens only +- 2026-10-27 · WP-04 · live provider smoke test lives in `tests/unit/live.ai.test.ts` behind `describe.skipIf(!GLM_API_KEY || CI)` — no separate script/runner · zero new tooling; CI (no keys) always skips; the run command is documented in the file header · a scripts/*.mjs runner with tsx +- 2026-10-27 · WP-04 · OpenAI/OpenRouter placeholder defaults (`gpt-4o`/`gpt-4o-mini`, `anthropic/claude-sonnet-4.5`/`openai/gpt-4o-mini`) exist so a preset is never model-less; §8.1 says these providers are "set via env" and env/settings always override them · leaving defaults empty and failing late · throwing when models are unset - 2026-10-27 · WP-03 · zod 4.6.7 for `lib/artifacts/schemas.ts` (spec text predates zod 4) · §0.4 rule 6: verified the API against the installed package; note zod 4 objects have no `.min/.max/.length` — count constraints live on `z.array(...)` · staying on zod 3 - 2026-10-27 · WP-03 · Legacy `validateSuite` (engine/glm.js) is not ported as-is; its quote-integrity core (`isExactQuote`, `normalizeQuoteStyle`) is ported into `lib/quotes/normalize.ts` and the suite-level validator is replaced by §7.3 resolve-by-reference (`resolveQuotes`) + per-artifact Zod validation · the new architecture stores QuoteRefs, not model text, so suite-shape validation is obsolete; spec wins over legacy JS · porting validateSuite into web/lib - 2026-10-27 · WP-03 · Quote matching is segment-anchored: a model quote is searched in its claimed segment + the next 2 (§7.3), not anywhere in the transcript like legacy `isExactQuote` · spec wins; a verbatim quote claimed on the wrong segment is dropped and refilled from candidates · keeping the legacy whole-source scan diff --git a/web/app/api/admin/ai/route.ts b/web/app/api/admin/ai/route.ts new file mode 100644 index 0000000..6be5b0b --- /dev/null +++ b/web/app/api/admin/ai/route.ts @@ -0,0 +1,18 @@ +// §7.5 — GET/PUT /api/admin/ai (owner only, M13). Thin wrappers over the +// framework-free handlers in lib/ai/admin.ts. The app_settings store arrives +// through the documented WP-02 seam; until the DB layer lands, PUT returns 503 +// and GET serves the env/default view. +import { adminAiGet, adminAiPut } from "@/lib/ai/admin"; +import { resolveAppSettingsStore } from "@/lib/ai/settings-server"; + + +export async function GET() { + const res = await adminAiGet({ store: resolveAppSettingsStore() }); + return Response.json(res.body, { status: res.status }); +} + +export async function PUT(request: Request) { + const body = await request.json().catch(() => null); + const res = await adminAiPut({ store: resolveAppSettingsStore() }, body); + return Response.json(res.body, { status: res.status }); +} diff --git a/web/app/api/admin/ai/test/route.ts b/web/app/api/admin/ai/test/route.ts new file mode 100644 index 0000000..9170263 --- /dev/null +++ b/web/app/api/admin/ai/test/route.ts @@ -0,0 +1,12 @@ +// §7.5 — POST /api/admin/ai/test (owner only): tiny prompt through the current +// (or body-specified) provider; returns OK/latency/usage or the error. +import { adminAiTest, type AdminTestBody } from "@/lib/ai/admin"; +import { resolveAppSettingsStore } from "@/lib/ai/settings-server"; + +export const maxDuration = 60; + +export async function POST(request: Request) { + const body = ((await request.json().catch(() => ({}))) ?? {}) as AdminTestBody; + const res = await adminAiTest({ store: resolveAppSettingsStore() }, body); + return Response.json(res.body, { status: res.status }); +} diff --git a/web/lib/ai/admin.ts b/web/lib/ai/admin.ts new file mode 100644 index 0000000..9825179 --- /dev/null +++ b/web/lib/ai/admin.ts @@ -0,0 +1,176 @@ +// §7.5 admin AI API core (UI ships in WP-09): GET/PUT /api/admin/ai and +// POST /api/admin/ai/test. Handlers are framework-free and take their +// dependencies at the documented interfaces (app_settings store, owner gate, +// provider chain factory) so route files stay thin and tests stay honest. +import { AiSelectionSchema, effectiveAiSelection, keyPresent, type AiSelection, type AppSettingsStore, type SelectionSource } from "./settings"; +import { PROVIDER_PRESETS, providerIds } from "./presets"; +import { PROVIDER_IDS } from "./types"; +import { buildProviderChain, modelFor } from "./registry"; +import { estimateCost } from "./prices"; +import { extractJson } from "./json"; +import { DEFAULT_AI_SELECTION, resolveCustom } from "./settings"; +import { requireOwner as defaultRequireOwner, type OwnerGateDeps, type OwnerGateResult } from "./owner"; +import type { LlmProvider } from "./types"; +import type { ModelTier } from "./registry"; + +type OwnerGateFn = (deps?: OwnerGateDeps) => Promise<OwnerGateResult>; + +export interface AdminDeps { + /** app_settings store; null = not wired yet (WP-02 seam) */ + store?: AppSettingsStore | null; + requireOwner?: OwnerGateFn; + env?: NodeJS.ProcessEnv; + /** chain factory override for the test button (tests) */ + chainFor?: (selection: AiSelection, tier: ModelTier) => LlmProvider[]; + fetchImpl?: typeof fetch; + sleep?: (ms: number) => Promise<void>; +} + +export interface AdminResponse { + status: number; + body: unknown; +} + +// ---- GET -------------------------------------------------------------------- + +export async function adminAiGet(deps: AdminDeps = {}): Promise<AdminResponse> { + const gate = await (deps.requireOwner ?? gateWith(defaultRequireOwner, deps))(); + if (!gate.ok) return { status: gate.status, body: { error: gate.reason } }; + + const env = deps.env ?? process.env; + const { selection, source } = await effectiveAiSelection(deps.store ?? null, { env }); + return { + status: 200, + body: { + selection, + source: source satisfies SelectionSource, + defaultSelection: DEFAULT_AI_SELECTION, + storeAvailable: Boolean(deps.store), + providers: providerIds().map((id) => { + const p = PROVIDER_PRESETS[id]; + return { + id: p.id, + label: p.label, + baseUrl: p.id === "custom" ? resolveCustom(env).baseUrl || null : p.baseUrl, + keyEnv: p.keyEnv, + keyPresent: keyPresent(p, env), + defaultLong: p.id === "custom" ? resolveCustom(env).longModel || null : p.defaultLong, + defaultFast: p.id === "custom" ? resolveCustom(env).fastModel || null : p.defaultFast, + }; + }), + }, + }; +} + +// ---- PUT -------------------------------------------------------------------- + +export async function adminAiPut(deps: AdminDeps = {}, body: unknown): Promise<AdminResponse> { + const gate = await (deps.requireOwner ?? gateWith(defaultRequireOwner, deps))(); + if (!gate.ok) return { status: gate.status, body: { error: gate.reason } }; + + const env = deps.env ?? process.env; + const parsed = AiSelectionSchema.safeParse(body); + if (!parsed.success) { + return { status: 400, body: { error: "invalid_selection", detail: parsed.error.issues.slice(0, 5).map((i) => `${i.path.join(".")}: ${i.message}`) } }; + } + if (parsed.data.provider === "custom" && !resolveCustom(env).baseUrl) { + return { status: 400, body: { error: "custom_missing_base_url", detail: "Set AI_CUSTOM_BASE_URL before switching to the custom provider." } }; + } + + const store = deps.store; + if (!store) { + return { status: 503, body: { error: "settings_store_unavailable" } }; + } + try { + await store.put("ai", parsed.data); + } catch (err) { + return { + status: 503, + body: { error: "settings_store_write_failed", detail: String(err).slice(0, 200) }, + }; + } + return { status: 200, body: { ok: true, selection: parsed.data, source: "settings" } }; +} + +// ---- POST /test ------------------------------------------------------------- + +export interface AdminTestBody { + provider?: string; + model?: string; + tier?: ModelTier; +} + +export async function adminAiTest(deps: AdminDeps = {}, body: AdminTestBody = {}): Promise<AdminResponse> { + const gate = await (deps.requireOwner ?? gateWith(defaultRequireOwner, deps))(); + if (!gate.ok) return { status: gate.status, body: { error: gate.reason } }; + + const env = deps.env ?? process.env; + let selection: AiSelection; + if (body.provider != null) { + if (!PROVIDER_IDS.includes(body.provider as never)) { + return { status: 400, body: { error: "unknown_provider" } }; + } + const base = (await effectiveAiSelection(deps.store ?? null, { env })).selection; + selection = { + provider: body.provider as AiSelection["provider"], + longModel: body.model ?? base.longModel, + fastModel: body.model ?? base.fastModel, + fallbackProvider: base.fallbackProvider, + }; + } else { + selection = (await effectiveAiSelection(deps.store ?? null, { env })).selection; + if (body.model) { + selection = { ...selection, ...(body.tier === "fast" ? { fastModel: body.model } : { longModel: body.model }) }; + } + } + + const tier: ModelTier = body.tier ?? "fast"; + const model = modelFor(selection, tier); + const chain = deps.chainFor + ? deps.chainFor(selection, tier) + : buildProviderChain(selection, { env, fetchImpl: deps.fetchImpl, sleep: deps.sleep }); + const provider = chain[0]; + + const started = Date.now(); + try { + const res = await provider.complete({ + system: "You are a health check. Return only JSON.", + user: 'Return the JSON object {"ok":true}.', + model, + json: true, + temperature: 0.2, + }); + const latencyMs = Date.now() - started; + const parsed = extractJson(res.text); + const respondsOk = Boolean(parsed && typeof parsed === "object" && (parsed as { ok?: unknown }).ok === true); + return { + status: 200, + body: { + ok: respondsOk, + provider: provider.id, + model, + latencyMs, + usage: res.usage ?? null, + costUsd: estimateCost(model, res.usage), + text: respondsOk ? undefined : res.text.slice(0, 200), + }, + }; + } catch (err) { + return { + status: 200, + body: { + ok: false, + provider: provider.id, + model, + latencyMs: Date.now() - started, + error: { code: (err as { code?: string }).code ?? "provider_failed", message: String(err).slice(0, 300) }, + }, + }; + } +} + +// ---- gate plumbing ---------------------------------------------------------- + +function gateWith(requireOwner: OwnerGateFn, deps: AdminDeps): () => Promise<OwnerGateResult> { + return () => requireOwner({ env: deps.env }); +} diff --git a/web/lib/ai/anthropic.ts b/web/lib/ai/anthropic.ts new file mode 100644 index 0000000..406d66d --- /dev/null +++ b/web/lib/ai/anthropic.ts @@ -0,0 +1,99 @@ +// §8.1 — native Anthropic Messages adapter (system on top, usage in tokens). +// Same call policy: 60 s timeout, 1 retry on 429/5xx with a 2 s backoff. +import { + DEFAULT_MAX_TOKENS, + DEFAULT_TIMEOUT_MS, + RETRY_BACKOFF_MS, + ProviderError, + type CompleteRequest, + type CompleteResult, + type LlmProvider, +} from "./types"; + +export interface AnthropicOptions { + apiKey: () => string | undefined; + baseUrl?: string; + timeoutMs?: number; + retryBackoffMs?: number; + maxRetries?: number; + fetchImpl?: typeof fetch; + sleep?: (ms: number) => Promise<void>; +} + +interface MessagesResponse { + content?: Array<{ type: string; text?: string }>; + usage?: { input_tokens?: number; output_tokens?: number }; +} + +export function createAnthropicProvider(opts: AnthropicOptions): LlmProvider { + const id = "anthropic"; + const baseUrl = opts.baseUrl ?? "https://api.anthropic.com"; + const timeoutMs = opts.timeoutMs ?? DEFAULT_TIMEOUT_MS; + const retryBackoffMs = opts.retryBackoffMs ?? RETRY_BACKOFF_MS; + const maxRetries = opts.maxRetries ?? 1; + const doFetch = opts.fetchImpl ?? fetch; + const sleep = opts.sleep ?? ((ms: number) => new Promise<void>((r) => setTimeout(r, ms))); + + return { + id, + async complete(req: CompleteRequest): Promise<CompleteResult> { + const key = opts.apiKey(); + if (!key) throw new ProviderError("no_key", id, `${id}: missing API key`); + + const body: Record<string, unknown> = { + model: req.model, + max_tokens: req.maxTokens ?? DEFAULT_MAX_TOKENS, + temperature: req.temperature ?? 0.4, + system: req.system, + messages: [{ role: "user", content: req.user }], + }; + + let attempt = 0; + for (;;) { + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), timeoutMs); + try { + const res = await doFetch(`${baseUrl}/v1/messages`, { + method: "POST", + headers: { + "Content-Type": "application/json", + "x-api-key": key, + "anthropic-version": "2023-06-01", + }, + body: JSON.stringify(body), + signal: controller.signal, + }); + if (!res.ok) { + const detail = (await res.text().catch(() => "")).slice(0, 200); + const retryable = res.status === 429 || res.status >= 500; + const code = res.status === 429 ? "rate_limited" : res.status >= 500 ? "server_error" : "bad_request"; + if (retryable && attempt < maxRetries) { + attempt++; + await sleep(retryBackoffMs); + continue; + } + throw new ProviderError(code, id, `${id}: http ${res.status}: ${detail}`, res.status, detail); + } + const data = (await res.json()) as MessagesResponse; + const text = (data.content ?? []) + .filter((b) => b.type === "text") + .map((b) => b.text ?? "") + .join(""); + if (!text) throw new ProviderError("empty", id, `${id}: empty completion`); + const usage = data.usage + ? { in: data.usage.input_tokens ?? 0, out: data.usage.output_tokens ?? 0 } + : undefined; + return { text, usage }; + } catch (err) { + if (err instanceof ProviderError) throw err; + if (err instanceof Error && err.name === "AbortError") { + throw new ProviderError("timeout", id, `${id}: timed out after ${timeoutMs}ms`); + } + throw new ProviderError("network", id, `${id}: ${err instanceof Error ? err.message : String(err)}`); + } finally { + clearTimeout(timer); + } + } + }, + }; +} diff --git a/web/lib/ai/fallback.ts b/web/lib/ai/fallback.ts new file mode 100644 index 0000000..2981bee --- /dev/null +++ b/web/lib/ai/fallback.ts @@ -0,0 +1,155 @@ +// §8.1 last resort — the deterministic builder (lib/engine, TS port of +// engine/lib.js) mapped onto §7.2 artifact kinds. Everything here is derived +// from the transcript or from user-provided metadata: nothing is invented. +// Content produced this way is stored with artifact status "fallback" (U10: +// "basic package labelled with the fallback banner") and is NOT held to the AI +// content minima (e.g. exactly 8 titles) — honesty beats schema padding. +import { + analyzeTranscript, + buildArticle, + buildSocialPack, + buildSummary, + buildTitles, + normalizeWs, + rankedSentences, +} from "@/lib/engine"; +import { candidateQuoteRefs } from "@/lib/quotes/candidates"; +import { trimAtSentenceBoundary } from "@/lib/artifacts/postcheck"; +import type { ArtifactKind, TitleStyle } from "@/lib/artifacts/schemas"; +import type { Segment } from "@/lib/quotes/types"; +import type { EpisodeMeta, ShowContext } from "@/lib/prompts"; + +export interface FallbackContext { + segments: Segment[]; + show: ShowContext; + meta?: EpisodeMeta; +} + +const DEFAULT_CTA = "Invite listeners to follow the show."; + +function transcriptText(segments: Segment[]): string { + return normalizeWs(segments.map((s) => s.text).join(" ")); +} + +/** Classify an extractive title into one of the 6 §7.2 styles. */ +function classifyTitle(text: string): TitleStyle { + if (/^(what|why|how)\b/i.test(text)) return "Question"; + if (/\d/.test(text)) return "Number"; + return "Curiosity"; +} + +/** First maxWords of the text (fallback content is extractive, not re-written). */ +function clipToWords(text: string, maxWords: number): string { + const words = text.split(/\s+/).filter(Boolean); + return words.length <= maxWords ? text : words.slice(0, maxWords).join(" "); +} + +function slugify(text: string): string { + return ( + text + .toLowerCase() + .replace(/[^a-z0-9]+/g, "-") + .replace(/^-+|-+$/g, "") + .slice(0, 60) || "episode" + ); +} + +export function buildDeterministicArtifact(kind: ArtifactKind, ctx: FallbackContext): Record<string, unknown> { + const text = transcriptText(ctx.segments); + const a = analyzeTranscript(text); + a.ranked = a.ranked ?? rankedSentences(a.sents, a.themes); + const cta = ctx.show.defaultCta || DEFAULT_CTA; + + switch (kind) { + case "titles": { + const items = buildTitles(a).map((t) => ({ + text: t.slice(0, 80), + style: classifyTitle(t), + starred: false, + })); + return { items }; + } + case "summary": { + const summary = buildSummary(a); + const oneLiner = trimAtSentenceBoundary(summary.text, 140) || a.keyQuotes[0]?.slice(0, 140) || ""; + return { oneLiner, short: clipToWords(summary.text, 80), long: summary.text }; + } + case "show_notes": { + const hook = a.keyQuotes.slice(0, 2).join(" "); + const bullets = a.keyQuotes.slice(2, 8); + const guestLine = ctx.meta?.guests?.length ? `Featuring ${ctx.meta.guests.join(" and ")}.` : undefined; + return { + hook, + bullets: bullets.length >= 4 ? bullets : a.keyQuotes.slice(0, 4), + ...(guestLine ? { guestLine } : {}), + quoteRefs: [], + cta, + }; + } + case "chapters": { + const timed = ctx.segments.some((s) => s.startMs !== null); + if (!timed) return { items: [] }; + const n = ctx.segments.length; + const target = Math.max(1, Math.min(8, Math.round(n / 6))); + const items: Array<{ startMs: number; title: string }> = []; + const used = new Set<string>(); + for (let m = 0; m < target; m++) { + const chunk = ctx.segments.slice(Math.floor((m * n) / target), Math.floor(((m + 1) * n) / target)); + if (!chunk.length || chunk[0].startMs === null) continue; + const localThemes = analyzeTranscript(chunk.map((s) => s.text).join(" ")).themes; + let title = localThemes[0] || a.themes[m] || "Episode section"; + if (used.has(title) && localThemes[1]) title = localThemes[1]; + used.add(title); + items.push({ + startMs: m === 0 ? 0 : (chunk[0].startMs as number), + title: (title.charAt(0).toUpperCase() + title.slice(1)).slice(0, 50), + }); + } + return { items }; + } + case "blog": { + const article = buildArticle(text, a); + const summary = buildSummary(a); + return { + seoTitle: trimAtSentenceBoundary(article.title, 60), + metaDescription: trimAtSentenceBoundary(summary.text, 155), + slug: slugify(article.title), + bodyMarkdown: article.sections + .map((sec) => `## ${sec.heading}\n\n${sec.paragraphs.join("\n\n")}`) + .join("\n\n"), + quoteRefs: [], + }; + } + case "newsletter": { + const summary = buildSummary(a); + const titles = buildTitles(a); + return { + subjects: [titles[0] || "New episode", titles[1] || "This week", titles[2] || "Latest show"].map((t) => + t.slice(0, 80), + ), + chosen: 0, + previewText: trimAtSentenceBoundary(summary.text, 90), + bodyMarkdown: `${summary.text}\n\n${cta}`, + quoteRefs: [], + ctaText: trimAtSentenceBoundary(cta, 30), + }; + } + case "social": { + const social = buildSocialPack(a); + const caption = trimAtSentenceBoundary(social.captions[0]?.text ?? a.keyQuotes[0] ?? "", 2200); + const hashtags = a.themes + .filter((t) => t.length >= 4) + .slice(0, 8) + .map((t) => `#${t.replace(/[^a-z0-9]/gi, "")}`) + .filter((h) => h.length > 1); + return { + linkedin: trimAtSentenceBoundary(caption, 1300), + thread: social.xPosts.map((p) => p.text).slice(0, 6), + instagram: { caption, hashtags }, + }; + } + case "quotes": { + return { items: candidateQuoteRefs(ctx.segments, { count: 5 }) }; + } + } +} diff --git a/web/lib/ai/generate.ts b/web/lib/ai/generate.ts new file mode 100644 index 0000000..1cb6a53 --- /dev/null +++ b/web/lib/ai/generate.ts @@ -0,0 +1,373 @@ +// §8.1 call policy, one artifact at a time (weak-model hardening: one artifact +// per call): +// +// primary provider (60 s timeout, 1 retry on 429/5xx with 2 s backoff) +// → fallback provider +// → deterministic builder (lib/engine) with status "fallback" +// +// JSON handling: response_format json_object where supported (adapter), Zod +// validation (§7.2 schemas), ONE repair call, then the next provider. +// Quote claims are resolved to verbatim offsets via lib/quotes (§7.3) BEFORE +// validation — model quote text is never stored or displayed. +// Post-checks (§8.4, U8): neverSay scan → one regenerate with feedback → +// phrase replacement; religion-neutral gate on generated framing → regenerate +// with feedback → next provider (never phrase-replace, never applied to +// transcript quotes); foreign URLs stripped deterministically throughout. +import { ARTIFACT_KINDS, ArtifactContentSchema, artifactContentSchema, type ArtifactKind } from "@/lib/artifacts/schemas"; +import { candidateQuoteRefs } from "@/lib/quotes/candidates"; +import { resolveQuotes, type QuoteRef, type Segment } from "@/lib/quotes"; +import { + buildArtifactPrompt, + PROMPT_VERSION, + type EpisodeMeta, + type PromptContext, + type ShowContext, +} from "@/lib/prompts"; +import type { VoiceProfile } from "./voice"; +import { extractJson, repairPrompt } from "./json"; +import { + applyDeterministicPostChecks, + scanContent, + stripForeignUrlsInContent, + type ContentViolations, +} from "./postcheck"; +import { buildDeterministicArtifact } from "./fallback"; +import { buildProviderChain, modelFor } from "./registry"; +import { estimateCost } from "./prices"; +import { effectiveAiSelection, type AiSelection, type AppSettingsStore } from "./settings"; +import type { AttemptError, LlmProvider } from "./types"; +import type { UsageEvent, UsageLogger } from "./usage"; + +export interface GenerateDeps { + /** full provider chain override (tests, admin test path) */ + chain?: LlmProvider[]; + /** explicit selection (skips the store) */ + selection?: AiSelection; + /** app_settings store — the first place the selection is read from */ + store?: AppSettingsStore | null; + /** usage_events logger (§7.1); WP-02 wires the DB-backed logger */ + usage?: UsageLogger | null; + workspaceId?: string | null; + episodeId?: string | null; + env?: NodeJS.ProcessEnv; + fetchImpl?: typeof fetch; + sleep?: (ms: number) => Promise<void>; + timeoutMs?: number; +} + +export interface GenerateInput { + kind: ArtifactKind; + segments: Segment[]; + show: ShowContext; + voice: VoiceProfile; + voiceSamples?: string[]; + meta?: EpisodeMeta; + /** regenerate instruction (≤200 chars) */ + instruction?: string; + /** analysis.candidate_quotes — computed from segments when absent */ + analysisCandidates?: QuoteRef[]; + deps?: GenerateDeps; +} + +export interface GenerateResult { + kind: ArtifactKind; + content: unknown; + status: "ai" | "fallback"; + provider: string | null; + model: string | null; + errorCode: string | null; + truncated: boolean; + regenerated: boolean; + usage: { in: number; out: number } | null; + costUsd: number; + promptVersion: string; + attempts: AttemptError[]; +} + +const QUOTE_KIND_COUNTS: Partial<Record<ArtifactKind, { field: "items" | "quoteRefs"; count: number }>> = { + quotes: { field: "items", count: 5 }, + show_notes: { field: "quoteRefs", count: 1 }, + blog: { field: "quoteRefs", count: 2 }, + newsletter: { field: "quoteRefs", count: 1 }, +}; + +/** §7.3 steps 2–3: turn model quote claims into verbatim offsets, in place. */ +function resolveModelQuotes( + content: Record<string, unknown>, + kind: ArtifactKind, + segments: Segment[], + candidates: QuoteRef[], +): void { + const spec = QUOTE_KIND_COUNTS[kind]; + if (!spec) return; + content[spec.field] = resolveQuotes(content[spec.field], segments, { + count: spec.count, + candidates, + }).quotes; +} + +function hasViolations(v: ContentViolations): boolean { + return v.neverSay.length > 0 || v.religion.length > 0; +} + +function violationFeedback(v: ContentViolations): string { + const parts: string[] = []; + if (v.neverSay.length) parts.push(`banned words/phrases: ${v.neverSay.join(", ")}`); + if (v.religion.length) parts.push(`religion-specific framing terms: ${v.religion.join(", ")}`); + return ( + `Your previous draft used ${parts.join(" and ")}. Those terms must not appear in any form. ` + + `Rewrite and return the full corrected JSON — same schema, none of those words.` + ); +} + +export async function generateArtifact(input: GenerateInput): Promise<GenerateResult> { + const { kind, segments, show, voice } = input; + const deps = input.deps ?? {}; + const env = deps.env ?? process.env; + + // ---- selection + chain (§8.1 order: app_settings.ai → env → defaults) ---- + let chain: LlmProvider[]; + let model: string; + if (deps.chain?.length) { + chain = deps.chain; + model = modelFor( + deps.selection ?? (await effectiveAiSelection(null, { env })).selection, + "long", + ); + } else { + const selection = deps.selection ?? (await effectiveAiSelection(deps.store ?? null, { env })).selection; + chain = buildProviderChain(selection, { + env, + fetchImpl: deps.fetchImpl, + sleep: deps.sleep, + timeoutMs: deps.timeoutMs, + }); + model = modelFor(selection, "long"); + } + + // ---- chapters need timestamps (M4: chapters only when timestamps exist) -- + if (kind === "chapters" && !segments.some((s) => s.startMs !== null)) { + return fallbackResult(input, [], "no_timestamps", false); + } + + // ---- prompt (§8.2) -------------------------------------------------------- + const promptCtx: PromptContext = { + show, + voice, + voiceSamples: input.voiceSamples, + meta: input.meta, + segments, + instruction: input.instruction, + }; + const prompt = buildArtifactPrompt(kind, promptCtx); + const schema = artifactContentSchema(kind); + const candidates = input.analysisCandidates ?? candidateQuoteRefs(segments); + const attempts: AttemptError[] = []; + + let totalUsage: { in: number; out: number } | null = null; + let totalCost = 0; + const usageLog = deps.usage ?? null; + + const recordUsage = async ( + provider: LlmProvider, + usedModel: string, + u: { in: number; out: number } | undefined, + ): Promise<void> => { + if (!u) return; + totalUsage = { in: (totalUsage?.in ?? 0) + u.in, out: (totalUsage?.out ?? 0) + u.out }; + const cost = estimateCost(usedModel, u); + totalCost += cost; + if (!usageLog) return; + const event: UsageEvent = { + workspaceId: deps.workspaceId ?? null, + episodeId: deps.episodeId ?? null, + kind: "llm", + provider: provider.id, + model: usedModel, + units: u.in + u.out, + costUsd: cost, + }; + try { + await usageLog(event); // logger failures must never fail generation + } catch (err) { + console.error("usage logger failed:", err); + } + }; + + const baseResult = (provider: LlmProvider, content: unknown, regenerated: boolean): GenerateResult => ({ + kind, + content, + status: "ai", + provider: provider.id, + model: model || provider.id, + errorCode: null, + truncated: prompt.truncated, + regenerated, + usage: totalUsage, + costUsd: totalCost, + promptVersion: PROMPT_VERSION, + attempts, + }); + + const buildRequest = (user: string) => ({ + system: prompt.system, + user, + model: model, + json: true, + temperature: prompt.temperature, + }); + + for (const provider of chain) { + // 1st attempt — the adapters apply timeout + 1 retry on 429/5xx internally. + let text: string; + try { + const res = await provider.complete(buildRequest(prompt.user)); + await recordUsage(provider, model, res.usage); + text = res.text; + } catch (err) { + attempts.push({ provider: provider.id, code: codeOf(err), detail: String(err).slice(0, 300) }); + continue; + } + + // parse + resolve quotes + validate (Zod §7.2) + let parsed = asObject(extractJson(text)); + if (parsed) resolveModelQuotes(parsed, kind, segments, candidates); + let validation = schema.safeParse(parsed); + + // ONE repair call (§8.1), same provider + if (!validation.success) { + const problem = parsed ? "it did not match the required schema" : "it was not valid JSON"; + try { + const repair = await provider.complete( + buildRequest(repairPrompt(prompt.user, JSON.stringify(schemaHint(kind)), problem)), + ); + await recordUsage(provider, model, repair.usage); + const reparsed = asObject(extractJson(repair.text)); + if (reparsed) { + resolveModelQuotes(reparsed, kind, segments, candidates); + const v2 = schema.safeParse(reparsed); + if (v2.success) { + parsed = reparsed; + validation = v2; + } + } + } catch (err) { + attempts.push({ provider: provider.id, code: codeOf(err), detail: String(err).slice(0, 300) }); + continue; + } + } + + if (!validation.success) { + attempts.push({ provider: provider.id, code: "invalid_json", detail: zodDetail(validation.error) }); + continue; + } + + // ---- post-checks (§8.4) ------------------------------------------------- + let content = parsed as Record<string, unknown>; + let violations = scanContent(content, voice); + let regenerated = false; + + if (hasViolations(violations)) { + // one regenerate with feedback + try { + const res = await provider.complete(buildRequest(prompt.user + `\n\n${violationFeedback(violations)}`)); + await recordUsage(provider, model, res.usage); + const retry = asObject(extractJson(res.text)); + if (!retry) { + attempts.push({ provider: provider.id, code: "invalid_json", detail: "postcheck regen unparseable" }); + continue; + } + resolveModelQuotes(retry, kind, segments, candidates); + content = retry; + regenerated = true; + } catch (err) { + attempts.push({ provider: provider.id, code: codeOf(err), detail: String(err).slice(0, 300) }); + continue; + } + + violations = scanContent(content, voice); + // religion framing that still violates is NOT phrase-replaceable — the + // gate applies to generated framing only (§8.4); this provider is done. + if (violations.religion.length) { + attempts.push({ provider: provider.id, code: "religion_gate", detail: violations.religion.join(",") }); + continue; + } + // neverSay leftovers are replaced deterministically (U8 last resort) + if (violations.neverSay.length) { + content = applyDeterministicPostChecks(content, voice, show.allowedUrls ?? []); + } + } + + const cleaned = stripForeignUrlsInContent(content, show.allowedUrls ?? []); + const final = schema.safeParse(cleaned); + if (!final.success) { + attempts.push({ provider: provider.id, code: "postcheck_invalid", detail: zodDetail(final.error) }); + continue; + } + return baseResult(provider, final.data, regenerated); + } + + return fallbackResult(input, attempts, "all_providers_failed", prompt.truncated); +} + +// ---- helpers ---------------------------------------------------------------- + +function asObject(v: unknown): Record<string, unknown> | null { + return v && typeof v === "object" && !Array.isArray(v) ? (v as Record<string, unknown>) : null; +} + +function zodDetail(error: { issues: Array<{ path: PropertyKey[]; message: string }> }): string { + return error.issues + .slice(0, 3) + .map((i) => `${i.path.join(".")}: ${i.message}`) + .join("; "); +} + +function codeOf(err: unknown): string { + const e = err as { code?: string; name?: string }; + return e?.code || e?.name || "provider_failed"; +} + +/** Compact inline shape for the repair prompt (flat, weak-model friendly). */ +function schemaHint(kind: ArtifactKind): Record<string, unknown> { + const shapes: Record<ArtifactKind, Record<string, unknown>> = { + show_notes: { hook: "string", bullets: ["string x4-7"], guestLine: "string?", quoteRefs: [{ segmentId: "string", exact: "string" }], cta: "string" }, + chapters: { items: [{ startMs: 0, title: "string ≤50" }] }, + summary: { oneLiner: "string ≤140", short: "string", long: "string" }, + titles: { items: [{ text: "string ≤80", style: "Curiosity|Benefit|Guest|Number|Question|Quote", starred: false }] }, + blog: { seoTitle: "string ≤60", metaDescription: "string ≤155", slug: "string", bodyMarkdown: "string", quoteRefs: [{ segmentId: "string", exact: "string" }] }, + newsletter: { subjects: ["s1", "s2", "s3"], chosen: 0, previewText: "string ≤90", bodyMarkdown: "string", quoteRefs: [{ segmentId: "string", exact: "string" }], ctaText: "string ≤30" }, + social: { linkedin: "string ≤1300", thread: ["≤280 x4-6"], instagram: { caption: "string ≤2200", hashtags: ["#tag x5-10"] } }, + quotes: { items: [{ segmentId: "string", exact: "string" }] }, + }; + return shapes[kind]; +} + +function fallbackResult( + input: GenerateInput, + attempts: AttemptError[], + errorCode: string, + truncated: boolean, +): GenerateResult { + return { + kind: input.kind, + content: buildDeterministicArtifact(input.kind, { + segments: input.segments, + show: input.show, + meta: input.meta, + }), + status: "fallback", + provider: null, + model: null, + errorCode, + truncated, + regenerated: false, + usage: null, + costUsd: 0, + promptVersion: PROMPT_VERSION, + attempts, + }; +} + +// re-exported so WP-06 has one import site for the §7.2 schema map +export { ARTIFACT_KINDS, ArtifactContentSchema }; diff --git a/web/lib/ai/glm.ts b/web/lib/ai/glm.ts new file mode 100644 index 0000000..74ae4cd --- /dev/null +++ b/web/lib/ai/glm.ts @@ -0,0 +1,188 @@ +// §8.1 — GLM presets: multi-endpoint retry + working-endpoint cache, ported +// from legacy `engine/glm.js`. GLM keys come in flavors (z.ai coding plan, +// z.ai open platform, bigmodel.cn) that are not interchangeable and not +// reachable-diagnosable from inside a serverless function, so a preset tries +// its known-good flavors per call and remembers the winner on the warm +// instance (globalThis — survives across invocations on the same lambda). +// Owner note (spec §8.1): error 1113 = the Z.ai account has no coding-plan +// resource package; the app then runs on the fallback provider automatically. +import { + DEFAULT_TIMEOUT_MS, + RETRY_BACKOFF_MS, + ProviderError, + type CompleteRequest, + type CompleteResult, + type LlmProvider, + type ProviderId, +} from "./types"; + +export interface GlmEndpoint { + url: string; + model: string; +} + +const CODING_V4 = "https://api.z.ai/api/coding/paas/v4/chat/completions"; +const ZAI_PAS_V4 = "https://api.z.ai/api/paas/v4/chat/completions"; +const BIGMODEL_V4 = "https://open.bigmodel.cn/api/paas/v4/chat/completions"; + +/** + * Endpoint+model flavors to try, most likely first. The chosen model leads; + * the preset's sibling default model follows (legacy behaviour: an endpoint + * that rejects one model may still serve the other), then the alternate host. + */ +export function glmEndpointCandidates(presetId: "glm-coding" | "glm-open", model: string): GlmEndpoint[] { + if (presetId === "glm-open") { + return [ + { url: BIGMODEL_V4, model }, + { url: BIGMODEL_V4, model: model === "glm-4.6" ? "glm-4-flash" : "glm-4.6" }, + ]; + } + return [ + { url: CODING_V4, model }, + { url: CODING_V4, model: model === "glm-4.6" ? "glm-4.5-air" : "glm-4.6" }, + { url: ZAI_PAS_V4, model }, + { url: BIGMODEL_V4, model }, + ]; +} + +// ---- working-endpoint cache (globalThis, per warm instance) ----------------- + +const CACHE_KEY = "__episodekitGlmWorking"; + +interface GlmGlobal { + [CACHE_KEY]?: Record<string, GlmEndpoint>; +} + +function cache(): Record<string, GlmEndpoint> { + const g = globalThis as typeof globalThis & GlmGlobal; + if (!g[CACHE_KEY]) g[CACHE_KEY] = {}; + return g[CACHE_KEY]!; +} + +export function resetGlmWorkingCache(): void { + const g = globalThis as typeof globalThis & GlmGlobal; + delete g[CACHE_KEY]; +} + +function cachedEndpoint(presetId: ProviderId, candidates: GlmEndpoint[]): GlmEndpoint | null { + const saved = cache()[presetId]; + if (!saved) return null; + return candidates.find((c) => c.url === saved.url && c.model === saved.model) ?? null; +} + +export interface GlmProviderOptions { + presetId: "glm-coding" | "glm-open"; + model: string; + apiKey: () => string | undefined; + timeoutMs?: number; + retryBackoffMs?: number; + fetchImpl?: typeof fetch; + sleep?: (ms: number) => Promise<void>; +} + +interface ChatCompletionResponse { + choices?: Array<{ message?: { content?: string | null } }>; + usage?: { prompt_tokens?: number; completion_tokens?: number }; +} + +export function createGlmProvider(opts: GlmProviderOptions): LlmProvider { + const { presetId, model } = opts; + const id = presetId; + const timeoutMs = opts.timeoutMs ?? DEFAULT_TIMEOUT_MS; + const retryBackoffMs = opts.retryBackoffMs ?? RETRY_BACKOFF_MS; + const doFetch = opts.fetchImpl ?? fetch; + const sleep = opts.sleep ?? ((ms: number) => new Promise<void>((r) => setTimeout(r, ms))); + + function requestBody(ep: GlmEndpoint, req: CompleteRequest): string { + return JSON.stringify({ + model: ep.model, + messages: [ + { role: "system", content: req.system }, + { role: "user", content: req.user }, + ], + temperature: req.temperature ?? 0.4, + max_tokens: req.maxTokens ?? 4096, + ...(req.json ? { response_format: { type: "json_object" } } : {}), + }); + } + + function headers(key: string): Record<string, string> { + return { "Content-Type": "application/json", Authorization: `Bearer ${key}` }; + } + + async function postOnce(ep: GlmEndpoint, body: string, key: string, signal: AbortSignal): Promise<Response> { + return doFetch(ep.url, { method: "POST", headers: headers(key), body, signal }); + } + + function parseOk(data: ChatCompletionResponse, ep: GlmEndpoint): CompleteResult { + const content = data.choices?.[0]?.message?.content; + if (!content) throw new ProviderError("empty", `${id}:${new URL(ep.url).host}`, `${id}: empty completion`); + return { + text: content, + usage: data.usage + ? { in: data.usage.prompt_tokens ?? 0, out: data.usage.completion_tokens ?? 0 } + : undefined, + }; + } + + async function attempt(ep: GlmEndpoint, req: CompleteRequest, key: string): Promise<CompleteResult> { + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), timeoutMs); + try { + const body = requestBody(ep, req); + const res = await postOnce(ep, body, key, controller.signal); + if (!res.ok) { + const detail = (await res.text().catch(() => "")).slice(0, 200); + // Same retry contract as the generic adapter, per endpoint flavor: + // 1 retry on 429/5xx with backoff, then move to the next flavor. + if (res.status === 429 || res.status >= 500) { + await sleep(retryBackoffMs); + const res2 = await postOnce(ep, body, key, controller.signal); + if (res2.ok) return parseOk((await res2.json()) as ChatCompletionResponse, ep); + } + throw new ProviderError( + res.status === 429 ? "rate_limited" : res.status >= 500 ? "server_error" : "bad_request", + `${id}:${new URL(ep.url).host}`, + `${id}: http ${res.status} @ ${ep.url}: ${detail}`, + res.status, + detail, + ); + } + return parseOk((await res.json()) as ChatCompletionResponse, ep); + } catch (err) { + if (err instanceof ProviderError) throw err; + if (err instanceof Error && err.name === "AbortError") { + throw new ProviderError("timeout", `${id}:${new URL(ep.url).host}`, `${id}: timed out after ${timeoutMs}ms`); + } + throw new ProviderError( + "network", + `${id}:${new URL(ep.url).host}`, + `${id}: ${err instanceof Error ? err.message : String(err)}`, + ); + } finally { + clearTimeout(timer); + } + } + + return { + id, + async complete(req: CompleteRequest): Promise<CompleteResult> { + const key = opts.apiKey(); + if (!key) throw new ProviderError("no_key", id, `${id}: missing GLM_API_KEY`); + const candidates = glmEndpointCandidates(presetId, model); + const cached = cachedEndpoint(presetId, candidates); + const ordered = cached ? [cached, ...candidates.filter((c) => c !== cached)] : candidates; + const errors: string[] = []; + for (const ep of ordered) { + try { + const result = await attempt(ep, req, key); + cache()[presetId] = { url: ep.url, model: ep.model }; + return result; + } catch (err) { + errors.push(err instanceof Error ? err.message : String(err)); + } + } + throw new ProviderError("all_endpoints_failed", id, `${id}: all endpoints failed [${errors.join(" | ")}]`); + }, + }; +} diff --git a/web/lib/ai/json.ts b/web/lib/ai/json.ts new file mode 100644 index 0000000..29028b3 --- /dev/null +++ b/web/lib/ai/json.ts @@ -0,0 +1,29 @@ +// JSON extraction for model output: LLMs wrap JSON in prose or ``` fences even +// in json_object mode. Tolerant parse; strict validation happens in Zod after. +export function extractJson(text: string): unknown | null { + const trimmed = text.trim(); + const candidates = [trimmed]; + const fenced = trimmed.match(/```(?:json)?\s*([\s\S]*?)```/); + if (fenced?.[1]) candidates.push(fenced[1].trim()); + const first = trimmed.indexOf("{"); + const last = trimmed.lastIndexOf("}"); + if (first >= 0 && last > first) candidates.push(trimmed.slice(first, last + 1)); + for (const c of candidates) { + try { + const parsed = JSON.parse(c); + if (parsed && typeof parsed === "object") return parsed; + } catch { + // try the next candidate + } + } + return null; +} + +/** The single §8.1 "repair" follow-up prompt. */ +export function repairPrompt(originalUser: string, schemaHint: string, problem: string): string { + return ( + `${originalUser}\n\n` + + `Your previous reply was not usable (${problem}). Return valid JSON for this schema:\n` + + `${schemaHint}\nReturn ONLY the JSON object, no prose, no code fences.` + ); +} diff --git a/web/lib/ai/openai-compat.ts b/web/lib/ai/openai-compat.ts new file mode 100644 index 0000000..9bb4060 --- /dev/null +++ b/web/lib/ai/openai-compat.ts @@ -0,0 +1,106 @@ +// §8.1 — the ONE generic OpenAI-compatible adapter. Every non-GLM preset is a +// config object over this. Call policy per attempt: 60 s timeout, 1 retry on +// 429/5xx with a 2 s backoff. +import { + DEFAULT_MAX_TOKENS, + DEFAULT_TIMEOUT_MS, + RETRY_BACKOFF_MS, + ProviderError, + type CompleteRequest, + type CompleteResult, + type LlmProvider, +} from "./types"; + +export interface OpenAiCompatOptions { + id: string; + baseUrl: string; + /** env lookup wrapped in a closure so tests can inject; keys never leave env */ + apiKey: () => string | undefined; + jsonMode?: boolean; + timeoutMs?: number; + retryBackoffMs?: number; + maxRetries?: number; + fetchImpl?: typeof fetch; + sleep?: (ms: number) => Promise<void>; +} + +interface ChatCompletionResponse { + choices?: Array<{ message?: { content?: string | null } }>; + usage?: { prompt_tokens?: number; completion_tokens?: number }; + error?: { message?: string }; +} + +export function createOpenAiCompatProvider(opts: OpenAiCompatOptions): LlmProvider { + const { + id, + baseUrl, + jsonMode = true, + timeoutMs = DEFAULT_TIMEOUT_MS, + retryBackoffMs = RETRY_BACKOFF_MS, + maxRetries = 1, + } = opts; + const doFetch = opts.fetchImpl ?? fetch; + const sleep = opts.sleep ?? ((ms: number) => new Promise<void>((r) => setTimeout(r, ms))); + + return { + id, + async complete(req: CompleteRequest): Promise<CompleteResult> { + const key = opts.apiKey(); + if (!key) throw new ProviderError("no_key", id, `${id}: missing API key`); + + const body: Record<string, unknown> = { + model: req.model, + messages: [ + { role: "system", content: req.system }, + { role: "user", content: req.user }, + ], + temperature: req.temperature ?? 0.4, + max_tokens: req.maxTokens ?? DEFAULT_MAX_TOKENS, + ...(req.json && jsonMode ? { response_format: { type: "json_object" } } : {}), + }; + + let attempt = 0; + for (;;) { + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), timeoutMs); + const onOuterAbort = () => controller.abort(); + req.signal?.addEventListener("abort", onOuterAbort); + try { + const res = await doFetch(`${baseUrl}/chat/completions`, { + method: "POST", + headers: { "Content-Type": "application/json", Authorization: `Bearer ${key}` }, + body: JSON.stringify(body), + signal: controller.signal, + }); + if (!res.ok) { + const detail = (await res.text().catch(() => "")).slice(0, 200); + const retryable = res.status === 429 || res.status >= 500; + const code = res.status === 429 ? "rate_limited" : res.status >= 500 ? "server_error" : "bad_request"; + if (retryable && attempt < maxRetries) { + attempt++; + await sleep(retryBackoffMs); + continue; + } + throw new ProviderError(code, id, `${id}: http ${res.status}: ${detail}`, res.status, detail); + } + const data = (await res.json()) as ChatCompletionResponse; + const content = data.choices?.[0]?.message?.content; + if (!content) throw new ProviderError("empty", id, `${id}: empty completion`); + const usage = data.usage + ? { in: data.usage.prompt_tokens ?? 0, out: data.usage.completion_tokens ?? 0 } + : undefined; + return { text: content, usage }; + } catch (err) { + if (err instanceof ProviderError) throw err; + if (err instanceof Error && err.name === "AbortError") { + throw new ProviderError("timeout", id, `${id}: timed out after ${timeoutMs}ms`); + } + throw new ProviderError("network", id, `${id}: ${err instanceof Error ? err.message : String(err)}`); + } finally { + clearTimeout(timer); + req.signal?.removeEventListener("abort", onOuterAbort); + } + } + }, + }; +} diff --git a/web/lib/ai/owner.ts b/web/lib/ai/owner.ts new file mode 100644 index 0000000..39098fd --- /dev/null +++ b/web/lib/ai/owner.ts @@ -0,0 +1,98 @@ +// M13 / §7.5 — owner-only boundary for the admin AI routes. Reads the Clerk +// user from the auth context and checks its primary email against +// OWNER_EMAILS (comma/space-separated env). +// +// Deny-by-default tie-break (DECISIONS.md 2026-10-27 · WP-04): when Clerk is +// not configured (no keys — WP-02 ships the middleware) or OWNER_EMAILS is +// unset, the gate returns 503 and the route refuses. There is deliberately NO +// insecure fallback (header-based identity etc.); /admin lights up when WP-02 +// auth + the env land. +export interface ClerkUserLike { + id: string; + primaryEmailAddress?: string | null; +} + +export interface OwnerGateResult { + ok: boolean; + status: 200 | 401 | 403 | 503; + reason: + | "ok" + | "unauthenticated" + | "not_owner" + | "owner_not_configured" + | "auth_unavailable"; + email?: string; +} + +export interface OwnerGateDeps { + /** override for tests / alternate auth sources */ + currentUser?: () => Promise<ClerkUserLike | null>; + ownerEmails?: string[]; + env?: NodeJS.ProcessEnv; +} + +let warnedAuthUnavailable = false; + +async function defaultCurrentUser(): Promise<ClerkUserLike | null> { + // Align the availability check with WP-02's convention (lib/env.ts): without + // both Clerk keys the app fails closed here — keyless previews never own AI + // settings, even though app pages render with a preview user there. + const { isClerkConfigured } = (await import("@/lib/env")) as { + isClerkConfigured: () => boolean; + }; + if (!isClerkConfigured()) { + if (!warnedAuthUnavailable) { + warnedAuthUnavailable = true; + console.warn("[admin] Clerk not configured — admin AI routes stay locked (503)."); + } + throw new Error("clerk_not_configured"); + } + try { + // @clerk/nextjs/server resolves under the Next.js bundler; in plain node + // (unit tests) the import may fail and is treated as auth_unavailable. + const mod = (await import("@clerk/nextjs/server")) as { + currentUser: () => Promise<ClerkUserLike | null>; + }; + return await mod.currentUser(); + } catch (err) { + if (!warnedAuthUnavailable) { + warnedAuthUnavailable = true; + console.warn( + "[admin] Clerk unavailable — admin AI routes stay locked:", + err instanceof Error ? err.message : String(err), + ); + } + throw err; + } +} + +export function parseOwnerEmails(value: string | undefined): string[] { + if (!value) return []; + return value + .split(/[,\s]+/) + .map((e) => e.trim().toLowerCase()) + .filter(Boolean); +} + +export async function requireOwner(deps?: OwnerGateDeps): Promise<OwnerGateResult> { + const env = deps?.env ?? process.env; + const ownerEmails = deps?.ownerEmails ?? parseOwnerEmails(env.OWNER_EMAILS); + if (!ownerEmails.length) { + return { ok: false, status: 503, reason: "owner_not_configured" }; + } + + let user: ClerkUserLike | null; + try { + const getCurrent = deps?.currentUser ?? defaultCurrentUser; + user = await getCurrent(); + } catch { + return { ok: false, status: 503, reason: "auth_unavailable" }; + } + if (!user) return { ok: false, status: 401, reason: "unauthenticated" }; + + const email = (user.primaryEmailAddress ?? "").toLowerCase(); + if (!email || !ownerEmails.includes(email)) { + return { ok: false, status: 403, reason: "not_owner", email: email || undefined }; + } + return { ok: true, status: 200, reason: "ok", email }; +} diff --git a/web/lib/ai/postcheck.ts b/web/lib/ai/postcheck.ts new file mode 100644 index 0000000..993de27 --- /dev/null +++ b/web/lib/ai/postcheck.ts @@ -0,0 +1,129 @@ +// §8.4 post-check wiring for AI-generated content (WP-03's pure functions in +// lib/artifacts/postcheck are the primitives; this module applies them to a +// parsed artifact JSON tree). +// +// Safety note (WP-03 decision, kept): scans run on GENERATED framing only. +// Content stores QuoteRefs (offsets), never model quote text, and the model's +// `exact` strings are consumed by quote resolution and discarded — so walking +// the content tree never touches transcript wording. +import { + isReligionNeutral, + neverSayViolations, + religionViolations, +} from "@/lib/artifacts/postcheck"; +import type { VoiceProfile } from "./voice"; + +/** Collect every string in a content tree (objects, arrays, nested). */ +export function collectStrings(value: unknown): string[] { + const out: string[] = []; + const walk = (v: unknown): void => { + if (typeof v === "string") out.push(v); + else if (Array.isArray(v)) v.forEach(walk); + else if (v && typeof v === "object") Object.values(v).forEach(walk); + }; + walk(value); + return out; +} + +/** Deep-clone a content tree, mapping every string through `fn`. */ +export function mapStrings<T>(value: T, fn: (s: string) => string): T { + const walk = (v: unknown): unknown => { + if (typeof v === "string") return fn(v); + if (Array.isArray(v)) return v.map(walk); + if (v && typeof v === "object") { + return Object.fromEntries(Object.entries(v).map(([k, val]) => [k, walk(val)])); + } + return v; + }; + return walk(value) as T; +} + +export interface ContentViolations { + neverSay: string[]; + religion: string[]; +} + +export function scanContent(content: unknown, voice: VoiceProfile): ContentViolations { + const joined = collectStrings(content).join("\n"); + return { + neverSay: neverSayViolations(joined, voice.neverSay), + religion: isReligionNeutral(joined) ? [] : religionViolations(joined), + }; +} + +/** + * Strip URLs that are not in the show's allow list from every string. + * An allow-list entry also covers its own subpaths ("https://show.example" + * allows "https://show.example/ep-12") — the admin-listed link is the show's + * site, and links under it are not hallucinated foreign domains. + */ +export function stripForeignUrlsInContent<T>(content: T, allowedUrls: readonly string[]): T { + return mapStrings(content, (s) => { + const stripped = s.replace(URL_RE, (url) => { + const core = url.replace(/[.,;:!?)]+$/, ""); + const trailing = url.slice(core.length); // sentence punctuation stays + return isAllowedUrl(core, allowedUrls) ? core + trailing : trailing; + }); + if (stripped === s) return s; + return stripped + .replace(/[ \t]+([.,;:!?])/g, "$1") + .replace(/[ \t]{2,}/g, " ") + .replace(/^[ \t]+/gm, "") + .trim(); + }); +} + +const URL_RE = /https?:\/\/[^\s<>)"\]}]+/gi; + +function isAllowedUrl(url: string, allowed: readonly string[]): boolean { + if (!allowed.length) return false; + return allowed.some((a) => { + if (!a) return false; + if (url === a) return true; + return url.startsWith(a.endsWith("/") ? a : `${a}/`); + }); +} + +/** + * The full deterministic post-treatment: strip foreign URLs, then replace + * banned phrases at word boundaries (last resort of the §8.4 flow). Returns + * the transformed content — callers re-validate with Zod afterwards. + */ +export function applyDeterministicPostChecks<T>( + content: T, + voice: VoiceProfile, + allowedUrls: readonly string[], +): T { + let out = stripForeignUrlsInContent(content, allowedUrls); + const remaining = neverSayViolations(collectStrings(out).join("\n"), voice.neverSay); + if (remaining.length) { + out = replaceBannedPhrases(out, remaining); + } + return out; +} + +/** + * Replace banned phrases (word-boundary, case-insensitive) with "" and clean + * up the whitespace left behind. Honest last resort after one regenerate with + * feedback — grammar may suffer; the banned word never ships (U8). + */ +export function replaceBannedPhrases<T>(content: T, terms: readonly string[]): T { + const pattern = new RegExp( + terms + .map((t) => t.trim().split(/\s+/).map(escapeRegex).join("\\s+")) + .join("|"), + "gi", + ); + return mapStrings(content, (s) => + s + .replace(new RegExp(`(?<!\\w)(?:${pattern.source})(?!\\w)`, "gi"), "") + .replace(/[ \t]{2,}/g, " ") + .replace(/[ \t]+([.,;:!?])/g, "$1") + .replace(/^[ \t]+/gm, "") + .trim(), + ); +} + +function escapeRegex(s: string): string { + return s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); +} diff --git a/web/lib/ai/presets.ts b/web/lib/ai/presets.ts new file mode 100644 index 0000000..6d96a2f --- /dev/null +++ b/web/lib/ai/presets.ts @@ -0,0 +1,97 @@ +// §8.1 preset table — one generic OpenAI-compatible adapter + config presets, +// plus a native Anthropic Messages adapter. GLM presets add the multi-endpoint +// retry + working-endpoint cache ported from legacy `engine/glm.js`. +import type { ProviderId } from "./types"; + +export interface ProviderPreset { + id: ProviderId; + label: string; + baseUrl: string; + keyEnv: string; + defaultLong: string; + defaultFast: string; + adapter: "openai" | "anthropic" | "glm"; + /** whether `response_format: json_object` is supported */ + jsonMode: boolean; +} + +export const PROVIDER_PRESETS: Record<ProviderId, ProviderPreset> = { + "glm-coding": { + id: "glm-coding", + label: "GLM (Coding Plan)", + baseUrl: "https://api.z.ai/api/coding/paas/v4", + keyEnv: "GLM_API_KEY", + defaultLong: "glm-4.6", + defaultFast: "glm-4.5-air", + adapter: "glm", + jsonMode: true, + }, + "glm-open": { + id: "glm-open", + label: "GLM (open.bigmodel.cn)", + baseUrl: "https://open.bigmodel.cn/api/paas/v4", + keyEnv: "GLM_API_KEY", + defaultLong: "glm-4.6", + defaultFast: "glm-4-flash", + adapter: "glm", + jsonMode: true, + }, + groq: { + id: "groq", + label: "Groq", + // chosen from Groq's current production model list at build time + // (DECISIONS.md 2026-10-27 · WP-04): gpt-oss-120b long / gpt-oss-20b fast + baseUrl: "https://api.groq.com/openai/v1", + keyEnv: "GROQ_API_KEY", + defaultLong: "openai/gpt-oss-120b", + defaultFast: "openai/gpt-oss-20b", + adapter: "openai", + jsonMode: true, + }, + openai: { + id: "openai", + label: "OpenAI", + baseUrl: "https://api.openai.com/v1", + keyEnv: "OPENAI_API_KEY", + defaultLong: "gpt-4o", + defaultFast: "gpt-4o-mini", + adapter: "openai", + jsonMode: true, + }, + openrouter: { + id: "openrouter", + label: "OpenRouter", + baseUrl: "https://openrouter.ai/api/v1", + keyEnv: "OPENROUTER_API_KEY", + defaultLong: "anthropic/claude-sonnet-4.5", + defaultFast: "openai/gpt-4o-mini", + adapter: "openai", + jsonMode: true, + }, + anthropic: { + id: "anthropic", + label: "Anthropic", + baseUrl: "https://api.anthropic.com", + keyEnv: "ANTHROPIC_API_KEY", + defaultLong: "claude-sonnet-4-5", + defaultFast: "claude-haiku-4-5", + adapter: "anthropic", + // Messages API has no response_format: json_object — JSON is instructed + // by the prompt and enforced by the Zod + repair path. + jsonMode: false, + }, + custom: { + id: "custom", + label: "Custom (OpenAI-compatible)", + baseUrl: "", // from AI_CUSTOM_BASE_URL — see settings.resolveCustom + keyEnv: "AI_CUSTOM_API_KEY", + defaultLong: "", // from AI_CUSTOM_MODEL_LONG + defaultFast: "", // from AI_CUSTOM_MODEL_FAST + adapter: "openai", + jsonMode: true, + }, +}; + +export function providerIds(): ProviderId[] { + return Object.keys(PROVIDER_PRESETS) as ProviderId[]; +} diff --git a/web/lib/ai/prices.ts b/web/lib/ai/prices.ts new file mode 100644 index 0000000..3516e6c --- /dev/null +++ b/web/lib/ai/prices.ts @@ -0,0 +1,43 @@ +// §8.1 — per-model price table for `usage_events.cost_usd` estimates. +// Prices are USD per 1M tokens, list price, verified 2026-10-27 from the +// vendor pricing pages (Groq console docs, OpenAI docs, Anthropic docs). +// Unknown models cost 0 (spec: "0 if unknown"). +// +// GLM models: the default provider runs on the GLM Coding Plan (subscription, +// no per-token bill) and bigmodel.cn list prices were not verifiable at build +// time — both are logged as 0 rather than guessed (DECISIONS.md, WP-04). +import type { TokenUsage } from "./types"; + +export interface ModelPrice { + inPerM: number; + outPerM: number; + /** where the number came from — kept next to the data so it never goes stale silently */ + source: string; +} + +export const MODEL_PRICES: Record<string, ModelPrice> = { + // Groq (console.groq.com/docs/models, 2026-10-27) + "openai/gpt-oss-120b": { inPerM: 0.15, outPerM: 0.6, source: "groq docs 2026-10-27" }, + "openai/gpt-oss-20b": { inPerM: 0.075, outPerM: 0.3, source: "groq docs 2026-10-27" }, + // OpenAI (developers.openai.com/api/docs/pricing, 2026-10-27) + "gpt-4o": { inPerM: 2.5, outPerM: 10, source: "openai docs 2026-10-27" }, + "gpt-4o-mini": { inPerM: 0.15, outPerM: 0.6, source: "openai docs 2026-10-27" }, + "gpt-4.1": { inPerM: 2, outPerM: 8, source: "openai docs 2026-10-27" }, + "gpt-4.1-mini": { inPerM: 0.4, outPerM: 1.6, source: "openai docs 2026-10-27" }, + // Anthropic (platform.claude.com docs pricing, 2026-10-27) + "claude-sonnet-4-5": { inPerM: 3, outPerM: 15, source: "anthropic docs 2026-10-27" }, + "claude-haiku-4-5": { inPerM: 1, outPerM: 5, source: "anthropic docs 2026-10-27" }, + "claude-opus-4-5": { inPerM: 5, outPerM: 25, source: "anthropic docs 2026-10-27" }, +}; + +export function priceFor(model: string): ModelPrice | null { + return MODEL_PRICES[model] ?? null; +} + +/** Estimated cost in USD for one call; 0 when the model has no verified price. */ +export function estimateCost(model: string, usage?: TokenUsage): number { + if (!usage) return 0; + const p = priceFor(model); + if (!p) return 0; + return (usage.in * p.inPerM + usage.out * p.outPerM) / 1_000_000; +} diff --git a/web/lib/ai/registry.ts b/web/lib/ai/registry.ts new file mode 100644 index 0000000..b220681 --- /dev/null +++ b/web/lib/ai/registry.ts @@ -0,0 +1,85 @@ +// §8.1 registry: build a provider (or the [primary, fallback] chain) from an +// AiSelection. This is the ONLY place provider objects are constructed. +import { PROVIDER_PRESETS, ProviderPreset } from "./presets"; +import { resolveCustom, type AiSelection } from "./settings"; +import { createOpenAiCompatProvider } from "./openai-compat"; +import { createAnthropicProvider } from "./anthropic"; +import { createGlmProvider } from "./glm"; +import type { LlmProvider } from "./types"; + +export type ModelTier = "long" | "fast"; + +export interface RegistryOptions { + env?: NodeJS.ProcessEnv; + /** which model slot providers are constructed with (default "long") */ + tier?: ModelTier; + fetchImpl?: typeof fetch; + sleep?: (ms: number) => Promise<void>; + timeoutMs?: number; +} + +export function modelFor(selection: AiSelection, tier: ModelTier): string { + return tier === "fast" ? selection.fastModel : selection.longModel; +} + +/** Resolve the preset's base URL + models, applying `custom`'s env indirection. */ +export function resolvePreset(selection: AiSelection, env: NodeJS.ProcessEnv): ProviderPreset { + const preset = PROVIDER_PRESETS[selection.provider]; + if (selection.provider === "custom") { + const custom = resolveCustom(env); + if (!custom.baseUrl) { + throw new Error("custom provider requires AI_CUSTOM_BASE_URL"); + } + return { + ...preset, + baseUrl: custom.baseUrl, + defaultLong: custom.longModel, + defaultFast: custom.fastModel, + }; + } + return preset; +} + +export function buildProvider(selection: AiSelection, opts?: RegistryOptions & { tier?: ModelTier }): LlmProvider { + const env = opts?.env ?? process.env; + const preset = resolvePreset(selection, env); + const key = () => env[preset.keyEnv]; + const shared = { + fetchImpl: opts?.fetchImpl, + sleep: opts?.sleep, + timeoutMs: opts?.timeoutMs, + }; + + if (preset.adapter === "glm") { + return createGlmProvider({ + presetId: preset.id as "glm-coding" | "glm-open", + model: modelFor(selection, opts?.tier ?? "long"), + apiKey: key, + ...shared, + }); + } + if (preset.adapter === "anthropic") { + return createAnthropicProvider({ apiKey: key, ...shared }); + } + return createOpenAiCompatProvider({ + id: preset.id, + baseUrl: preset.baseUrl, + apiKey: key, + jsonMode: preset.jsonMode, + ...shared, + }); +} + +/** + * The §8.1 call-policy chain: [primary, fallback]. The fallback is skipped + * when it names the same provider as the primary (retrying the identical + * configuration is not a fallback). + */ +export function buildProviderChain(selection: AiSelection, opts?: RegistryOptions): LlmProvider[] { + const tier = opts?.tier ?? "long"; + const chain: LlmProvider[] = [buildProvider(selection, { ...opts, tier })]; + if (selection.fallbackProvider !== selection.provider) { + chain.push(buildProvider({ ...selection, provider: selection.fallbackProvider }, { ...opts, tier })); + } + return chain; +} diff --git a/web/lib/ai/settings-server.ts b/web/lib/ai/settings-server.ts new file mode 100644 index 0000000..b72a294 --- /dev/null +++ b/web/lib/ai/settings-server.ts @@ -0,0 +1,28 @@ +// Server-side resolution of the app_settings store. WP-02's Drizzle schema +// (`app_settings`: key text pk, value jsonb) is the persistence; DATABASE_URL +// unset (keyless preview) → null, so GET serves the env/default view and PUT +// reports 503 settings_store_unavailable. Generation reads env/defaults. +import { eq } from "drizzle-orm"; +import { getDb } from "@/db"; +import { appSettings } from "@/db/schema"; +import type { AppSettingsStore } from "./settings"; + +export function resolveAppSettingsStore(): AppSettingsStore | null { + if (!process.env.DATABASE_URL) return null; + return { + async get(key: string): Promise<unknown | null> { + const rows = await getDb() + .select({ value: appSettings.value }) + .from(appSettings) + .where(eq(appSettings.key, key)) + .limit(1); + return rows[0]?.value ?? null; + }, + async put(key: string, value: unknown): Promise<void> { + await getDb() + .insert(appSettings) + .values({ key, value }) + .onConflictDoUpdate({ target: appSettings.key, set: { value } }); + }, + }; +} diff --git a/web/lib/ai/settings.ts b/web/lib/ai/settings.ts new file mode 100644 index 0000000..74245d6 --- /dev/null +++ b/web/lib/ai/settings.ts @@ -0,0 +1,98 @@ +// §8.1 selection order: `app_settings.ai` (set in /admin) → env +// (AI_PROVIDER / AI_MODEL_LONG / AI_MODEL_FAST / AI_FALLBACK_PROVIDER) → +// defaults. Keys only ever live in env vars; the admin surface reports key +// presence, never values. +import { z } from "zod"; +import { PROVIDER_IDS, ProviderId } from "./types"; +import { PROVIDER_PRESETS, ProviderPreset } from "./presets"; + +/** `app_settings` row key holding the AI selection (§7.1: "ai provider selection (no secrets)"). */ +export const AI_SETTINGS_KEY = "ai"; + +export const AiSelectionSchema = z.object({ + provider: z.enum(PROVIDER_IDS as [ProviderId, ...ProviderId[]]), + longModel: z.string().min(1), + fastModel: z.string().min(1), + fallbackProvider: z.enum(PROVIDER_IDS as [ProviderId, ...ProviderId[]]), +}); +export type AiSelection = z.output<typeof AiSelectionSchema>; + +export const DEFAULT_AI_SELECTION: AiSelection = { + provider: "glm-coding", + longModel: PROVIDER_PRESETS["glm-coding"].defaultLong, + fastModel: PROVIDER_PRESETS["glm-coding"].defaultFast, + fallbackProvider: "groq", +}; + +/** + * Documented persistence interface for `app_settings`. WP-02's DB layer + * provides the Drizzle-backed implementation; until then the admin routes + * receive `null` (GET serves the env/default view, PUT returns 503) and the + * selection simply comes from env/defaults. No other module touches the DB. + */ +export interface AppSettingsStore { + get(key: string): Promise<unknown | null>; + put(key: string, value: unknown): Promise<void>; +} + +export type SelectionSource = "settings" | "env" | "default"; + +/** Env-based selection — only counts when AI_PROVIDER names a known preset. */ +export function selectionFromEnv(env: NodeJS.ProcessEnv): AiSelection | null { + const id = env.AI_PROVIDER as ProviderId | undefined; + if (!id || !PROVIDER_IDS.includes(id)) return null; + const preset = PROVIDER_PRESETS[id]; + let longModel = env.AI_MODEL_LONG || preset.defaultLong; + let fastModel = env.AI_MODEL_FAST || preset.defaultFast; + if (id === "custom") { + longModel = env.AI_CUSTOM_MODEL_LONG || longModel; + fastModel = env.AI_CUSTOM_MODEL_FAST || fastModel; + } + const fallback = env.AI_FALLBACK_PROVIDER as ProviderId | undefined; + return { + provider: id, + longModel, + fastModel, + fallbackProvider: fallback && PROVIDER_IDS.includes(fallback) ? fallback : DEFAULT_AI_SELECTION.fallbackProvider, + }; +} + +/** The `custom` preset resolves its base URL + models from env at use time. */ +export function resolveCustom(env: NodeJS.ProcessEnv): { baseUrl: string; longModel: string; fastModel: string } { + return { + baseUrl: env.AI_CUSTOM_BASE_URL || "", + longModel: env.AI_CUSTOM_MODEL_LONG || "", + fastModel: env.AI_CUSTOM_MODEL_FAST || "", + }; +} + +export interface EffectiveSelection { + selection: AiSelection; + source: SelectionSource; +} + +export async function effectiveAiSelection( + store?: AppSettingsStore | null, + opts?: { env?: NodeJS.ProcessEnv }, +): Promise<EffectiveSelection> { + const env = opts?.env ?? process.env; + if (store) { + try { + const raw = await store.get(AI_SETTINGS_KEY); + const parsed = AiSelectionSchema.safeParse(raw); + if (parsed.success) return { selection: parsed.data, source: "settings" }; + } catch { + // unreadable store must never break generation — env/defaults still apply + } + } + const fromEnv = selectionFromEnv(env); + if (fromEnv) return { selection: fromEnv, source: "env" }; + return { selection: DEFAULT_AI_SELECTION, source: "default" }; +} + +/** "key present ✓/✗" for the admin surface — booleans only, never values. */ +export function keyPresent(preset: ProviderPreset, env: NodeJS.ProcessEnv): boolean { + if (!env[preset.keyEnv]) return false; + if (preset.id === "custom") return Boolean(resolveCustom(env).baseUrl); // key AND base URL + return true; +} diff --git a/web/lib/ai/types.ts b/web/lib/ai/types.ts new file mode 100644 index 0000000..8a4ce11 --- /dev/null +++ b/web/lib/ai/types.ts @@ -0,0 +1,92 @@ +// §8.1 — every provider implements ONE interface. All AI calls in the app go +// through `lib/ai` (hard rule); keys only ever come from env vars. +export type ProviderId = + | "glm-coding" + | "glm-open" + | "groq" + | "openai" + | "openrouter" + | "anthropic" + | "custom"; + +export const PROVIDER_IDS: ProviderId[] = [ + "glm-coding", + "glm-open", + "groq", + "openai", + "openrouter", + "anthropic", + "custom", +]; + +export interface CompleteRequest { + system: string; + user: string; + model: string; + /** ask the provider for a JSON object (response_format json_object where supported) */ + json: boolean; + temperature?: number; + maxTokens?: number; + signal?: AbortSignal; +} + +export interface TokenUsage { + /** prompt tokens */ + in: number; + /** completion tokens */ + out: number; +} + +export interface CompleteResult { + text: string; + usage?: TokenUsage; +} + +export interface LlmProvider { + /** preset id, or a provider-qualified label for endpoint-level attempts */ + id: string; + complete(req: CompleteRequest): Promise<CompleteResult>; +} + +export type ProviderErrorCode = + | "no_key" + | "timeout" + | "rate_limited" + | "server_error" + | "bad_request" + | "empty" + | "network" + | "all_endpoints_failed" + | "provider_failed"; + +/** One failed provider attempt, surfaced for admin/debug. */ +export interface AttemptError { + provider: string; + code: ProviderErrorCode | string; + detail?: string; +} + +export class ProviderError extends Error { + readonly code: ProviderErrorCode; + readonly providerId: string; + readonly status?: number; + readonly detail?: string; + + constructor(code: ProviderErrorCode, providerId: string, message?: string, status?: number, detail?: string) { + super(message || `${providerId}: ${code}${status ? ` (http ${status})` : ""}`); + this.name = "ProviderError"; + this.code = code; + this.providerId = providerId; + this.status = status; + this.detail = detail; + } +} + +/** Default per-call output budget. Blog bodies (900–1300 words) fit well within it. */ +export const DEFAULT_MAX_TOKENS = 4096; + +/** §8.1 call policy: primary timeout is 60 s. */ +export const DEFAULT_TIMEOUT_MS = 60_000; + +/** 1 retry on 429/5xx with a 2 s backoff (§8.1). */ +export const RETRY_BACKOFF_MS = 2_000; diff --git a/web/lib/ai/usage.ts b/web/lib/ai/usage.ts new file mode 100644 index 0000000..676996f --- /dev/null +++ b/web/lib/ai/usage.ts @@ -0,0 +1,22 @@ +// §7.1 usage_events logging interface. The DB-backed logger lands with the +// WP-02 schema; until then every call is logged in a structured line so cost +// visibility never silently disappears. Loggers must never throw into the +// generation path (generate.ts guards anyway). +export type UsageKind = "asr" | "llm" | "free_tool"; + +export interface UsageEvent { + workspaceId: string | null; + episodeId: string | null; + kind: UsageKind; + provider: string; + model: string; + /** for llm: total tokens (prompt + completion); for asr: audio seconds */ + units: number; + costUsd: number; +} + +export type UsageLogger = (event: UsageEvent) => Promise<void> | void; + +export const consoleUsageLogger: UsageLogger = (event) => { + console.log(JSON.stringify({ usage_event: event })); +}; diff --git a/web/lib/ai/voice.ts b/web/lib/ai/voice.ts new file mode 100644 index 0000000..8a14daa --- /dev/null +++ b/web/lib/ai/voice.ts @@ -0,0 +1,160 @@ +// §7.4 voice profile — shape, defaults, learn + preview. +import { z } from "zod"; +import type { LlmProvider } from "./types"; + +export const VoiceProfileSchema = z.object({ + tone: z.array(z.string().min(1)).length(3), + formality: z.number().int().min(1).max(5), + energy: z.number().int().min(1).max(5), + humor: z.number().int().min(1).max(5), + person: z.enum(["I", "we", "they"]), + sentenceLength: z.enum(["short", "medium", "long"]), + emoji: z.enum(["none", "light", "heavy"]), + signaturePhrases: z.array(z.string().min(1)).max(5), + neverSay: z.array(z.string().min(1)).max(30), + alwaysInclude: z.array(z.string().min(1)).max(5), + ctaStyle: z.string().min(1).max(120), +}); +export type VoiceProfile = z.output<typeof VoiceProfileSchema>; + +export const DEFAULT_VOICE_PROFILE: VoiceProfile = { + tone: ["clear", "warm", "curious"], + formality: 3, + energy: 3, + humor: 2, + person: "we", + sentenceLength: "medium", + emoji: "none", + signaturePhrases: [], + neverSay: ["delve", "game-changer", "unlock", "dive deep", "in today's fast-paced world"], + alwaysInclude: [], + ctaStyle: "Invite listeners to follow the show.", +}; + +// ---- shared deps for the voice functions ------------------------------------ + +export interface VoiceDeps { + /** full provider chain override (tests) */ + chain?: LlmProvider[]; + /** registry inputs when no chain is given */ + env?: NodeJS.ProcessEnv; + fetchImpl?: typeof fetch; + sleep?: (ms: number) => Promise<void>; +} + +export type LearnVoiceResult = + | { ok: true; profile: VoiceProfile; provider: string; model: string; repaired: boolean } + | { ok: false; profile: VoiceProfile; provider: string | null; errorCode: string }; + +export type PreviewVoiceResult = + | { ok: true; text: string; provider: string; model: string } + | { ok: false; errorCode: string }; + +export function parseVoiceProfile(json: unknown): VoiceProfile | null { + const parsed = VoiceProfileSchema.safeParse(json); + return parsed.success ? parsed.data : null; +} + +// ---- voice learn + preview orchestration ------------------------------------ +// Provider plumbing for §7.4. Prompt builders receive the profile as a +// parameter, so there is no runtime import cycle with lib/prompts. +import { buildVoiceLearnPrompt } from "@/lib/prompts/voice-learn"; +import { buildVoicePreviewPrompt } from "@/lib/prompts/voice-preview"; +import type { ShowContext } from "@/lib/prompts"; +import { extractJson, repairPrompt } from "./json"; +import { buildProviderChain, type ModelTier } from "./registry"; +import type { AiSelection } from "./settings"; + +export interface VoiceChainDeps { + /** full provider chain override (tests, admin test button) */ + chain?: LlmProvider[]; + /** registry inputs used when no chain is given (with `selection`, required then) */ + selection?: AiSelection; + env?: NodeJS.ProcessEnv; + fetchImpl?: typeof fetch; + sleep?: (ms: number) => Promise<void>; + timeoutMs?: number; +} + +function chainFrom(deps: VoiceChainDeps | undefined, tier: ModelTier = "fast"): LlmProvider[] { + if (deps?.chain?.length) return deps.chain; + if (!deps?.selection) throw new Error("voice functions need a provider chain or a selection"); + return buildProviderChain(deps.selection, { + env: deps.env, + fetchImpl: deps.fetchImpl, + sleep: deps.sleep, + timeoutMs: deps.timeoutMs, + tier, + }); +} + +function modelFor(deps: VoiceChainDeps | undefined): string { + return deps?.selection ? deps.selection.fastModel : "glm-4.5-air"; +} + +const SCHEMA_HINT = `{"tone":["adjective","adjective","adjective"],"formality":1-5,"energy":1-5,"humor":1-5,"person":"I"|"we"|"they","sentenceLength":"short"|"medium"|"long","emoji":"none"|"light"|"heavy","signaturePhrases":[],"neverSay":[],"alwaysInclude":[],"ctaStyle":"..."}`; + +export async function learnVoiceProfile(args: { + samples: string[]; + deps?: VoiceChainDeps; +}): Promise<LearnVoiceResult> { + const chain = chainFrom(args.deps); + const model = modelFor(args.deps); + const prompt = buildVoiceLearnPrompt(args.samples); + const baseUser = prompt.user; + + for (const provider of chain) { + try { + const res = await provider.complete({ + system: prompt.system, + user: baseUser, + model, + json: true, + temperature: prompt.temperature, + }); + const parsed = parseVoiceProfile(extractJson(res.text)); + if (parsed) return { ok: true, profile: parsed, provider: provider.id, model, repaired: false }; + + // one §8.1 repair call, same provider + const repaired = await provider.complete({ + system: prompt.system, + user: repairPrompt(baseUser, SCHEMA_HINT, "not a valid voice profile JSON"), + model, + json: true, + temperature: prompt.temperature, + }); + const parsed2 = parseVoiceProfile(extractJson(repaired.text)); + if (parsed2) return { ok: true, profile: parsed2, provider: provider.id, model, repaired: true }; + } catch { + // provider failed — try the next one; total failure returns the default profile + } + } + return { ok: false, profile: DEFAULT_VOICE_PROFILE, provider: null, errorCode: "voice_learn_failed" }; +} + +export async function previewVoice(args: { + voice: VoiceProfile; + show?: ShowContext; + topic?: string; + deps?: VoiceChainDeps; +}): Promise<PreviewVoiceResult> { + const chain = chainFrom(args.deps); + const model = modelFor(args.deps); + const prompt = buildVoicePreviewPrompt(args.show ?? { name: "the show" }, args.voice, args.topic); + for (const provider of chain) { + try { + const res = await provider.complete({ + system: prompt.system, + user: prompt.user, + model, + json: false, + temperature: prompt.temperature, + }); + const text = res.text.trim(); + if (text) return { ok: true, text, provider: provider.id, model }; + } catch { + // next provider + } + } + return { ok: false, errorCode: "voice_preview_failed" }; +} diff --git a/web/lib/prompts/analyze.ts b/web/lib/prompts/analyze.ts new file mode 100644 index 0000000..7a5a4cf --- /dev/null +++ b/web/lib/prompts/analyze.ts @@ -0,0 +1,50 @@ +// §8.2 — analyze step (fast model, temperature 0.2): topics, candidate quotes, +// chapters when timestamps exist. Feeds `analysis` (§7.1) and quote backfill. +import { z } from "zod"; +import { buildSystemPrompt, assembleUserPrompt, TEMPERATURE_ANALYZE, type PromptContext } from "./index"; + +export const AnalyzeResultSchema = z.object({ + topics: z.array(z.string().min(1)).min(3).max(10), + candidateQuotes: z + .array(z.object({ segmentId: z.string().min(1), exact: z.string().min(1) })) + .min(3) + .max(15), + chapters: z + .array(z.object({ startMs: z.number().int().min(0), title: z.string().min(1).max(50) })) + .max(12) + .optional(), +}); +export type AnalyzeResult = z.output<typeof AnalyzeResultSchema>; + +const RULES = `Analyze this podcast episode transcript as JSON. +Rules: +- "topics": 3-10 topics the episode covers, 1-4 words each. +- "candidateQuotes": 5-12 of the most powerful moments, each {"segmentId": "...", "exact": "..."} with "exact" copied character-for-character from that numbered transcript line. +- "chapters": only if the lines carry timestamps: 5-12 chapters {"startMs" from the lines, "title" at most 50 chars}, first at 0, at least 2 minutes apart. Omit the key when there are no timestamps.`; + +const SCHEMA_EXAMPLE = `{ + "topics": ["topic one", "topic two"], + "candidateQuotes": [{ "segmentId": "seg_000123", "exact": "exact words from that segment" }], + "chapters": [{ "startMs": 0, "title": "Opening" }] +}`; + +export interface AnalyzePrompt { + system: string; + user: string; + schema: typeof AnalyzeResultSchema; + temperature: number; + tier: "fast"; + truncated: boolean; +} + +export function buildAnalyzePrompt(ctx: PromptContext): AnalyzePrompt { + const { user, truncated } = assembleUserPrompt({ rules: RULES, schemaExample: SCHEMA_EXAMPLE, ctx }); + return { + system: buildSystemPrompt(ctx.show, ctx.voice), + user, + schema: AnalyzeResultSchema, + temperature: TEMPERATURE_ANALYZE, + tier: "fast", + truncated, + }; +} diff --git a/web/lib/prompts/blog.ts b/web/lib/prompts/blog.ts new file mode 100644 index 0000000..42d2b1c --- /dev/null +++ b/web/lib/prompts/blog.ts @@ -0,0 +1,33 @@ +// §8.2 + §7.2 — blog post: 900–1300 words, H2s, SEO title ≤60, meta ≤155, +// slug, exactly 2 verified quotes via {{quote:n}} placeholders. +import { BlogSchema } from "@/lib/artifacts/schemas"; +import { assembleUserPrompt, TEMPERATURE_LONG, type AssembledPrompt, type PromptContext } from "./index"; + +const RULES = `Write a blog post from this episode as JSON. +Rules: +- "seoTitle": at most 60 characters. +- "metaDescription": at most 155 characters. +- "slug": kebab-case. +- "bodyMarkdown": 900-1300 words in Markdown with ## section headings. Where a verified quote belongs, put the placeholder {{quote:1}} or {{quote:2}} on its own line — the two quotes are inserted from "quoteRefs" by order (1 then 2). +- "quoteRefs": exactly 2 quotes, each {"segmentId": "...", "exact": "..."} copied character-for-character from the numbered transcript lines. +Use only facts from the transcript.`; + +const SCHEMA_EXAMPLE = `{ + "seoTitle": "SEO title, max 60 chars", + "metaDescription": "meta description, max 155 chars", + "slug": "episode-slug", + "bodyMarkdown": "intro\\n\\n## Section\\n\\ntext\\n\\n{{quote:1}}\\n\\nmore text\\n\\n{{quote:2}}", + "quoteRefs": [ + { "segmentId": "seg_000123", "exact": "exact words from that segment" }, + { "segmentId": "seg_000456", "exact": "another exact quote" } + ] +}`; + +export function buildBlogPrompt(ctx: PromptContext): AssembledPrompt & { + schema: typeof BlogSchema; + temperature: number; + tier: "long" | "fast"; +} { + const { user, truncated } = assembleUserPrompt({ rules: RULES, schemaExample: SCHEMA_EXAMPLE, ctx }); + return { user, truncated, schema: BlogSchema, temperature: TEMPERATURE_LONG, tier: "long" }; +} diff --git a/web/lib/prompts/chapters.ts b/web/lib/prompts/chapters.ts new file mode 100644 index 0000000..2ff27a7 --- /dev/null +++ b/web/lib/prompts/chapters.ts @@ -0,0 +1,27 @@ +// §8.2 + §7.2 — chapters: 5–12, first at 00:00, ≥2 min apart, title ≤50 chars. +import { ChaptersSchema } from "@/lib/artifacts/schemas"; +import { assembleUserPrompt, TEMPERATURE_LONG, type AssembledPrompt, type PromptContext } from "./index"; + +const RULES = `Write podcast chapters for this episode as JSON. +Rules: +- "items": 5 to 12 chapters. +- Use startMs values taken from the numbered transcript lines (milliseconds); the first chapter starts at 0. +- Chapters must be at least 2 minutes (120000 ms) apart. +- "title": at most 50 characters, descriptive, no clickbait. +- Only chapter ranges the transcript actually covers.`; + +const SCHEMA_EXAMPLE = `{ + "items": [ + { "startMs": 0, "title": "Opening" }, + { "startMs": 180000, "title": "Main theme" } + ] +}`; + +export function buildChaptersPrompt(ctx: PromptContext): AssembledPrompt & { + schema: typeof ChaptersSchema; + temperature: number; + tier: "long" | "fast"; +} { + const { user, truncated } = assembleUserPrompt({ rules: RULES, schemaExample: SCHEMA_EXAMPLE, ctx }); + return { user, truncated, schema: ChaptersSchema, temperature: TEMPERATURE_LONG, tier: "long" }; +} diff --git a/web/lib/prompts/index.ts b/web/lib/prompts/index.ts new file mode 100644 index 0000000..5a79c30 --- /dev/null +++ b/web/lib/prompts/index.ts @@ -0,0 +1,244 @@ +// §8.2 prompts. Shared system prompt (verbatim skeleton from the spec with +// show/voice substitutions), the per-artifact user-prompt assembly, and the +// transcript budget rule. +// +// Weak-model hardening (GLM/Groq): low temperature (0.4 long, 0.2 analyze), +// flat schemas, one artifact per call, JSON shape examples inline. +import type { z } from "zod"; +import type { ArtifactKind } from "@/lib/artifacts/schemas"; +import { rankedSentences, sentences, themes, normalizeWs } from "@/lib/engine"; +import { formatTranscriptLines } from "@/lib/quotes/format"; +import type { Segment } from "@/lib/quotes/types"; +import type { VoiceProfile } from "@/lib/ai/voice"; +import { buildShowNotesPrompt } from "./show-notes"; +import { buildChaptersPrompt } from "./chapters"; +import { buildSummaryPrompt } from "./summary"; +import { buildTitlesPrompt } from "./titles"; +import { buildBlogPrompt } from "./blog"; +import { buildNewsletterPrompt } from "./newsletter"; +import { buildSocialPrompt } from "./social"; +import { buildQuotesPrompt } from "./quotes"; + +/** Bump when prompt text or rules change; recorded on generation results. */ +export const PROMPT_VERSION = "2026-10-27.1"; + +export const TEMPERATURE_LONG = 0.4; +export const TEMPERATURE_ANALYZE = 0.2; + +// ---- transcript budget (§8.2) ----------------------------------------------- + +/** ~60k tokens, estimated at 4 chars/token. */ +export const TRANSCRIPT_TOKEN_BUDGET = 60_000; + +export function estimateTokens(text: string): number { + return Math.ceil(text.length / 4); +} + +export interface BudgetedTranscript { + lines: string[]; + truncated: boolean; +} + +/** + * Budget rule (§8.2): if the transcript exceeds ~60k tokens, keep the first + * 40% of lines + the last 20% + top-ranked middle segments (engine + * `rankedSentences`) until the budget is filled. Always records `truncated`. + */ +export function budgetTranscriptLines( + lines: string[], + opts?: { maxTokens?: number; ranked?: Array<{ s: string; i: number }> }, +): BudgetedTranscript { + const maxTokens = opts?.maxTokens ?? TRANSCRIPT_TOKEN_BUDGET; + const full = lines.join("\n"); + if (estimateTokens(full) <= maxTokens) return { lines, truncated: false }; + + const headCount = Math.floor(lines.length * 0.4); + const tailCount = Math.floor(lines.length * 0.2); + const head = lines.slice(0, headCount); + const tail = lines.slice(lines.length - tailCount); + const middle = lines.slice(headCount, lines.length - tailCount); + + const headPart = [...head]; + // extreme edge first: head+tail alone exceed the budget — shrink the head + // from its END (the episode opening always survives) + while (headPart.length > 1 && estimateTokens([...headPart, ...tail].join("\n")) > maxTokens) { + headPart.pop(); + } + const kept = [...headPart, ...tail]; + let budget = maxTokens - estimateTokens(kept.join("\n")); + + if (budget > 0 && middle.length) { + const middleSet = new Set(middle); + const ranked = opts?.ranked + ? opts.ranked.filter((r) => middleSet.has(r.s)) + : rankLines(middle).map((r) => ({ s: r.s, i: r.i })); + const picked: string[] = []; + for (const r of ranked) { + const cost = estimateTokens(r.s + "\n"); + if (cost > budget) continue; + picked.push(r.s); + budget -= cost; + } + // keep episode order: middle lines in original sequence + const pickedSet = new Set(picked); + kept.splice(headPart.length, 0, ...middle.filter((l) => pickedSet.has(l))); + } + + return { lines: kept, truncated: true }; +} + +function rankLines(lines: string[]): Array<{ s: string; i: number; score: number }> { + const text = normalizeWs(lines.join(" ")); + const th = themes(text, 10).filter((t) => t.length >= 5); + return rankedSentences(sentences(text), th); +} + +// ---- context ---------------------------------------------------------------- + +export interface ShowContext { + name: string; + defaultCta?: string | null; + /** URLs the show may reference; foreign links are stripped post-generation */ + allowedUrls?: string[]; + vocabulary?: string[]; +} + +export interface EpisodeMeta { + title?: string | null; + number?: number | null; + guests?: string[]; +} + +export interface PromptContext { + show: ShowContext; + voice: VoiceProfile; + /** 1–3 past show notes; ≤2 samples, ≤400 words each make it into the prompt */ + voiceSamples?: string[]; + meta?: EpisodeMeta; + segments: Segment[]; + /** regenerate instruction (≤200 chars) */ + instruction?: string; +} + +// ---- shared system prompt (§8.2, verbatim skeleton) ------------------------- + +function fmtList(list: readonly string[]): string { + return list.length ? list.join(", ") : "none"; +} + +export function buildSystemPrompt(show: ShowContext, voice: VoiceProfile): string { + return [ + `You write podcast content for the show "${show.name}". Match the show's voice: tone ${voice.tone.join(", ")};`, + `formality ${voice.formality}/5; energy ${voice.energy}/5; humor ${voice.humor}/5; write as "${voice.person}"; ${voice.sentenceLength}`, + `sentences; emoji: ${voice.emoji}. You may use these phrases sparingly: ${fmtList(voice.signaturePhrases)}.`, + `Never use: ${fmtList(voice.neverSay)}. Include where natural: ${fmtList(voice.alwaysInclude)}.`, + `Use only facts from the transcript. Never invent names, numbers, links or claims.`, + `To quote, return {"segmentId": "...", "exact": "..."} with text copied character-for-character`, + `from that segment. Anything not copied exactly is discarded.`, + `Return only JSON matching the schema below.`, + ].join("\n"); +} + +// ---- user prompt assembly (§8.2) -------------------------------------------- + +const MAX_SAMPLES = 2; +const MAX_SAMPLE_WORDS = 400; + +function clipWords(text: string, maxWords: number): string { + const words = text.split(/\s+/); + return words.length <= maxWords ? text : words.slice(0, maxWords).join(" "); +} + +export interface AssembledPrompt { + user: string; + truncated: boolean; +} + +export function assembleUserPrompt(parts: { + rules: string; + schemaExample: string; + ctx: PromptContext; + maxTokens?: number; +}): AssembledPrompt { + const { rules, schemaExample, ctx } = parts; + const blocks: string[] = [rules.trim(), `JSON schema:\n${schemaExample.trim()}`]; + + const meta: string[] = []; + if (ctx.meta?.title) meta.push(`Episode title: ${ctx.meta.title}`); + if (ctx.meta?.number != null) meta.push(`Episode number: ${ctx.meta.number}`); + if (ctx.meta?.guests?.length) meta.push(`Guests: ${ctx.meta.guests.join(", ")}`); + if (ctx.show.defaultCta) meta.push(`Show CTA: ${ctx.show.defaultCta}`); + if (ctx.show.vocabulary?.length) meta.push(`Vocabulary (spell exactly): ${ctx.show.vocabulary.join(", ")}`); + if (meta.length) blocks.push(meta.join("\n")); + + const samples = (ctx.voiceSamples ?? []).slice(0, MAX_SAMPLES).map((s) => clipWords(s.trim(), MAX_SAMPLE_WORDS)); + if (samples.length) { + blocks.push( + samples + .map((s, i) => `Style reference ${i + 1} — style reference only (do not copy content):\n${s}`) + .join("\n\n"), + ); + } + + if (ctx.instruction) { + blocks.push(`Additional instruction from the user (follow it): ${ctx.instruction.trim()}`); + } + + let truncated = false; + if (ctx.segments.length) { + const numbered = formatTranscriptLines(ctx.segments); + const budget = budgetTranscriptLines(numbered.split("\n"), { maxTokens: parts.maxTokens }); + blocks.push(`TRANSCRIPT:\n${budget.lines.join("\n")}`); + truncated = budget.truncated; + } + + return { user: blocks.join("\n\n"), truncated }; +} + +// ---- per-artifact dispatch -------------------------------------------------- + +export interface ArtifactPrompt { + system: string; + user: string; + /** zod schema the parsed JSON must satisfy */ + schema: z.ZodTypeAny; + temperature: number; + tier: "long" | "fast"; + truncated: boolean; +} + +interface BuiltPrompt { + user: string; + /** zod schema the parsed JSON must satisfy (§7.2 shape for this kind) */ + schema: z.ZodTypeAny; + temperature: number; + tier: "long" | "fast"; + truncated?: boolean; +} +type PromptBuilder = (ctx: PromptContext) => BuiltPrompt; + +function builderFor(kind: ArtifactKind): PromptBuilder { + const map: Record<ArtifactKind, PromptBuilder> = { + show_notes: buildShowNotesPrompt, + chapters: buildChaptersPrompt, + summary: buildSummaryPrompt, + titles: buildTitlesPrompt, + blog: buildBlogPrompt, + newsletter: buildNewsletterPrompt, + social: buildSocialPrompt, + quotes: buildQuotesPrompt, + }; + return map[kind]; +} + +export function buildArtifactPrompt(kind: ArtifactKind, ctx: PromptContext): ArtifactPrompt { + const built = builderFor(kind)(ctx); + return { + system: buildSystemPrompt(ctx.show, ctx.voice), + user: built.user, + schema: built.schema, + temperature: built.temperature, + tier: built.tier, + truncated: built.truncated ?? false, + }; +} diff --git a/web/lib/prompts/newsletter.ts b/web/lib/prompts/newsletter.ts new file mode 100644 index 0000000..9c8f18a --- /dev/null +++ b/web/lib/prompts/newsletter.ts @@ -0,0 +1,31 @@ +// §8.2 + §7.2 — newsletter: 3 subjects, chosen index, preview ≤90, body 250–400 +// words, exactly 1 verified quote, CTA text ≤30. +import { NewsletterSchema } from "@/lib/artifacts/schemas"; +import { assembleUserPrompt, TEMPERATURE_LONG, type AssembledPrompt, type PromptContext } from "./index"; + +const RULES = `Write a newsletter email from this episode as JSON. +Rules: +- "subjects": exactly 3 subject-line options. +- "chosen": index (0, 1 or 2) of the strongest subject. +- "previewText": at most 90 characters. +- "bodyMarkdown": 250-400 words. Where the verified quote belongs, put the placeholder {{quote:1}} on its own line. +- "quoteRefs": exactly 1 quote, {"segmentId": "...", "exact": "..."} copied character-for-character from a numbered transcript line. +- "ctaText": at most 30 characters, consistent with the show CTA.`; + +const SCHEMA_EXAMPLE = `{ + "subjects": ["subject 1", "subject 2", "subject 3"], + "chosen": 0, + "previewText": "preview, max 90 chars", + "bodyMarkdown": "email body\\n\\n{{quote:1}}\\n\\nclosing", + "quoteRefs": [{ "segmentId": "seg_000123", "exact": "exact words from that segment" }], + "ctaText": "Listen now" +}`; + +export function buildNewsletterPrompt(ctx: PromptContext): AssembledPrompt & { + schema: typeof NewsletterSchema; + temperature: number; + tier: "long" | "fast"; +} { + const { user, truncated } = assembleUserPrompt({ rules: RULES, schemaExample: SCHEMA_EXAMPLE, ctx }); + return { user, truncated, schema: NewsletterSchema, temperature: TEMPERATURE_LONG, tier: "long" }; +} diff --git a/web/lib/prompts/quotes.ts b/web/lib/prompts/quotes.ts new file mode 100644 index 0000000..52967a7 --- /dev/null +++ b/web/lib/prompts/quotes.ts @@ -0,0 +1,25 @@ +// §8.2 + §7.3 — quotes: 5 pull quotes the model claims as {segmentId, exact}; +// the server resolves them to verbatim offsets (model text never displayed). +import { QuotesSchema } from "@/lib/artifacts/schemas"; +import { assembleUserPrompt, TEMPERATURE_LONG, type AssembledPrompt, type PromptContext } from "./index"; + +const RULES = `Pick the 5 most powerful moments from this episode as JSON. +Rules: +- "items": exactly 5 quotes, each {"segmentId": "...", "exact": "..."}. +- "exact" must be the text copied character-for-character from that numbered transcript line — same words, same punctuation. Anything not copied exactly is discarded and replaced. +- Prefer complete thoughts that stand alone, roughly 40-240 characters.`; + +const SCHEMA_EXAMPLE = `{ + "items": [ + { "segmentId": "seg_000123", "exact": "exact words copied from that segment" } + ] +}`; + +export function buildQuotesPrompt(ctx: PromptContext): AssembledPrompt & { + schema: typeof QuotesSchema; + temperature: number; + tier: "long" | "fast"; +} { + const { user, truncated } = assembleUserPrompt({ rules: RULES, schemaExample: SCHEMA_EXAMPLE, ctx }); + return { user, truncated, schema: QuotesSchema, temperature: TEMPERATURE_LONG, tier: "long" }; +} diff --git a/web/lib/prompts/shorten.ts b/web/lib/prompts/shorten.ts new file mode 100644 index 0000000..82f9664 --- /dev/null +++ b/web/lib/prompts/shorten.ts @@ -0,0 +1,20 @@ +// §8.2 — shorten-to-fit (U13): copy a text to a hard character limit in the +// show's voice. Plain text in, plain text out. +import { buildSystemPrompt, type ShowContext } from "./index"; +import type { VoiceProfile } from "@/lib/ai/voice"; + +export function buildShortenPrompt(args: { + show: ShowContext; + voice: VoiceProfile; + text: string; + maxChars: number; +}): { system: string; user: string; json: false; temperature: number; tier: "fast" } { + const system = buildSystemPrompt(args.show, args.voice); + const user = [ + `Shorten the TEXT below to at most ${args.maxChars} characters.`, + `Keep the meaning, the voice, and any structure. Do not add facts.`, + `Return ONLY the shortened text — no JSON, no explanations, no character counts.`, + `TEXT:\n${args.text}`, + ].join("\n"); + return { system, user, json: false, temperature: 0.2, tier: "fast" }; +} diff --git a/web/lib/prompts/show-notes.ts b/web/lib/prompts/show-notes.ts new file mode 100644 index 0000000..a17d688 --- /dev/null +++ b/web/lib/prompts/show-notes.ts @@ -0,0 +1,29 @@ +// §8.2 + §7.2 — show notes: hook, 4–7 bullets, optional guest line, ≤1 quote, CTA. +import { ShowNotesSchema } from "@/lib/artifacts/schemas"; +import { assembleUserPrompt, TEMPERATURE_LONG, type AssembledPrompt, type PromptContext } from "./index"; + +const RULES = `Write show notes for this episode as JSON. +Rules: +- "hook": 2-3 sentences that make someone press play. +- "bullets": 4-7 "In this episode" bullets; one concrete takeaway each, no fluff. +- "guestLine": one sentence introducing the guest — only when there is a guest; omit the key otherwise. +- "quoteRefs": at most 1 quote, as {"segmentId": "...", "exact": "..."} copied character-for-character from that numbered transcript line. +- "cta": call to action, consistent with the show CTA given below. +Chapters are rendered separately — do not include them.`; + +const SCHEMA_EXAMPLE = `{ + "hook": "2-3 sentence hook", + "bullets": ["bullet 1", "bullet 2", "bullet 3", "bullet 4"], + "guestLine": "one line about the guest (omit if no guest)", + "quoteRefs": [{ "segmentId": "seg_000123", "exact": "exact words from that segment" }], + "cta": "call to action" +}`; + +export function buildShowNotesPrompt(ctx: PromptContext): AssembledPrompt & { + schema: typeof ShowNotesSchema; + temperature: number; + tier: "long" | "fast"; +} { + const { user, truncated } = assembleUserPrompt({ rules: RULES, schemaExample: SCHEMA_EXAMPLE, ctx }); + return { user, truncated, schema: ShowNotesSchema, temperature: TEMPERATURE_LONG, tier: "long" }; +} diff --git a/web/lib/prompts/social.ts b/web/lib/prompts/social.ts new file mode 100644 index 0000000..5f23437 --- /dev/null +++ b/web/lib/prompts/social.ts @@ -0,0 +1,27 @@ +// §8.2 + §7.2 — social: LinkedIn ≤1300, thread 4–6 posts ≤280, Instagram +// caption ≤2200 + 5–10 hashtags. +import { SocialSchema } from "@/lib/artifacts/schemas"; +import { assembleUserPrompt, TEMPERATURE_LONG, type AssembledPrompt, type PromptContext } from "./index"; + +const RULES = `Write social posts for this episode as JSON. +Rules: +- "linkedin": at most 1300 characters, one post. +- "thread": 4 to 6 short posts, each at most 280 characters, telling one story in order. +- "instagram.caption": at most 2200 characters. +- "instagram.hashtags": 5 to 10 hashtags, relevant to the episode. +Quotes must be copied character-for-character from the transcript if used.`; + +const SCHEMA_EXAMPLE = `{ + "linkedin": "LinkedIn post, max 1300 chars", + "thread": ["post 1 (max 280)", "post 2 (max 280)", "post 3 (max 280)", "post 4 (max 280)"], + "instagram": { "caption": "caption, max 2200 chars", "hashtags": ["#tag1", "#tag2", "#tag3", "#tag4", "#tag5"] } +}`; + +export function buildSocialPrompt(ctx: PromptContext): AssembledPrompt & { + schema: typeof SocialSchema; + temperature: number; + tier: "long" | "fast"; +} { + const { user, truncated } = assembleUserPrompt({ rules: RULES, schemaExample: SCHEMA_EXAMPLE, ctx }); + return { user, truncated, schema: SocialSchema, temperature: TEMPERATURE_LONG, tier: "long" }; +} diff --git a/web/lib/prompts/summary.ts b/web/lib/prompts/summary.ts new file mode 100644 index 0000000..a1f1304 --- /dev/null +++ b/web/lib/prompts/summary.ts @@ -0,0 +1,25 @@ +// §8.2 + §7.2 — summary: one-liner ≤140, short 50–80 words, long 150–220 words. +import { SummarySchema } from "@/lib/artifacts/schemas"; +import { assembleUserPrompt, TEMPERATURE_LONG, type AssembledPrompt, type PromptContext } from "./index"; + +const RULES = `Summarize this episode as JSON. +Rules: +- "oneLiner": at most 140 characters, what the episode is about. +- "short": 50-80 words. +- "long": 150-220 words. +Use only facts from the transcript.`; + +const SCHEMA_EXAMPLE = `{ + "oneLiner": "one sentence, max 140 chars", + "short": "50-80 word summary", + "long": "150-220 word summary" +}`; + +export function buildSummaryPrompt(ctx: PromptContext): AssembledPrompt & { + schema: typeof SummarySchema; + temperature: number; + tier: "long" | "fast"; +} { + const { user, truncated } = assembleUserPrompt({ rules: RULES, schemaExample: SCHEMA_EXAMPLE, ctx }); + return { user, truncated, schema: SummarySchema, temperature: TEMPERATURE_LONG, tier: "long" }; +} diff --git a/web/lib/prompts/titles.ts b/web/lib/prompts/titles.ts new file mode 100644 index 0000000..ec914d3 --- /dev/null +++ b/web/lib/prompts/titles.ts @@ -0,0 +1,27 @@ +// §8.2 + §7.2 — titles: exactly 8, ≤80 chars, 6 styles, star the best. +import { TitlesSchema } from "@/lib/artifacts/schemas"; +import { assembleUserPrompt, TEMPERATURE_LONG, type AssembledPrompt, type PromptContext } from "./index"; + +const RULES = `Write episode title options as JSON. +Rules: +- "items": exactly 8 titles. +- "text": at most 80 characters each. +- "style": one of "Curiosity", "Benefit", "Guest", "Number", "Question", "Quote" — vary the styles across the 8. +- "starred": true for the single strongest title, false for all others. +No clickbait that the transcript cannot back up.`; + +const SCHEMA_EXAMPLE = `{ + "items": [ + { "text": "title under 80 chars", "style": "Curiosity", "starred": true }, + { "text": "another title", "style": "Benefit", "starred": false } + ] +}`; + +export function buildTitlesPrompt(ctx: PromptContext): AssembledPrompt & { + schema: typeof TitlesSchema; + temperature: number; + tier: "long" | "fast"; +} { + const { user, truncated } = assembleUserPrompt({ rules: RULES, schemaExample: SCHEMA_EXAMPLE, ctx }); + return { user, truncated, schema: TitlesSchema, temperature: TEMPERATURE_LONG, tier: "long" }; +} diff --git a/web/lib/prompts/voice-learn.ts b/web/lib/prompts/voice-learn.ts new file mode 100644 index 0000000..5a145b7 --- /dev/null +++ b/web/lib/prompts/voice-learn.ts @@ -0,0 +1,66 @@ +// §8.2 + §7.4 — voice learn: 1–3 past show notes → the voice profile JSON. +import { buildSystemPrompt, assembleUserPrompt, TEMPERATURE_ANALYZE, type PromptContext } from "./index"; +import { VoiceProfileSchema, type VoiceProfile } from "@/lib/ai/voice"; +import { DEFAULT_VOICE_PROFILE } from "@/lib/ai/voice"; + +const RULES = `Learn this author's voice from their past show notes and return their voice profile as JSON. +Rules: +- "tone": exactly 3 adjectives. +- "formality", "energy", "humor": integers 1-5 (5 = most). +- "person": the pronoun they write as: "I", "we" or "they". +- "sentenceLength": "short", "medium" or "long". +- "emoji": "none", "light" or "heavy". +- "signaturePhrases": up to 5 phrases they actually reuse. +- "neverSay": up to 30 words or phrases this author would never use (stock AI phrasing goes here). +- "alwaysInclude": up to 5 phrases they consistently use. +- "ctaStyle": how they close, at most 120 characters.`; + +const SCHEMA_EXAMPLE = `{ + "tone": ["clear", "warm", "direct"], + "formality": 3, + "energy": 3, + "humor": 2, + "person": "we", + "sentenceLength": "medium", + "emoji": "none", + "signaturePhrases": [], + "neverSay": ["delve"], + "alwaysInclude": [], + "ctaStyle": "Invite listeners to follow the show." +}`; + +/** Samples stand in for transcript segments: same numbered-line treatment is + * unnecessary, so a minimal context is synthesized (no metadata, no CTA). */ +export function buildVoiceLearnPrompt( + samples: string[], + voice: VoiceProfile = DEFAULT_VOICE_PROFILE, +): { + system: string; + user: string; + schema: typeof VoiceProfileSchema; + temperature: number; + tier: "fast"; +} { + const ctx: PromptContext = { + show: { name: "the show" }, + voice, + segments: [], + }; + const { user } = assembleUserPrompt({ + rules: RULES, + schemaExample: SCHEMA_EXAMPLE, + ctx, + maxTokens: 8_000, // samples are short; a tight budget keeps this fast + }); + const sampleBlocks = samples + .slice(0, 3) + .map((s, i) => `Writing sample ${i + 1}:\n${s.trim()}`) + .join("\n\n"); + return { + system: buildSystemPrompt({ name: "the show" }, voice), + user: `${user}\n\n${sampleBlocks}`, + schema: VoiceProfileSchema, + temperature: TEMPERATURE_ANALYZE, + tier: "fast", + }; +} diff --git a/web/lib/prompts/voice-preview.ts b/web/lib/prompts/voice-preview.ts new file mode 100644 index 0000000..3527971 --- /dev/null +++ b/web/lib/prompts/voice-preview.ts @@ -0,0 +1,18 @@ +// §8.2 + §7.4 — voice preview: write one short sample paragraph IN the voice +// (plain text, not JSON) so the user can check the learned profile (M9). +import { buildSystemPrompt, type ShowContext } from "./index"; +import type { VoiceProfile } from "@/lib/ai/voice"; + +export function buildVoicePreviewPrompt( + show: ShowContext, + voice: VoiceProfile, + topic?: string, +): { system: string; user: string; json: false; temperature: number; tier: "fast" } { + const system = buildSystemPrompt(show, voice); + const user = [ + `Write ONE short sample paragraph (2-4 sentences) in this show's voice, as if introducing a new episode.`, + topic ? `Topic of the sample: ${topic}.` : `Pick a plausible episode topic for this show.`, + `Plain text only — no JSON, no headings, no quotes syntax.`, + ].join(" "); + return { system, user, json: false, temperature: 0.4, tier: "fast" }; +} diff --git a/web/tests/unit/ai.adapters.test.ts b/web/tests/unit/ai.adapters.test.ts new file mode 100644 index 0000000..f6a5a1e --- /dev/null +++ b/web/tests/unit/ai.adapters.test.ts @@ -0,0 +1,195 @@ +// Adapter behaviour: OpenAI-compatible generic adapter (timeout, 1 retry on +// 429/5xx with backoff, usage mapping, json mode), anthropic Messages adapter, +// GLM multi-endpoint retry + working-endpoint cache (ported from engine/glm.js). +import { afterEach, describe, expect, it } from "vitest"; +import { createOpenAiCompatProvider } from "@/lib/ai/openai-compat"; +import { createAnthropicProvider } from "@/lib/ai/anthropic"; +import { createGlmProvider, resetGlmWorkingCache } from "@/lib/ai/glm"; +import { ProviderError } from "@/lib/ai/types"; +import { chatJson, noSleep, scriptedFetch, TOKENS } from "./helpers/ai"; + +afterEach(() => resetGlmWorkingCache()); + +const ok = () => scriptedFetch([chatJson("{\"ok\":true}", TOKENS)]); + +describe("OpenAI-compatible adapter", () => { + const base = { id: "test", baseUrl: "https://api.example/v1", apiKey: () => "k", sleep: noSleep }; + + it("posts system/user messages with json mode and maps usage", async () => { + const fetchImpl = ok(); + const p = createOpenAiCompatProvider({ ...base, fetchImpl }); + const res = await p.complete({ system: "sys", user: "usr", model: "m1", json: true }); + expect(res.text).toBe("{\"ok\":true}"); + expect(res.usage).toEqual({ in: 100, out: 50 }); + const body = JSON.parse(String(fetchImpl.calls[0].init.body)); + expect(body.messages).toEqual([ + { role: "system", content: "sys" }, + { role: "user", content: "usr" }, + ]); + expect(body.model).toBe("m1"); + expect(body.response_format).toEqual({ type: "json_object" }); + expect((fetchImpl.calls[0].init.headers as Record<string, string>).Authorization).toBe("Bearer k"); + }); + + it("omits response_format when json mode is unsupported", async () => { + const f = scriptedFetch([chatJson("plain text")]); + const p = createOpenAiCompatProvider({ ...base, fetchImpl: f, jsonMode: false }); + await p.complete({ system: "s", user: "u", model: "m", json: true }); + expect(JSON.parse(String(f.calls[0].init.body)).response_format).toBeUndefined(); + }); + + it("throws no_key before any request when the key is missing", async () => { + const fetchImpl = ok(); + const p = createOpenAiCompatProvider({ ...base, apiKey: () => undefined, fetchImpl }); + await expect(p.complete({ system: "s", user: "u", model: "m", json: true })).rejects.toMatchObject({ + code: "no_key", + }); + expect(fetchImpl.calls).toHaveLength(0); + }); + + it("retries once on 429 with backoff, then succeeds", async () => { + const sleeps: number[] = []; + const fetchImpl = scriptedFetch([429, chatJson("{\"ok\":true}")]); + const p = createOpenAiCompatProvider({ ...base, fetchImpl, sleep: async (ms) => { sleeps.push(ms); } }); + const res = await p.complete({ system: "s", user: "u", model: "m", json: true }); + expect(res.text).toBe("{\"ok\":true}"); + expect(fetchImpl.calls).toHaveLength(2); + expect(sleeps).toEqual([2000]); + }); + + it("retries once on 5xx and surfaces server_error when the retry fails too", async () => { + const fetchImpl = scriptedFetch([500, 503]); + const p = createOpenAiCompatProvider({ ...base, fetchImpl, sleep: noSleep }); + await expect(p.complete({ system: "s", user: "u", model: "m", json: true })).rejects.toMatchObject({ + code: "server_error", + status: 503, + }); + expect(fetchImpl.calls).toHaveLength(2); + }); + + it("does not retry 4xx (bad_request) and never on 401/403", async () => { + const fetchImpl = scriptedFetch([401]); + const p = createOpenAiCompatProvider({ ...base, fetchImpl, sleep: noSleep }); + await expect(p.complete({ system: "s", user: "u", model: "m", json: true })).rejects.toMatchObject({ + code: "bad_request", + }); + expect(fetchImpl.calls).toHaveLength(1); + }); + + it("aborts with timeout when the endpoint hangs past the deadline", async () => { + // fetch that never resolves until aborted + const slow = ((url: string | URL | Request, init?: RequestInit) => + new Promise<Response>((_resolve, reject) => { + init?.signal?.addEventListener("abort", () => + reject(new DOMException("aborted", "AbortError")), + ); + })) as typeof fetch; + const p = createOpenAiCompatProvider({ ...base, fetchImpl: slow, timeoutMs: 20, sleep: noSleep }); + await expect(p.complete({ system: "s", user: "u", model: "m", json: true })).rejects.toMatchObject({ + code: "timeout", + }); + }); + + it("reports empty when the response has no content", async () => { + const fetchImpl = scriptedFetch([{ choices: [{ message: {} }] }]); + const p = createOpenAiCompatProvider({ ...base, fetchImpl, sleep: noSleep }); + await expect(p.complete({ system: "s", user: "u", model: "m", json: true })).rejects.toMatchObject({ + code: "empty", + }); + }); +}); + +describe("Anthropic Messages adapter", () => { + it("posts the native Messages shape with system on top and maps usage", async () => { + const fetchImpl = scriptedFetch([ + { + content: [{ type: "text", text: "{\"ok\":true}" }], + usage: { input_tokens: 12, output_tokens: 5 }, + }, + ]); + const p = createAnthropicProvider({ apiKey: () => "k", fetchImpl, sleep: noSleep }); + const res = await p.complete({ system: "sys", user: "usr", model: "claude-x", json: true }); + expect(res.text).toBe("{\"ok\":true}"); + expect(res.usage).toEqual({ in: 12, out: 5 }); + expect(fetchImpl.calls[0].url).toBe("https://api.anthropic.com/v1/messages"); + const headers = fetchImpl.calls[0].init.headers as Record<string, string>; + expect(headers["x-api-key"]).toBe("k"); + expect(headers["anthropic-version"]).toBeTruthy(); + const body = JSON.parse(String(fetchImpl.calls[0].init.body)); + expect(body.system).toBe("sys"); + expect(body.messages).toEqual([{ role: "user", content: "usr" }]); + expect(body.model).toBe("claude-x"); + expect(body.response_format).toBeUndefined(); + }); + + it("retries 429/5xx once like the OpenAI-compatible adapter", async () => { + const fetchImpl = scriptedFetch([429, { content: [{ type: "text", text: "hi" }], usage: { input_tokens: 1, output_tokens: 1 } }]); + const p = createAnthropicProvider({ apiKey: () => "k", fetchImpl, sleep: noSleep }); + const res = await p.complete({ system: "s", user: "u", model: "m", json: false }); + expect(res.text).toBe("hi"); + expect(fetchImpl.calls).toHaveLength(2); + }); +}); + +describe("GLM multi-endpoint retry + working-endpoint cache (port of engine/glm.js)", () => { + it("tries the next endpoint flavor when one fails and caches the winner", async () => { + // coding endpoint 401s for both models (no plan resource, error 1113), + // plain z.ai paas endpoint works + const fetchImpl = scriptedFetch([401, 401, chatJson("{\"ok\":true}")]); + const p = createGlmProvider({ + presetId: "glm-coding", + model: "glm-4.6", + apiKey: () => "k", + fetchImpl, + sleep: noSleep, + }); + const res = await p.complete({ system: "s", user: "u", model: "glm-4.6", json: true }); + expect(res.text).toBe("{\"ok\":true}"); + expect(fetchImpl.calls.length).toBeGreaterThanOrEqual(2); + expect(fetchImpl.calls[0].url).toContain("/api/coding/paas/v4"); + const urls = fetchImpl.calls.map((c) => c.url); + expect(urls.some((u) => u.includes("/api/paas/v4")) || urls.some((u) => u.includes("bigmodel"))).toBe(true); + }); + + it("second call goes to the cached working endpoint first", async () => { + const fetchImpl = scriptedFetch([401, chatJson("{\"ok\":true}"), chatJson("{\"ok\":2}")]); + const p = createGlmProvider({ + presetId: "glm-coding", + model: "glm-4.6", + apiKey: () => "k", + fetchImpl, + sleep: noSleep, + }); + await p.complete({ system: "s", user: "u", model: "glm-4.6", json: true }); + await p.complete({ system: "s", user: "u", model: "glm-4.6", json: true }); + // first call: [0]=coding fail, [1]=winner; second call must start at the winner + expect(fetchImpl.calls[2].url).toBe(fetchImpl.calls[1].url); + }); + + it("glm-open endpoints stay on bigmodel.cn", async () => { + const fetchImpl = scriptedFetch([chatJson("{\"ok\":true}")]); + const p = createGlmProvider({ + presetId: "glm-open", + model: "glm-4-flash", + apiKey: () => "k", + fetchImpl, + sleep: noSleep, + }); + await p.complete({ system: "s", user: "u", model: "glm-4-flash", json: true }); + for (const c of fetchImpl.calls) expect(c.url).toContain("open.bigmodel.cn"); + }); + + it("throws all_endpoints_failed when every flavor fails", async () => { + const fetchImpl = scriptedFetch([401, 401, 401, 401]); + const p = createGlmProvider({ + presetId: "glm-coding", + model: "glm-4.6", + apiKey: () => "k", + fetchImpl, + sleep: noSleep, + }); + await expect(p.complete({ system: "s", user: "u", model: "glm-4.6", json: true })).rejects.toBeInstanceOf( + ProviderError, + ); + }); +}); diff --git a/web/tests/unit/ai.admin.test.ts b/web/tests/unit/ai.admin.test.ts new file mode 100644 index 0000000..e6e80c4 --- /dev/null +++ b/web/tests/unit/ai.admin.test.ts @@ -0,0 +1,243 @@ +// §7.5 admin AI routes at the documented interface: owner gate (OWNER_EMAILS + +// Clerk user), GET/PUT /api/admin/ai, POST /api/admin/ai/test, and U18 — a +// saved selection changes the next generation's provider/model. +import { afterEach, describe, expect, it } from "vitest"; +import { requireOwner, type ClerkUserLike } from "@/lib/ai/owner"; +import { adminAiGet, adminAiPut, adminAiTest } from "@/lib/ai/admin"; +import { generateArtifact } from "@/lib/ai/generate"; +import { DEFAULT_VOICE_PROFILE } from "@/lib/ai/voice"; +import { DEFAULT_AI_SELECTION } from "@/lib/ai/settings"; +import { chatJson, demoSegments, fakeProvider, scriptedFetch, noSleep } from "./helpers/ai"; + +afterEach(() => { + delete (globalThis as Record<string, unknown>).__episodekitGlmWorking; +}); + +const envOf = (e: Record<string, string>) => e as NodeJS.ProcessEnv; + +type Body = Record<string, unknown>; +function bodyOf(res: { body: unknown }): Body { + return res.body as Body; +} + +function memoryStore(initial: unknown = null) { + const calls: Array<{ key: string; value: unknown }> = []; + let v = initial; + return { + calls, + get: async (key: string) => (key === "ai" ? v : null), + put: async (key: string, value: unknown) => { + calls.push({ key, value }); + if (key === "ai") v = value; + }, + }; +} + +function owner(gate: Parameters<typeof requireOwner>[0]) { + return () => requireOwner(gate); +} + +const ownerOk = owner({ + ownerEmails: ["owner@show.fm"], + currentUser: async () => + ({ id: "u_1", primaryEmailAddress: "owner@show.fm" }) as ClerkUserLike, +}); + +describe("requireOwner (owner-only boundary, M13)", () => { + it("accepts a Clerk user whose primary email is in OWNER_EMAILS", async () => { + const r = await requireOwner({ + ownerEmails: ["owner@show.fm"], + currentUser: async () => ({ id: "u", primaryEmailAddress: "Owner@Show.fm" }) as ClerkUserLike, + }); + expect(r.ok).toBe(true); + }); + + it("401 for anonymous, 403 for non-owner", async () => { + const anon = await requireOwner({ ownerEmails: ["o@x"], currentUser: async () => null }); + expect(anon.ok).toBe(false); + expect(anon.status).toBe(401); + const stranger = await requireOwner({ + ownerEmails: ["o@x"], + currentUser: async () => ({ id: "u", primaryEmailAddress: "other@x" }) as ClerkUserLike, + }); + expect(stranger.ok).toBe(false); + expect(stranger.status).toBe(403); + }); + + it("denies by default when OWNER_EMAILS is not configured (tie-break logged)", async () => { + const r = await requireOwner({ + ownerEmails: [], + currentUser: async () => ({ id: "u", primaryEmailAddress: "o@x" }) as ClerkUserLike, + }); + expect(r.status).toBe(503); + expect(r.reason).toBe("owner_not_configured"); + }); + + it("denies when the auth layer is unavailable (no Clerk keys)", async () => { + const r = await requireOwner({ + ownerEmails: ["o@x"], + currentUser: async () => { throw new Error("Missing publishableKey"); }, + }); + expect(r.status).toBe(503); + expect(r.reason).toBe("auth_unavailable"); + }); +}); + +describe("GET /api/admin/ai", () => { + it("returns the effective selection, its source, and per-provider key presence", async () => { + const res = await adminAiGet({ + requireOwner: ownerOk, + store: null, + env: envOf({ GLM_API_KEY: "k" }), + }); + expect(res.status).toBe(200); + expect(bodyOf(res).source).toBe("default"); + expect(bodyOf(res).selection).toEqual(DEFAULT_AI_SELECTION); + expect(bodyOf(res).storeAvailable).toBe(false); + const providers = bodyOf(res).providers as Array<{ id: string; keyPresent: boolean }>; + expect(providers.find((p) => p.id === "glm-coding")?.keyPresent).toBe(true); + expect(providers.find((p) => p.id === "groq")?.keyPresent).toBe(false); + }); + + it("reads a configured store", async () => { + const store = memoryStore({ + provider: "openai", + longModel: "gpt-4o", + fastModel: "gpt-4o-mini", + fallbackProvider: "groq", + }); + const res = await adminAiGet({ requireOwner: ownerOk, store, env: envOf({}) }); + expect(bodyOf(res).source).toBe("settings"); + const selection = bodyOf(res).selection as { provider: string }; + expect(selection.provider).toBe("openai"); + }); + + it("the gate blocks before anything else", async () => { + const res = await adminAiGet({ + requireOwner: owner({ ownerEmails: ["o@x"], currentUser: async () => null }), + store: null, + env: envOf({}), + }); + expect(res.status).toBe(401); + }); +}); + +describe("PUT /api/admin/ai (U18)", () => { + const body = { + provider: "groq", + longModel: "openai/gpt-oss-120b", + fastModel: "openai/gpt-oss-20b", + fallbackProvider: "glm-coding", + }; + + it("persists a valid selection at the app_settings interface", async () => { + const store = memoryStore(); + const res = await adminAiPut({ requireOwner: ownerOk, store, env: envOf({}) }, body); + expect(res.status).toBe(200); + expect(bodyOf(res).selection).toEqual(body); + expect(store.calls).toEqual([{ key: "ai", value: body }]); + }); + + it("400 on invalid selections (unknown provider, empty model, bad fallback)", async () => { + const store = memoryStore(); + for (const bad of [ + { ...body, provider: "skynet" }, + { ...body, longModel: "" }, + { ...body, fallbackProvider: "nope" }, + {}, + null, + ]) { + const res = await adminAiPut({ requireOwner: ownerOk, store, env: envOf({}) }, bad); + expect(res.status).toBe(400); + } + expect(store.calls).toHaveLength(0); + }); + + it("400 when the custom provider has no base URL configured", async () => { + const store = memoryStore(); + const res = await adminAiPut( + { requireOwner: ownerOk, store, env: envOf({}) }, + { provider: "custom", longModel: "m", fastModel: "m", fallbackProvider: "groq" }, + ); + expect(res.status).toBe(400); + }); + + it("503 when no settings store is wired yet (documented WP-02 seam)", async () => { + const res = await adminAiPut({ requireOwner: ownerOk, store: null, env: envOf({}) }, body); + expect(res.status).toBe(503); + }); + + it("the next generation after PUT uses the saved provider and model", async () => { + const store = memoryStore(); + await adminAiPut({ requireOwner: ownerOk, store, env: envOf({}) }, body); + const fetchImpl = scriptedFetch([ + chatJson( + JSON.stringify({ + oneLiner: "Focus is a trainable skill.", + short: "The episode examines attention in ordinary weeks.", + long: "The episode examines attention in ordinary weeks and offers one reset ritual.", + }), + ), + ]); + // generateArtifact with the same store → chain resolves to groq + saved model + const res = await generateArtifact({ + kind: "summary", + segments: demoSegments, + show: { name: "S", defaultCta: "c", allowedUrls: [] }, + voice: DEFAULT_VOICE_PROFILE, + deps: { + store, + env: envOf({ GROQ_API_KEY: "g" }), + fetchImpl, + sleep: noSleep, + }, + }); + expect(res.status).toBe("ai"); + expect(res.provider).toBe("groq"); + const reqBody = JSON.parse(String(fetchImpl.calls[0].init.body)); + expect(reqBody.model).toBe("openai/gpt-oss-120b"); + expect(fetchImpl.calls[0].url).toBe("https://api.groq.com/openai/v1/chat/completions"); + }); +}); + +describe("POST /api/admin/ai/test", () => { + it("returns ok with latency and usage for a working provider", async () => { + const chain = [fakeProvider("groq", [JSON.stringify({ ok: true })], { in: 10, out: 3 })]; + const res = await adminAiTest( + { requireOwner: ownerOk, env: envOf({}), chainFor: () => chain }, + {}, + ); + expect(res.status).toBe(200); + expect(bodyOf(res).ok).toBe(true); + expect(bodyOf(res).provider).toBe("groq"); + expect(bodyOf(res).latencyMs).toBeGreaterThanOrEqual(0); + expect(bodyOf(res).usage).toEqual({ in: 10, out: 3 }); + }); + + it("returns ok:false with the error code when the provider fails", async () => { + const chain = [fakeProvider("groq", [new Error("1113 no coding-plan resource")])]; + const res = await adminAiTest( + { + requireOwner: ownerOk, + env: envOf({}), + chainFor: (selection) => { + chain[0] = fakeProvider(selection.provider, [new Error("1113 no coding-plan resource")]); + return chain; + }, + }, + { provider: "glm-coding" }, + ); + expect(res.status).toBe(200); + expect(bodyOf(res).ok).toBe(false); + expect(bodyOf(res).error).toBeTruthy(); + expect(bodyOf(res).provider).toBe("glm-coding"); + }); + + it("400 on an unknown provider in the body", async () => { + const res = await adminAiTest( + { requireOwner: ownerOk, env: envOf({}), chainFor: () => [] }, + { provider: "skynet" }, + ); + expect(res.status).toBe(400); + }); +}); diff --git a/web/tests/unit/ai.fallback.test.ts b/web/tests/unit/ai.fallback.test.ts new file mode 100644 index 0000000..88db8f4 --- /dev/null +++ b/web/tests/unit/ai.fallback.test.ts @@ -0,0 +1,141 @@ +// Deterministic fallback builder: engine output mapped to §7.2 artifact kinds, +// honestly derived from the transcript only. +import { describe, expect, it } from "vitest"; +import { buildDeterministicArtifact } from "@/lib/ai/fallback"; +import { quoteText } from "@/lib/quotes"; +import { demoSegments, makeSegments } from "./helpers/ai"; +import { normalizeWs } from "@/lib/engine"; +import type { ArtifactKind } from "@/lib/artifacts/schemas"; + +const show = { name: "The Focus Show", defaultCta: "Follow the show.", allowedUrls: [] }; +const fullText = demoSegments.map((s) => s.text).join(" "); + +type Obj = Record<string, unknown>; +function build(kind: ArtifactKind, ctx?: { meta?: { guests: string[] } }): Obj { + return buildDeterministicArtifact(kind, { + segments: demoSegments, + show, + ...(ctx?.meta ? { meta: ctx.meta } : {}), + }) as Obj; +} + +describe("buildDeterministicArtifact", () => { + it("titles are ≤80 chars with valid styles", () => { + const items = build("titles").items as Array<{ text: string; style: string; starred: boolean }>; + expect(items.length).toBeGreaterThanOrEqual(4); + for (const t of items) { + expect(t.text.length).toBeLessThanOrEqual(80); + expect(["Curiosity", "Benefit", "Guest", "Number", "Question", "Quote"]).toContain(t.style); + expect(t.starred).toBe(false); + } + }); + + it("summary respects the one-liner limit and is extractive", () => { + const c = build("summary") as { oneLiner: string; short: string; long: string }; + expect(c.oneLiner.length).toBeLessThanOrEqual(140); + expect(normalizeWs(c.long)).toBeTruthy(); + for (const field of [c.short, c.long]) { + // every fallback summary sentence exists in the transcript + for (const s of field.split(/(?<=[.!?])\s+/)) { + const cleaned = s.replace(/^[“"']+|[“"']+$/g, ""); + if (cleaned.length > 20) expect(fullText).toContain(cleaned); + } + } + }); + + it("show notes bullets are transcript sentences and CTA comes from settings", () => { + const c = build("show_notes") as { bullets: string[]; cta: string; quoteRefs: unknown[] }; + expect(c.bullets.length).toBeGreaterThanOrEqual(4); + for (const b of c.bullets) expect(fullText).toContain(b); + expect(c.cta).toBe(show.defaultCta); + expect(c.quoteRefs).toEqual([]); + }); + + it("guest line only appears when guests are provided", () => { + const noGuests = build("show_notes") as { guestLine?: string }; + expect(noGuests.guestLine).toBeUndefined(); + const withGuests = build("show_notes", { meta: { guests: ["Maya", "Dev"] } }) as { + guestLine?: string; + }; + expect(withGuests.guestLine).toContain("Maya"); + expect(withGuests.guestLine).toContain("Dev"); + }); + + it("chapters use real timestamps, first at 0, and are honest", () => { + const items = build("chapters").items as Array<{ startMs: number; title: string }>; + expect(items.length).toBeGreaterThanOrEqual(1); + expect(items[0].startMs).toBe(0); + for (const item of items) { + expect(item.title.length).toBeLessThanOrEqual(50); + expect(item.startMs).toBeGreaterThanOrEqual(0); + } + }); + + it("blog keeps SEO limits and references no invented quotes", () => { + const c = build("blog") as { + seoTitle: string; + metaDescription: string; + slug: string; + bodyMarkdown: string; + quoteRefs: unknown[]; + }; + expect(c.seoTitle.length).toBeLessThanOrEqual(60); + expect(c.metaDescription.length).toBeLessThanOrEqual(155); + expect(c.slug).toMatch(/^[a-z0-9-]+$/); + expect(c.bodyMarkdown).toContain("## "); + expect(c.quoteRefs).toEqual([]); + }); + + it("newsletter and social shapes are complete", () => { + const n = build("newsletter") as { + subjects: string[]; + chosen: number; + previewText: string; + ctaText: string; + }; + expect(n.subjects).toHaveLength(3); + expect(n.chosen).toBe(0); + expect(n.previewText.length).toBeLessThanOrEqual(90); + expect(n.ctaText.length).toBeLessThanOrEqual(30); + + const s = build("social") as { + linkedin: string; + thread: string[]; + instagram: { caption: string; hashtags: string[] }; + }; + expect(s.linkedin.length).toBeLessThanOrEqual(1300); + for (const t of s.thread) expect(t.length).toBeLessThanOrEqual(280); + expect(s.instagram.caption.length).toBeLessThanOrEqual(2200); + expect(s.instagram.hashtags.length).toBeGreaterThanOrEqual(5); + for (const h of s.instagram.hashtags) expect(h).toMatch(/^#[a-z0-9]+$/i); + }); + + it("quotes are verbatim segment text within limits", () => { + const items = build("quotes").items as Array<{ segmentId: string; start: number; end: number }>; + expect(items.length).toBeGreaterThan(0); + for (const ref of items) { + const text = quoteText(ref, demoSegments); + const seg = demoSegments.find((s) => s.id === ref.segmentId)!; + expect(seg.text).toContain(text); + expect(text.length).toBeGreaterThanOrEqual(40); + expect(text.length).toBeLessThanOrEqual(240); + } + }); + + it("a tiny transcript still produces safe fallback content (no crash)", () => { + const tiny = makeSegments("This is one short episode sentence about focus."); + for (const kind of [ + "show_notes", + "chapters", + "summary", + "titles", + "blog", + "newsletter", + "social", + "quotes", + ] as const) { + const c = buildDeterministicArtifact(kind, { segments: tiny, show }); + expect(c).toBeTruthy(); + } + }); +}); diff --git a/web/tests/unit/ai.generate.test.ts b/web/tests/unit/ai.generate.test.ts new file mode 100644 index 0000000..7078b2a --- /dev/null +++ b/web/tests/unit/ai.generate.test.ts @@ -0,0 +1,298 @@ +// §8.1 call policy end-to-end at the library boundary: provider chain with +// retries, JSON repair, §7.3 quote resolution, §8.4 post-check wiring (U8), +// deterministic fallback labelled as such (U10), usage_events logging. +import { describe, expect, it, vi } from "vitest"; +import { generateArtifact } from "@/lib/ai/generate"; +import { DEFAULT_VOICE_PROFILE } from "@/lib/ai/voice"; +import { quoteText } from "@/lib/quotes"; +import { failingProvider, fakeProvider, demoSegments, TOKENS } from "./helpers/ai"; +import type { UsageEvent } from "@/lib/ai/usage"; +import type { GenerateDeps } from "@/lib/ai/generate"; +import type { LlmProvider } from "@/lib/ai/types"; + +const show = { + name: "The Focus Show", + defaultCta: "Follow the show wherever you listen.", + allowedUrls: ["https://thefocus.show"], +}; + +const SHOW_NOTES = { + hook: "Focus is a skill you can train.", + bullets: [ + "Why attention feels scarce", + "The two-minute reset ritual", + "How to say no at work", + "Deep work in ordinary weeks", + ], + quoteRefs: [{ segmentId: "seg_000000", exact: demoSegments[0].text.slice(0, 60) }], + cta: "Follow the show wherever you listen.", +}; + +function depsWithChain(chain: LlmProvider[], extra: Partial<GenerateDeps> = {}): GenerateDeps { + return { chain, workspaceId: "ws_1", episodeId: "ep_1", ...extra }; +} + +/** GenerateResult.content is unknown by contract; narrow at the call site. */ +function contentOf(res: { content: unknown }): Record<string, unknown> { + return res.content as Record<string, unknown>; +} + +describe("happy path", () => { + it("returns validated AI content with resolved quote offsets", async () => { + const p = fakeProvider("groq", [JSON.stringify(SHOW_NOTES)], TOKENS); + const res = await generateArtifact({ + kind: "show_notes", + segments: demoSegments, + show, + voice: DEFAULT_VOICE_PROFILE, + deps: depsWithChain([p]), + }); + expect(res.status).toBe("ai"); + expect(res.provider).toBe("groq"); + expect(res.errorCode).toBeNull(); + const ref = (contentOf(res).quoteRefs as { segmentId: string; start: number; end: number }[])[0]; + expect(ref.segmentId).toBe("seg_000000"); + // resolution trims edge whitespace before storing offsets + expect(demoSegments[0].text.slice(ref.start, ref.end)).toBe( + demoSegments[0].text.slice(0, 60).trim(), + ); + }); + + it("uses the long model for artifact kinds at temperature 0.4 with json mode", async () => { + const p = fakeProvider("groq", [JSON.stringify(SHOW_NOTES)]); + await generateArtifact({ + kind: "show_notes", + segments: demoSegments, + show, + voice: DEFAULT_VOICE_PROFILE, + deps: depsWithChain([p]), + }); + expect(p.calls[0].model).toBe("glm-4.6"); // default selection long model + expect(p.calls[0].json).toBe(true); + expect(p.calls[0].temperature).toBe(0.4); + }); +}); + +describe("JSON handling: response_format + Zod + one repair call", () => { + it("repairs unparseable output with one follow-up call that asks for valid JSON", async () => { + const p = fakeProvider("groq", ["not json at all", JSON.stringify(SHOW_NOTES)]); + const res = await generateArtifact({ + kind: "show_notes", + segments: demoSegments, + show, + voice: DEFAULT_VOICE_PROFILE, + deps: depsWithChain([p]), + }); + expect(res.status).toBe("ai"); + expect(p.calls).toHaveLength(2); + expect(p.calls[1].user).toContain("valid JSON"); + expect(p.calls[1].user).toContain(p.calls[0].user); // original prompt preserved + }); + + it("moves to the fallback provider when the repair also fails validation", async () => { + const bad = fakeProvider("glm-coding", ["garbage", "{\"bullets\": []}"]); + const good = fakeProvider("groq", [JSON.stringify(SHOW_NOTES)]); + const res = await generateArtifact({ + kind: "show_notes", + segments: demoSegments, + show, + voice: DEFAULT_VOICE_PROFILE, + deps: depsWithChain([bad, good]), + }); + expect(res.status).toBe("ai"); + expect(res.provider).toBe("groq"); + expect(bad.calls).toHaveLength(2); // initial + repair, then gave up + }); +}); + +describe("U10: every provider fails → deterministic package labelled fallback", () => { + it("falls back to the deterministic builder with status fallback", async () => { + const chain = [ + failingProvider("glm-coding", "1113 no coding-plan resource"), + failingProvider("groq", "rate limited"), + ]; + const res = await generateArtifact({ + kind: "show_notes", + segments: demoSegments, + show, + voice: DEFAULT_VOICE_PROFILE, + deps: depsWithChain(chain), + }); + expect(res.status).toBe("fallback"); + expect(res.provider).toBeNull(); + expect(res.model).toBeNull(); + expect(res.errorCode).toBe("all_providers_failed"); + // content is derived from the transcript, not invented + const c = contentOf(res); + for (const b of c.bullets as string[]) { + expect(demoSegments.map((s) => s.text).join(" ")).toContain(b); + } + expect(c.cta).toBe(show.defaultCta); + }); + + it("fallback quotes artifact resolves verbatim refs from candidates", async () => { + const res = await generateArtifact({ + kind: "quotes", + segments: demoSegments, + show, + voice: DEFAULT_VOICE_PROFILE, + deps: depsWithChain([failingProvider("glm-coding")]), + }); + expect(res.status).toBe("fallback"); + const items = contentOf(res).items as { segmentId: string; start: number; end: number }[]; + expect(Array.isArray(items)).toBe(true); + for (const ref of items) { + const seg = demoSegments.find((s) => s.id === ref.segmentId); + expect(seg).toBeTruthy(); + expect(seg!.text).toContain(quoteText(ref, demoSegments)); + } + }); + + it("chapters without timestamps never fabricate times", async () => { + const noTime = demoSegments.map((s) => ({ ...s, startMs: null, endMs: null })); + const res = await generateArtifact({ + kind: "chapters", + segments: noTime, + show, + voice: DEFAULT_VOICE_PROFILE, + deps: depsWithChain([failingProvider("glm-coding")]), + }); + expect(res.status).toBe("fallback"); + expect(contentOf(res).items).toEqual([]); + expect(res.errorCode).toBe("no_timestamps"); + }); +}); + +describe("U8: banned words (voice neverSay) never survive into an artifact", () => { + const banned = (text: string) => JSON.stringify({ ...SHOW_NOTES, hook: text }); + + it("regenerates once with feedback when the draft uses a banned word", async () => { + const p = fakeProvider("groq", [ + banned(`Let us delve into focus.`), + JSON.stringify(SHOW_NOTES), + ]); + const res = await generateArtifact({ + kind: "show_notes", + segments: demoSegments, + show, + voice: DEFAULT_VOICE_PROFILE, // neverSay: delve, game-changer, ... + deps: depsWithChain([p]), + }); + expect(p.calls).toHaveLength(2); + expect(p.calls[1].user).toContain("delve"); + expect(p.calls[1].user.toLowerCase()).toContain("banned"); + expect(res.status).toBe("ai"); + expect(res.regenerated).toBe(true); + }); + + it("replaces the phrase deterministically when the retry still violates", async () => { + const p = fakeProvider("groq", [ + banned(`We delve into attention.`), + banned(`We delve into attention some more.`), + ]); + const res = await generateArtifact({ + kind: "show_notes", + segments: demoSegments, + show, + voice: DEFAULT_VOICE_PROFILE, + deps: depsWithChain([p]), + }); + expect(res.status).toBe("ai"); + expect(res.regenerated).toBe(true); + expect(JSON.stringify(res.content)).not.toMatch(/delve/i); + expect(contentOf(res).hook).toBe("We into attention some more."); + }); + + it("a violation that cannot be phrase-replaced pushes the artifact to the next provider", async () => { + const bad = fakeProvider("glm-coding", [ + JSON.stringify({ ...SHOW_NOTES, bullets: ["sermon notes for sunday", "focus", "energy", "habits"] }), + JSON.stringify({ ...SHOW_NOTES, bullets: ["still a sermon reference", "focus", "energy", "habits"] }), + ]); + const good = fakeProvider("groq", [JSON.stringify(SHOW_NOTES)]); + const res = await generateArtifact({ + kind: "show_notes", + segments: demoSegments, + show, + voice: DEFAULT_VOICE_PROFILE, + deps: depsWithChain([bad, good]), + }); + expect(res.provider).toBe("groq"); + expect(JSON.stringify(res.content)).not.toMatch(/sermon/i); + }); +}); + +describe("§8.4 foreign URL strip", () => { + it("removes links not in show settings, keeps allowed ones", async () => { + const withLinks = JSON.stringify({ + ...SHOW_NOTES, + bullets: [ + "Read the study at https://made-up.example/paper", + "Full notes at https://thefocus.show/ep-12", + "Focus basics", + "Saying no", + ], + }); + const p = fakeProvider("groq", [withLinks]); + const res = await generateArtifact({ + kind: "show_notes", + segments: demoSegments, + show, + voice: DEFAULT_VOICE_PROFILE, + deps: depsWithChain([p]), + }); + const joined = (contentOf(res).bullets as string[]).join(" "); + expect(joined).not.toContain("made-up.example"); + expect(joined).toContain("https://thefocus.show/ep-12"); + }); +}); + +describe("usage_events logging (§7.1)", () => { + it("logs one event per successful call with est. cost from the price table", async () => { + const events: UsageEvent[] = []; + const usage = vi.fn(async (e: UsageEvent) => { events.push(e); }); + const p = fakeProvider("groq", [ + "broken json", + JSON.stringify(SHOW_NOTES), + ], TOKENS); + await generateArtifact({ + kind: "show_notes", + segments: demoSegments, + show, + voice: DEFAULT_VOICE_PROFILE, + deps: depsWithChain([p], { + usage, + selection: { + provider: "groq", + longModel: "openai/gpt-oss-120b", + fastModel: "openai/gpt-oss-20b", + fallbackProvider: "groq", + }, + }), + }); + expect(events).toHaveLength(2); + expect(events[0]).toMatchObject({ + workspaceId: "ws_1", + episodeId: "ep_1", + kind: "llm", + provider: "groq", + model: "openai/gpt-oss-120b", + }); + expect(events[0].units).toBe(TOKENS.in + TOKENS.out); + // verified Groq list price: $0.15/M in, $0.60/M out + expect(events[0].costUsd).toBeCloseTo((100 * 0.15 + 50 * 0.6) / 1_000_000, 12); + expect(events[1].costUsd).toBeGreaterThan(0); + }); + + it("a failing usage logger never fails generation", async () => { + const usage = vi.fn(async () => { throw new Error("db down"); }); + const p = fakeProvider("groq", [JSON.stringify(SHOW_NOTES)]); + const res = await generateArtifact({ + kind: "show_notes", + segments: demoSegments, + show, + voice: DEFAULT_VOICE_PROFILE, + deps: depsWithChain([p], { usage }), + }); + expect(res.status).toBe("ai"); + }); +}); diff --git a/web/tests/unit/ai.postcheck-wiring.test.ts b/web/tests/unit/ai.postcheck-wiring.test.ts new file mode 100644 index 0000000..de997c4 --- /dev/null +++ b/web/tests/unit/ai.postcheck-wiring.test.ts @@ -0,0 +1,69 @@ +// §8.4 wiring primitives in lib/ai/postcheck.ts: URL allowance (with subpath +// rule) and banned-phrase replacement (U8 last resort). +import { describe, expect, it } from "vitest"; +import { + applyDeterministicPostChecks, + collectStrings, + replaceBannedPhrases, + scanContent, + stripForeignUrlsInContent, +} from "@/lib/ai/postcheck"; +import { DEFAULT_VOICE_PROFILE } from "@/lib/ai/voice"; + +describe("stripForeignUrlsInContent", () => { + it("keeps allowed URLs and their subpaths, strips foreign ones", () => { + const content = { + a: "Listen at https://show.example/ep-12 today.", + b: "See https://evil.example.net/track.", + c: "Base https://show.example works too.", + }; + const out = stripForeignUrlsInContent(content, ["https://show.example"]); + expect(out.a).toBe("Listen at https://show.example/ep-12 today."); + expect(out.b).toBe("See."); + expect(out.c).toBe("Base https://show.example works too."); + }); + + it("strips everything when the allow list is empty", () => { + const out = stripForeignUrlsInContent({ t: "Go to https://x.example." }, []); + expect(out.t).toBe("Go to."); + }); + + it("leaves text without URLs untouched (no whitespace surprises)", () => { + const text = "Keep this exactly — including double spaces.\n\nAnd breaks."; + expect(stripForeignUrlsInContent({ text }, ["https://show.example"]).text).toBe(text); + }); +}); + +describe("replaceBannedPhrases (U8 last resort)", () => { + it("removes banned terms at word boundaries and cleans up whitespace", () => { + const out = replaceBannedPhrases( + { hook: "We delve into game-changing ideas to unlock focus.", bullets: ["stay grounded"] }, + ["delve", "game-changer", "unlock"], + ); + expect(out.hook).toBe("We into game-changing ideas to focus."); + expect(out.bullets).toEqual(["stay grounded"]); + }); + + it("multi-word banned terms match across whitespace variance", () => { + const out = replaceBannedPhrases({ t: "time to dive deep now" }, ["dive deep"]); + expect(out.t).toBe("time to now"); + }); +}); + +describe("scanContent + applyDeterministicPostChecks", () => { + it("scans nested strings for neverSay and religion framing", () => { + const v = scanContent({ a: ["We delve into the sermon."], b: "clean" }, DEFAULT_VOICE_PROFILE); + expect(v.neverSay).toEqual(["delve"]); + expect(v.religion).toContain("sermon"); + }); + + it("applyDeterministicPostChecks strips URLs then banned phrases", () => { + const out = applyDeterministicPostChecks( + { t: "We unlock focus at https://evil.example now — delve in." }, + DEFAULT_VOICE_PROFILE, + [], + ); + expect(out.t).toBe("We focus at now — in."); + expect(collectStrings(out).join(" ")).not.toMatch(/delve|unlock|evil\.example/i); + }); +}); diff --git a/web/tests/unit/ai.registry.test.ts b/web/tests/unit/ai.registry.test.ts new file mode 100644 index 0000000..377fdec --- /dev/null +++ b/web/tests/unit/ai.registry.test.ts @@ -0,0 +1,157 @@ +// §8.1 provider registry: preset table, selection order (app_settings.ai → +// env → defaults), key presence, provider chain construction. +import { afterEach, describe, expect, it } from "vitest"; +import { DEFAULT_AI_SELECTION, effectiveAiSelection, selectionFromEnv } from "@/lib/ai/settings"; +import { PROVIDER_PRESETS, providerIds } from "@/lib/ai/presets"; +import { buildProvider, buildProviderChain } from "@/lib/ai/registry"; +import { keyPresent } from "@/lib/ai/settings"; +import { scriptedFetch, noSleep } from "./helpers/ai"; + +function envOf(entries: Record<string, string | undefined>): NodeJS.ProcessEnv { + return entries as NodeJS.ProcessEnv; +} + +afterEach(() => { + delete (globalThis as Record<string, unknown>).__episodekitGlmWorking; +}); + +describe("§8.1 preset table", () => { + it("has all seven presets with the spec'd base URLs and default models", () => { + expect(providerIds()).toEqual([ + "glm-coding", + "glm-open", + "groq", + "openai", + "openrouter", + "anthropic", + "custom", + ]); + expect(PROVIDER_PRESETS["glm-coding"].baseUrl).toBe("https://api.z.ai/api/coding/paas/v4"); + expect(PROVIDER_PRESETS["glm-coding"].defaultLong).toBe("glm-4.6"); + expect(PROVIDER_PRESETS["glm-coding"].defaultFast).toBe("glm-4.5-air"); + expect(PROVIDER_PRESETS["glm-coding"].keyEnv).toBe("GLM_API_KEY"); + expect(PROVIDER_PRESETS["glm-open"].baseUrl).toBe("https://open.bigmodel.cn/api/paas/v4"); + expect(PROVIDER_PRESETS["glm-open"].defaultFast).toBe("glm-4-flash"); + expect(PROVIDER_PRESETS.groq.baseUrl).toBe("https://api.groq.com/openai/v1"); + expect(PROVIDER_PRESETS.openai.baseUrl).toBe("https://api.openai.com/v1"); + expect(PROVIDER_PRESETS.openrouter.baseUrl).toBe("https://openrouter.ai/api/v1"); + expect(PROVIDER_PRESETS.anthropic.adapter).toBe("anthropic"); + expect(PROVIDER_PRESETS.anthropic.baseUrl).toContain("api.anthropic.com"); + expect(PROVIDER_PRESETS.custom.keyEnv).toBe("AI_CUSTOM_API_KEY"); + }); + + it("marks glm-coding as default and groq as fallback (spec §8.1)", () => { + expect(DEFAULT_AI_SELECTION.provider).toBe("glm-coding"); + expect(DEFAULT_AI_SELECTION.longModel).toBe("glm-4.6"); + expect(DEFAULT_AI_SELECTION.fastModel).toBe("glm-4.5-air"); + expect(DEFAULT_AI_SELECTION.fallbackProvider).toBe("groq"); + }); +}); + +describe("selection order: app_settings.ai → env → defaults", () => { + it("returns defaults when no store and no env", async () => { + const { selection, source } = await effectiveAiSelection(null, { + env: envOf({}), + }); + expect(selection).toEqual(DEFAULT_AI_SELECTION); + expect(source).toBe("default"); + }); + + it("reads a valid store object wholesale", async () => { + const store = { + get: async () => ({ + provider: "openai", + longModel: "gpt-4o", + fastModel: "gpt-4o-mini", + fallbackProvider: "anthropic", + }), + put: async () => {}, + }; + const { selection, source } = await effectiveAiSelection(store, { env: envOf({}) }); + expect(selection.provider).toBe("openai"); + expect(selection.fallbackProvider).toBe("anthropic"); + expect(source).toBe("settings"); + }); + + it("falls through to env when the store object is invalid", async () => { + const store = { get: async () => ({ provider: "nope" }), put: async () => {} }; + const { selection, source } = await effectiveAiSelection(store, { + env: envOf({ + AI_PROVIDER: "groq", + AI_MODEL_LONG: "openai/gpt-oss-120b", + AI_MODEL_FAST: "openai/gpt-oss-20b", + AI_FALLBACK_PROVIDER: "glm-coding", + }), + }); + expect(selection.provider).toBe("groq"); + expect(selection.longModel).toBe("openai/gpt-oss-120b"); + expect(selection.fallbackProvider).toBe("glm-coding"); + expect(source).toBe("env"); + }); + + it("falls through to defaults when the store throws", async () => { + const store = { get: async () => { throw new Error("db down"); }, put: async () => {} }; + const { selection, source } = await effectiveAiSelection(store, { env: envOf({}) }); + expect(selection).toEqual(DEFAULT_AI_SELECTION); + expect(source).toBe("default"); + }); + + it("env provider without explicit models uses the preset defaults", () => { + const s = selectionFromEnv(envOf({ AI_PROVIDER: "groq" })); + expect(s).not.toBeNull(); + expect(s!.provider).toBe("groq"); + expect(s!.longModel).toBe(PROVIDER_PRESETS.groq.defaultLong); + expect(s!.fastModel).toBe(PROVIDER_PRESETS.groq.defaultFast); + expect(s!.fallbackProvider).toBe(DEFAULT_AI_SELECTION.fallbackProvider); + }); + + it("ignores an invalid AI_PROVIDER env and returns null (caller uses defaults)", () => { + expect(selectionFromEnv(envOf({ AI_PROVIDER: "skynet" }))).toBeNull(); + expect(selectionFromEnv(envOf({}))).toBeNull(); + }); +}); + +describe("key presence (admin shows ✓/✗, keys never leave env)", () => { + it("reports per-provider key presence from env", () => { + const env = envOf({ GLM_API_KEY: "k", AI_CUSTOM_BASE_URL: "https://x.example" }); + expect(keyPresent(PROVIDER_PRESETS["glm-coding"], env)).toBe(true); + expect(keyPresent(PROVIDER_PRESETS.groq, env)).toBe(false); + // custom needs base URL + key + expect(keyPresent(PROVIDER_PRESETS.custom, env)).toBe(false); + expect( + keyPresent(PROVIDER_PRESETS.custom, envOf({ AI_CUSTOM_BASE_URL: "https://x", AI_CUSTOM_API_KEY: "k" })), + ).toBe(true); + }); +}); + +describe("provider chain construction", () => { + it("returns [primary, fallback] and skips the fallback when identical to primary", () => { + const sel = { ...DEFAULT_AI_SELECTION }; + expect(buildProviderChain(sel, { env: envOf({ GLM_API_KEY: "k", GROQ_API_KEY: "g" }) }).map((p) => p.id)) + .toEqual(["glm-coding", "groq"]); + const same = { provider: "groq", longModel: "m", fastModel: "m", fallbackProvider: "groq" } as const; + expect(buildProviderChain(same, { env: envOf({ GROQ_API_KEY: "g" }) }).map((p) => p.id)).toEqual(["groq"]); + }); + + it("throws a configuration error for custom without base URL (U10 still applies)", () => { + expect(() => buildProviderChain({ ...DEFAULT_AI_SELECTION, provider: "custom" }, { env: envOf({}) })).toThrow(); + }); + + it("passes the tier model to the underlying adapter request", async () => { + const fetchImpl = scriptedFetch([ + { choices: [{ message: { content: "{\"ok\":true}" } }], usage: { prompt_tokens: 1, completion_tokens: 1 } }, + ]); + const provider = buildProvider( + { provider: "groq", longModel: "long-model-x", fastModel: "fast-model-y", fallbackProvider: "groq" }, + { env: envOf({ GROQ_API_KEY: "g" }), fetchImpl, sleep: noSleep }, + ); + await provider.complete({ system: "s", user: "u", model: "long-model-x", json: true }); + await provider.complete({ system: "s", user: "u", model: "fast-model-y", json: true }); + expect(fetchImpl.calls[0].url).toBe("https://api.groq.com/openai/v1/chat/completions"); + const body0 = JSON.parse(String(fetchImpl.calls[0].init.body)); + expect(body0.model).toBe("long-model-x"); + const body1 = JSON.parse(String(fetchImpl.calls[1].init.body)); + expect(body1.model).toBe("fast-model-y"); + expect(body1.response_format).toEqual({ type: "json_object" }); + }); +}); diff --git a/web/tests/unit/ai.voice.test.ts b/web/tests/unit/ai.voice.test.ts new file mode 100644 index 0000000..449dfc8 --- /dev/null +++ b/web/tests/unit/ai.voice.test.ts @@ -0,0 +1,84 @@ +// §7.4 voice profile: shape, defaults, learn + preview functions. +import { describe, expect, it } from "vitest"; +import { DEFAULT_VOICE_PROFILE, VoiceProfileSchema, learnVoiceProfile, previewVoice, type VoiceProfile } from "@/lib/ai/voice"; +import { fakeProvider, failingProvider } from "./helpers/ai"; + +const PROFILE: VoiceProfile = { + tone: ["dry", "direct", "curious"], + formality: 2, + energy: 4, + humor: 3, + person: "I", + sentenceLength: "short", + emoji: "light", + signaturePhrases: ["here's the thing"], + neverSay: ["synergy"], + alwaysInclude: [], + ctaStyle: "Ask for one reply.", +}; + +describe("§7.4 defaults and schema", () => { + it("default profile matches the spec", () => { + expect(DEFAULT_VOICE_PROFILE).toEqual({ + tone: ["clear", "warm", "curious"], + formality: 3, + energy: 3, + humor: 2, + person: "we", + sentenceLength: "medium", + emoji: "none", + signaturePhrases: [], + neverSay: ["delve", "game-changer", "unlock", "dive deep", "in today's fast-paced world"], + alwaysInclude: [], + ctaStyle: "Invite listeners to follow the show.", + }); + expect(VoiceProfileSchema.safeParse(DEFAULT_VOICE_PROFILE).success).toBe(true); + }); + + it("rejects out-of-range sliders and overlong lists", () => { + expect(VoiceProfileSchema.safeParse({ ...PROFILE, formality: 9 }).success).toBe(false); + expect(VoiceProfileSchema.safeParse({ ...PROFILE, tone: ["a", "b"] }).success).toBe(false); + expect(VoiceProfileSchema.safeParse({ ...PROFILE, neverSay: Array(31).fill("x") }).success).toBe(false); + }); +}); + +describe("learnVoiceProfile", () => { + it("parses a valid model response into the profile", async () => { + const p = fakeProvider("groq", [JSON.stringify(PROFILE)]); + const res = await learnVoiceProfile({ samples: ["Past show notes text."], deps: { chain: [p] } }); + expect(res.ok).toBe(true); + if (res.ok) expect(res.profile).toEqual(PROFILE); + expect(p.calls[0].json).toBe(true); + }); + + it("repairs once, then falls back to the default profile on failure", async () => { + const repairable = fakeProvider("groq", ["nope", JSON.stringify(PROFILE)]); + const r1 = await learnVoiceProfile({ samples: ["s"], deps: { chain: [repairable] } }); + expect(r1.ok).toBe(true); + expect(repairable.calls).toHaveLength(2); + + const broken = fakeProvider("groq", ["garbage", "still garbage"]); + const r2 = await learnVoiceProfile({ samples: ["s"], deps: { chain: [broken] } }); + expect(r2.ok).toBe(false); + if (!r2.ok) { + expect(r2.profile).toEqual(DEFAULT_VOICE_PROFILE); + expect(r2.errorCode).toBeTruthy(); + } + }); + + it("a provider outage returns the default profile, not a crash", async () => { + const res = await learnVoiceProfile({ samples: ["s"], deps: { chain: [failingProvider("glm-coding")] } }); + expect(res.ok).toBe(false); + if (!res.ok) expect(res.profile).toEqual(DEFAULT_VOICE_PROFILE); + }); +}); + +describe("previewVoice", () => { + it("returns plain model text (not JSON)", async () => { + const p = fakeProvider("groq", ["Here is the thing: focus is a habit."]); + const res = await previewVoice({ voice: PROFILE, deps: { chain: [p] } }); + expect(res.ok).toBe(true); + if (res.ok) expect(res.text).toContain("focus is a habit"); + expect(p.calls[0].json).toBe(false); + }); +}); diff --git a/web/tests/unit/helpers/ai.ts b/web/tests/unit/helpers/ai.ts new file mode 100644 index 0000000..7da5418 --- /dev/null +++ b/web/tests/unit/helpers/ai.ts @@ -0,0 +1,93 @@ +// Shared fixtures for the lib/ai test suite: segments, fake providers and a +// scripted fetch for adapter tests. +import { vi } from "vitest"; +import { normalizeWs, sentences, stripSpeakerLabels } from "@/lib/engine"; +import type { LlmProvider, CompleteRequest, CompleteResult, TokenUsage } from "@/lib/ai/types"; +import type { Segment } from "@/lib/quotes/types"; +import { readFileSync } from "node:fs"; + +export const demoEpisode = readFileSync( + new URL("../../fixtures/demo-episode.txt", import.meta.url), + "utf8", +); + +export function makeSegments(text: string, perSegment = 2): Segment[] { + const sents = sentences(normalizeWs(stripSpeakerLabels(text))); + const segments: Segment[] = []; + for (let i = 0; i < sents.length; i += perSegment) { + const idx = segments.length; + segments.push({ + id: `seg_${String(idx).padStart(6, "0")}`, + idx, + speaker: idx % 2 === 0 ? "Maya" : null, + startMs: idx * 60_000, + endMs: (idx + 1) * 60_000, + text: sents.slice(i, i + perSegment).join(" "), + }); + } + return segments; +} + +export const demoSegments = makeSegments(normalizeWs(demoEpisode.split("\n---")[0])); + +/** A provider whose complete() is a vi.fn() returning canned results in order. */ +export function fakeProvider( + id: string, + results: Array<string | Error>, + usage?: TokenUsage, +): LlmProvider & { calls: CompleteRequest[] } { + const calls: CompleteRequest[] = []; + let i = 0; + return { + id, + calls, + complete: vi.fn(async (req: CompleteRequest): Promise<CompleteResult> => { + calls.push(req); + const r = results[Math.min(i, results.length - 1)]; + i++; + if (r instanceof Error) throw r; + return { text: r, usage }; + }), + }; +} + +/** A provider that always fails with the given error. */ +export function failingProvider(id: string, message = "boom"): LlmProvider { + return { id, complete: vi.fn(async () => { throw new Error(message); }) }; +} + +/** OpenAI-compatible JSON payload for a successful chat completion. */ +export function chatJson(text: string, usage?: TokenUsage) { + return { + choices: [{ message: { content: text } }], + ...(usage ? { usage: { prompt_tokens: usage.in, completion_tokens: usage.out } } : {}), + }; +} + +export type FetchCall = { url: string; init: RequestInit }; + +/** + * Scripted fetch: entries are checked in order; each entry is either a status + * number (response with that status, body "{\"error\":\"x\"}") or a parsed + * JSON body for a 200 response. + */ +export function scriptedFetch( + entries: Array<number | Record<string, unknown>>, +): typeof fetch & { calls: FetchCall[] } { + const calls: FetchCall[] = []; + let i = 0; + const fn = (async (url: string | URL | Request, init?: RequestInit) => { + calls.push({ url: String(url), init: init ?? {} }); + const e = entries[Math.min(i, entries.length - 1)]; + i++; + if (typeof e === "number") { + return new Response(JSON.stringify({ error: { message: "x" } }), { status: e }); + } + return new Response(JSON.stringify(e), { status: 200 }); + }) as typeof fetch; + return Object.assign(fn, { calls }) as typeof fetch & { calls: FetchCall[] }; +} + +export const noSleep = () => Promise.resolve(); + +export const TOKENS = { in: 100, out: 50 }; diff --git a/web/tests/unit/live.ai.test.ts b/web/tests/unit/live.ai.test.ts new file mode 100644 index 0000000..fb33457 --- /dev/null +++ b/web/tests/unit/live.ai.test.ts @@ -0,0 +1,61 @@ +// LIVE provider smoke test — NOT run in CI (skips without GLM_API_KEY; also +// skips when CI=true even if a key leaks into the environment). +// +// Run locally with real keys: +// set -a; source /path/to/.env.relaunch; set +a +// cd web && GLM_API_KEY=$GLM_API_KEY pnpm vitest run tests/unit/live.ai.test.ts +import { describe, expect, it } from "vitest"; +import { DEFAULT_AI_SELECTION } from "@/lib/ai/settings"; +import { buildProvider } from "@/lib/ai/registry"; +import { generateArtifact } from "@/lib/ai/generate"; +import { DEFAULT_VOICE_PROFILE } from "@/lib/ai/voice"; + +const live = Boolean(process.env.GLM_API_KEY) && process.env.CI !== "true"; +const t0 = () => Date.now(); + +describe.skipIf(!live)("live: GLM coding endpoint through the registry", () => { + it("answers a tiny JSON prompt", async () => { + const p = buildProvider(DEFAULT_AI_SELECTION, { + tier: "fast", + env: process.env as NodeJS.ProcessEnv, + }); + const start = t0(); + const res = await p.complete({ + system: 'You are a health check. Return only JSON.', + user: 'Return the JSON object {"ok":true}.', + model: DEFAULT_AI_SELECTION.fastModel, + json: true, + temperature: 0.2, + }); + const latencyMs = t0() - start; + const parsed = JSON.parse(res.text); + expect(parsed.ok).toBe(true); + console.log( + `[live] provider=glm-coding model=${DEFAULT_AI_SELECTION.fastModel} latencyMs=${latencyMs} usage=${JSON.stringify(res.usage ?? null)}`, + ); + }, 90_000); + + it("generates a real artifact via generateArtifact (fallback-labelled on provider outage)", async () => { + const text = `Host: Welcome back to the show. Today we talk about attention in ordinary weeks. +Guest: Thanks for having me. The biggest thing I learned is that focus is a trained skill, not a gift. +Host: Tell us about the two-minute reset you teach. +Guest: The idea is simple. You stop, breathe for two minutes, and write the one next action on paper. +Host: People love that. It works because it removes the decision, not the effort.`; + const { makeSegments } = await import("./helpers/ai"); + const segments = makeSegments(text, 1); + const start = t0(); + const res = await generateArtifact({ + kind: "summary", + segments, + show: { name: "Live Test Show", defaultCta: "Follow.", allowedUrls: [] }, + voice: DEFAULT_VOICE_PROFILE, + deps: { env: process.env as NodeJS.ProcessEnv, workspaceId: null, episodeId: null }, + }); + console.log( + `[live] generateArtifact status=${res.status} provider=${res.provider} model=${res.model} ` + + `errorCode=${res.errorCode} latencyMs=${t0() - start} costUsd=${res.costUsd} usage=${JSON.stringify(res.usage)}`, + ); + expect(["ai", "fallback"]).toContain(res.status); + expect(res.content).toBeTruthy(); + }, 120_000); +}); diff --git a/web/tests/unit/prompts.test.ts b/web/tests/unit/prompts.test.ts new file mode 100644 index 0000000..0d0b509 --- /dev/null +++ b/web/tests/unit/prompts.test.ts @@ -0,0 +1,195 @@ +// §8.2 prompts: shared system prompt (verbatim skeleton), per-artifact user +// prompts, transcript budget rule, weak-model hardening, PROMPT_VERSION. +import { describe, expect, it } from "vitest"; +import { + PROMPT_VERSION, + TRANSCRIPT_TOKEN_BUDGET, + buildSystemPrompt, + budgetTranscriptLines, + buildArtifactPrompt, + estimateTokens, +} from "@/lib/prompts"; +import { buildAnalyzePrompt } from "@/lib/prompts/analyze"; +import { buildVoiceLearnPrompt } from "@/lib/prompts/voice-learn"; +import { buildShortenPrompt } from "@/lib/prompts/shorten"; +import { DEFAULT_VOICE_PROFILE } from "@/lib/ai/voice"; +import { demoSegments } from "./helpers/ai"; +import type { VoiceProfile } from "@/lib/ai/voice"; + +const show = { name: "The Focus Show", defaultCta: "Follow the show.", allowedUrls: [] }; +const longSample = Array.from({ length: 500 }, (_, i) => `word${i}`).join(" "); + +const ctx = { + show, + voice: DEFAULT_VOICE_PROFILE, + voiceSamples: [longSample, longSample, longSample], + meta: { title: "Attention", number: 12, guests: ["Maya"] }, + segments: demoSegments, +}; + +describe("PROMPT_VERSION + shared system prompt", () => { + it("exposes a prompt version constant", () => { + expect(typeof PROMPT_VERSION).toBe("string"); + expect(PROMPT_VERSION.length).toBeGreaterThan(0); + }); + + it("fills the §8.2 verbatim skeleton with show and voice", () => { + const voice: VoiceProfile = { + ...DEFAULT_VOICE_PROFILE, + tone: ["clear", "warm", "curious"], + signaturePhrases: ["tiny experiments"], + neverSay: ["delve", "game-changer"], + alwaysInclude: [], + }; + const s = buildSystemPrompt(show, voice); + expect(s).toContain('for the show "The Focus Show"'); + expect(s).toContain("tone clear, warm, curious"); + expect(s).toContain("formality 3/5"); + expect(s).toContain("energy 3/5"); + expect(s).toContain("humor 2/5"); + expect(s).toContain('write as "we"'); + expect(s).toContain("medium\nsentences"); + expect(s).toContain("emoji: none"); + expect(s).toContain("You may use these phrases sparingly: tiny experiments."); + expect(s).toContain("Never use: delve, game-changer."); + expect(s).toContain("Include where natural: none."); + expect(s).toContain("Never invent names, numbers, links or claims."); + expect(s).toContain('return {"segmentId": "...", "exact": "..."}'); + expect(s).toContain("Return only JSON matching the schema below."); + }); +}); + +describe("per-artifact user prompts", () => { + it("show_notes carries counts, metadata, ≤2 bounded samples and numbered transcript", () => { + const p = buildArtifactPrompt("show_notes", ctx); + expect(p.user).toContain("4-7"); + expect(p.user).toContain("Episode title: Attention"); + expect(p.user).toContain("Episode number: 12"); + expect(p.user).toContain("Guests: Maya"); + expect(p.user).toContain("Follow the show."); + // 3 samples of 500 words each → only 2 samples, each ≤400 words survive + expect(p.user.match(/style reference only/gi)?.length).toBe(2); + expect(p.user).not.toContain("word460"); + expect(p.user).toContain("[seg_000000|Maya|00:00]"); + expect(p.tier).toBe("long"); + expect(p.temperature).toBe(0.4); + expect(p.truncated).toBe(false); + }); + + it("each artifact prompt returns its §7.2 schema for validation", () => { + for (const kind of [ + "show_notes", + "chapters", + "summary", + "titles", + "blog", + "newsletter", + "social", + "quotes", + ] as const) { + const p = buildArtifactPrompt(kind, ctx); + expect(p.schema).toBeTruthy(); + expect(p.system).toContain("Return only JSON matching the schema below."); + } + }); + + it("regenerate instruction is appended when given", () => { + const p = buildArtifactPrompt("titles", { ...ctx, instruction: "punchier" }); + expect(p.user).toContain("punchier"); + }); + + it("analyze uses the fast model tier at temperature 0.2 (weak-model hardening)", () => { + const p = buildAnalyzePrompt(ctx); + expect(p.tier).toBe("fast"); + expect(p.temperature).toBe(0.2); + expect(p.user).toContain("candidateQuotes"); + }); + + it("quotes artifact asks for exact character-for-character copies", () => { + const p = buildArtifactPrompt("quotes", ctx); + expect(p.user).toContain("character-for-character"); + expect(p.user).toContain("5"); + }); +}); + +describe("transcript budget (§8.2: >~60k tokens → first 40% + last 20% + top-ranked)", () => { + it("estimateTokens uses the chars/4 approximation", () => { + expect(estimateTokens("abcdefgh")).toBe(2); + }); + + it("keeps everything and reports not-truncated under the budget", () => { + const lines = ["a", "b", "c"]; + const r = budgetTranscriptLines(lines, { maxTokens: 1000 }); + expect(r.truncated).toBe(false); + expect(r.lines).toEqual(lines); + }); + + it("truncates: first 40% + last 20% kept, extreme overflow trims the head end", () => { + const lines = Array.from({ length: 20 }, (_, i) => `line-${i} filler filler filler filler`); + const r = budgetTranscriptLines(lines, { maxTokens: 30 }); + expect(r.truncated).toBe(true); + expect(r.lines).toContain("line-0 filler filler filler filler"); // episode start survives + expect(r.lines).toContain("line-19 filler filler filler filler"); + const head = lines.slice(0, 8); + const tail = lines.slice(16); + for (const l of r.lines) { + expect(head.includes(l) || tail.includes(l)).toBe(true); + } + }); + + it("fills leftover budget with top-ranked middle lines, in episode order", () => { + const lines = Array.from({ length: 20 }, (_, i) => `line-${i} filler filler filler filler`); + const ranked = lines.map((_, i) => ({ s: lines[i], i, score: 20 - i })); + // 12 head+tail lines ≈ 105 tokens; 135 leaves room for ~3 ranked middle lines + const r = budgetTranscriptLines(lines, { maxTokens: 135, ranked }); + expect(r.truncated).toBe(true); + const head = lines.slice(0, 8); + const tail = lines.slice(16); + const middle = r.lines.filter((l) => !head.includes(l) && !tail.includes(l)); + expect(middle).toEqual([ + "line-8 filler filler filler filler", // highest-ranked middle line first in rank... + "line-9 filler filler filler filler", + "line-10 filler filler filler filler", + ]); // ...but stored in episode order, which coincides here + }); + + it("big transcripts flip the truncated flag on the artifact prompt", () => { + const many: typeof demoSegments = []; + for (let i = 0; i < 4000; i++) { + many.push({ + id: `seg_${String(i).padStart(6, "0")}`, + idx: i, + speaker: null, + startMs: i * 1000, + endMs: (i + 1) * 1000, + text: `Segment ${i} talks about focus habits attention energy work and ordinary weeks with padding ${"x".repeat(60)}`, + }); + } + const p = buildArtifactPrompt("summary", { ...ctx, segments: many }); + expect(p.truncated).toBe(true); + expect(estimateTokens(p.user.split("TRANSCRIPT:")[1] ?? "")).toBeLessThanOrEqual(TRANSCRIPT_TOKEN_BUDGET + 100); + }); +}); + +describe("auxiliary prompts", () => { + it("voice-learn asks for the §7.4 profile shape", () => { + const p = buildVoiceLearnPrompt(["A past show note sample text."]); + expect(p.user).toContain("formality"); + expect(p.user).toContain("neverSay"); + expect(p.user).toContain("ctaStyle"); + expect(p.tier).toBe("fast"); + expect(p.temperature).toBe(0.2); + }); + + it("shorten carries the target limit and stays plain text", () => { + const p = buildShortenPrompt({ + show, + voice: DEFAULT_VOICE_PROFILE, + text: "Long text", + maxChars: 4000, + }); + expect(p.user).toContain("4000"); + expect(p.json).toBe(false); + expect(p.tier).toBe("fast"); + }); +});