Skip to content

WP-04: AI provider registry + prompts - #6

Merged
gitAbid merged 2 commits into
mainfrom
wp-04-ai-registry
Oct 27, 2026
Merged

gitAbid merged 2 commits into
mainfrom
wp-04-ai-registry

Conversation

@gitAbid

@gitAbid gitAbid commented Oct 27, 2026 •

Copy link
Copy Markdown
Owner

Scope (WP-04 only)

web/lib/ai/ + web/lib/prompts/ + admin AI API routes per spec §8.1, §8.2, §7.4, §7.5. No admin UI (WP-09), no orchestration routes (WP-06). Branched from main@4b19927 (WP-03); origin/main (WP-01 + WP-02) merged into this branch and the two documented WP-02 seams are wired for real — details under "WP-02 integration" below.

Deliverables

  • Registry (web/lib/ai/): one generic OpenAI-compatible adapter + config presets exactly per §8.1 (glm-coding default, glm-open, groq fallback, openai, openrouter, anthropic native Messages, custom); selection order app_settings.ai → env → defaults; GLM multi-endpoint retry + working-endpoint cache ported from engine/glm.js; §8.1 call policy — primary (60 s timeout, 1 retry on 429/5xx with 2 s backoff) → fallback provider → deterministic builder (lib/engine) with status fallback; JSON via response_format json_object where supported + Zod (§7.2 schemas) + one repair call; §7.3 quote-by-reference resolution (claims capped per kind, verbatim offsets, candidate backfill) BEFORE validation; §7.1 usage_events logging with est. cost from lib/ai/prices.ts (vendor-verified prices, 0 when unknown).
  • Prompts (web/lib/prompts/): all 12 files per §8.2 + PROMPT_VERSION; shared system prompt verbatim skeleton; per-artifact rules with §7.2 counts; ≤2 voice samples ≤400 words ("style reference only"); 60 k-token transcript budget (first 40% + last 20% + top-ranked middle, records truncated); weak-model hardening (0.4 long / 0.2 analyze, flat schemas, inline JSON examples).
  • Voice (§7.4): profile schema + spec defaults; learnVoiceProfile (fast model, one repair, default profile on failure) and previewVoice.
  • Post-check wiring (§8.4, U8): neverSay scan → one regenerate with feedback → deterministic phrase replacement; religion-neutral gate on generated framing only (regen → next provider; never applied to transcript quotes); foreign-URL strip with subpath-aware allow list.
  • Admin AI API (§7.5): GET/PUT /api/admin/ai, POST /api/admin/ai/test — thin route wrappers over framework-free handlers; owner-only via requireOwner (Clerk currentUser + OWNER_EMAILS).

WP-02 integration (merged mid-flight, seams completed)

  • resolveAppSettingsStore() now returns a real Drizzle store over WP-02's app_settings table (get by key, put upsert); null only when DATABASE_URL is unset (keyless preview) → GET serves env/default view, PUT 503 settings_store_unavailable; unreachable DB → PUT 503 settings_store_write_failed.
  • requireOwner classifies with WP-02's lib/env.isClerkConfigured(): keyless previews are always denied (503) — app pages may render with a preview user there, but AI settings switching must never work under fake auth. U18 tests inject the gate + an in-memory store at the same documented interfaces.

Acceptance evidence

U8 — no banned word survives (tests/unit/ai.generate.test.ts, tests/unit/ai.postcheck-wiring.test.ts)

  • Draft with banned word → provider receives a second call whose prompt contains the banned term + "banned" feedback; clean retry accepted (regenerated: true).
  • Retry still violating → phrase replaced deterministically ("We delve into attention some more." → "We into attention some more."), result schema-valid, neverSay clean.
  • Religion-framing violation → regenerate → still violating → artifact pushed to next provider (never phrase-replaced).
  • Foreign URLs stripped, allow-listed URLs (incl. subpaths) kept.

U10 — all providers fail → deterministic package labelled fallback (ai.generate.test.ts "U10" block + live run)

  • Fake failing chain [glm-coding, groq] → status: "fallback", provider/model: null, errorCode: "all_providers_failed", bullets are transcript sentences, CTA from show settings; quotes artifact falls back to verbatim candidate refs; chapters without timestamps return {items: []} (no_timestamps), never fabricated times.
  • Also observed against the real failing GLM endpoint chain (see live test below): deterministic package labelled fallback in 567 ms.

U18 — owner switches provider/model; test button OK/error (tests/unit/ai.admin.test.ts)

  • PUT persists a valid selection through the store interface (in-memory store in tests; real Drizzle store wired for runtime); invalid bodies → 400; custom without base URL → 400; unwired store → 503; DB write failure → 503.
  • Next generateArtifact with the same store hits the saved provider/model (groq + saved model asserted on the request body).
  • POST test returns {ok: true, latencyMs, usage} for a working chain and {ok: false, error} on failure; unknown provider → 400; gate 401/403/503 covered.

Checks

  • pnpm lint green · pnpm typecheck 0 errors · pnpm build green (merged tree, incl. WP-02 routes/middleware) · pnpm test: 248 passed / 2 skipped (live) across 24 files (WP-04 added 92 tests to WP-03's 93; WP-02's suite rides along green) · pnpm e2e on the merged tree: 18 passed (all 4 spec files, local production build).
  • Preview: https://episodekit-4yfov19q2-pera-byte.vercel.app — home 200; all three admin endpoints verified live to deny correctly without owner config (503 {"error":"owner_not_configured"}).

Live provider test — attempted, blocked by key state (no fake results)

Ran tests/unit/live.ai.test.ts locally against the real endpoints with .env.relaunch sourced:

  • GLM: registry called the real glm-coding flavors — observed 401 {"error":{"code":"401","message":"token expired or incorrect"}} on all four (z.ai coding ×2 models, z.ai paas, open.bigmodel.cn). The GLM_API_KEY in .env.relaunch is 11 characters — not a valid key shape. Blocker (owner): provide a valid GLM key (or activate the coding-plan resource per §8.1 — error 1113 was never reached because auth fails first). The multi-endpoint fallback order behaved exactly as ported.
  • Groq: ASR_PROVIDER in .env.relaunch is not groq and ASR_API_KEY returns 401 from api.groq.com — no Groq LLM key available. Blocker (owner): provide GROQ_API_KEY for the fallback live test.
  • The live test file skips in CI (skipIf(!GLM_API_KEY || CI)) and will produce model id/latency/usage numbers with valid keys; rerun command is in its header. No fabricated numbers are recorded here.

Decisions logged (docs/relaunch/DECISIONS.md)

Groq fallback models chosen from Groq's current model list: openai/gpt-oss-120b (long) / openai/gpt-oss-20b (fast) — llama-3.3/3.1 chat models are now Enterprise-only per console.groq.com/docs/models (2026-10-27); gpt-oss models are self-serve with verified prices ($0.15/$0.60 and $0.075/$0.30 per M) and 131k context. Also: vendor-verified price table only (GLM rows absent → 0); requireOwner deny-by-default, keyless previews never own AI settings; wholesale selection precedence; fallback content not padded to AI minima; quote resolution before validation with candidate backfill; subpath-aware URL allow list; units = total tokens for llm events; live test as CI-skipped vitest; OpenAI/OpenRouter placeholder defaults; WP-02 seams completed via merge.

Blockers / notes for the integrator

  1. Owner: valid GLM_API_KEY (current one is an 11-char placeholder) and GROQ_API_KEY needed before live generation and the WP-10 eval gate.
  2. WP-06: call generateArtifact per artifact kind; persist status/provider/model/error_code on the artifact row; truncated is on GenerateResult (no §7.1 column — decide storage or ignore).
  3. WP-09: /admin UI consumes GET/PUT /api/admin/ai + POST /api/admin/ai/test as-is; with Clerk + OWNER_EMAILS configured the gate opens with no further code.
  4. CI note: two pre-existing @types/node peer warnings from vitest (present since WP-00, untouched).

lib/ai: one generic OpenAI-compatible adapter + §8.1 preset table
(glm-coding default, glm-open, groq fallback, openai, openrouter,
anthropic native Messages, custom), selection order app_settings.ai →
env → defaults, GLM multi-endpoint retry + working-endpoint cache
ported from engine/glm.js, §8.1 call policy (60s timeout, 1 retry on
429/5xx with 2s backoff → fallback provider → deterministic builder
labelled fallback), JSON mode + Zod validation + one repair call,
§7.3 quote-by-reference resolution before validation, §8.4 post-check
wiring (neverSay scan → regenerate with feedback → phrase replace,
religion gate, foreign-URL strip), §7.1 usage_events logging with
verified per-model prices (0 when unknown).

lib/prompts: shared §8.2 system prompt (verbatim skeleton), 12
per-artifact prompts with PROMPT_VERSION, 60k-token transcript budget
(first 40% + last 20% + top-ranked middle), weak-model hardening
(temp 0.4 long / 0.2 analyze, flat schemas, inline JSON examples).

Voice (§7.4): profile schema + defaults, learn + preview functions.

Admin AI API (§7.5): GET/PUT /api/admin/ai + POST /api/admin/ai/test,
owner-gated via requireOwner (OWNER_EMAILS + Clerk currentUser, deny-
by-default 503 when unconfigured); app_settings store is an injected
interface with a documented WP-02 seam.

U8 U10 U18 covered by unit tests; live smoke test included but
CI-skipped (tests/unit/live.ai.test.ts). E2E_PORT makes the
Playwright port configurable for parallel worktrees.

Decisions logged in docs/relaunch/DECISIONS.md.
# Conflicts:
#	docs/relaunch/DECISIONS.md
#	web/package.json
#	web/playwright.config.ts
#	web/pnpm-lock.yaml
@gitAbid
gitAbid merged commit a458802 into main Oct 27, 2026
2 checks passed
@gitAbid
gitAbid deleted the wp-04-ai-registry branch October 27, 2026 13:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant