Enrichment verification layer (LLM-as-judge evals for summary/digest/topics)

[VGonPa]

Problem

xbrain has rubrics (the generation spec) and validate.py/guardrails.yaml (mechanical checks: valid slugs, primary_topic ∈ topics, 1-4 topics, non-empty summary/digest). But there is no semantic verification of the generated enrichment:

• no faithfulness check — a hallucinated number/claim in a summary or digest passes;
• summary length (1-3 sentences) is rubric-only, unenforced;
• topic correctness isn't checked (only slug validity);
• no LLM-as-judge / eval set / regression harness.

The 178 summaries + 193 digests just backfilled were verified only mechanically + by human spot-checks + agent self-report — no independent judge reviewed them. This is the "Evals are the new PRD" gap: guardrail on generation, none on output.

Proposal

An xbrain verify step — an LLM-as-judge ensemble that scores each output (summary / video digest / topics) for faithfulness (claims/numbers supported by the source transcript + frames?) and rubric-adherence, emits a per-item verdict PASS / REVIEW / FAIL + cited flags, and audits the consequential verdicts with a judge≠party pass. Mirrors the existing cv-guardrail (judges → aggregate → verifier); can hook deslop for voice. Keyless worksheet+agents engine, like the rest of xbrain.

Plan

Spec: zz-support-files/docs/implementation-plans/enrichment-verification.md.

• PR-1 — verify machinery: rubric-verify.md, verification.py (select / export worksheet / import / aggregate / render report), xbrain verify CLI. Pure functions, unit-tested.
• PR-2 — verifier-audit stage (judge≠party re-check of FAIL/divergent clusters).
• PR-3 — run over the corpus (178 summaries + 193 digests) → report.
• Follow-ups: api engine; write verdict onto the record + generate badge; deslop voice hook.

Decisions

Report-only (v1) · N=3 judges + verifier · PASS/REVIEW/FAIL, faithfulness primary · worksheet/agents engine · targets summary+digest+topics.

🤖 Generated with Claude Code