feat: enrichment verification layer (verify + verifier-audit + staleness badge)#84
Merged
Conversation
PR-2 of the verification layer. A second, independent audit of ONLY the
consequential aggregated verdicts (verdict==FAIL OR divergent): the auditor
re-checks each judge flag against the source and CONFIRMS or REVOKES it, then
the merge recomputes the verdict deterministically — a confirmed flag keeps the
FAIL, a record whose flags are ALL revoked drops to REVIEW (or PASS if no
adherence issue remains), and a divergence-only tie is resolved by the auditor.
Catches both a lone hallucinated flag and a shared blind spot of the N judges,
mirroring cv-guardrail's cv-fact-verifier.
- new module `verification_audit.py`: consequential_records (selector),
export_audit_worksheet (source+output+flags+current verdict), import_audit_
judgments, load_report_records, merge_audit (deterministic re-verdict).
- new `rubric-verify-audit.md` ({language}) — the auditor's fail-safe CONFIRM/
REVOKE criteria (judge≠party, default-to-CONFIRM).
- wire `xbrain verify --audit` (export) and `--audit --apply` (merge + re-render
verify-report.{json,md}); report-only, never mutates the store.
- additively surface an `· audited` marker in the shared report renderer.
- README: document the `verify` command incl. the `--audit` stage.
Tests-first (TDD): selection, worksheet round-trip, merge/re-verdict + edge
cases (confirmed stays FAIL, all-revoked flips, divergent-no-flags, blind-spot
escalation, malformed/casing guards) + CLI end-to-end. Full `poe check` green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The panel converged on a fundamental gap: the merge re-verdict could silently wash a real FAIL to PASS. Redesign the seam around one enforceable invariant — a verdict lowers ONLY when the specific cited evidence that produced it is explicitly revoked; guards only ever escalate, never de-escalate a pre-existing FAIL. Safety lives in code, mirroring cv-guardrail.apply_verifier_audits. CRITICAL fixes: 1. Floor on the prior verdict: `_final_verdict` derives from the SHARED `verification.derive_verdict` core and clamps against the record's prior verdict. A raw verdict=FAIL (faithfulness/adherence PASS) or a flagless faithfulness FAIL has no revocable evidence → STAYS FAIL. A divergence-only tie is resolved by the auditor only when the prior verdict ≤ REVIEW. 2. Flag key = full normalised (claim, issue) pair (matching _union_flags), and a CONFIRM always wins over a REVOKE on the same key — no ordering-dependent erasure of a confirmed flag. 3. Axis scoping: flags now carry `axis` (faithfulness|adherence), tagged at the judge (rubric-verify.md) and threaded through _union_flags. A faithfulness FAIL clears only when every FAITHFULNESS flag is revoked; revoking an adherence note can never wash it. 4. Deterministic backstops: a confidence gate (a REVOKE applies only at confidence ≥ 0.7, else kept+gated) and a corpus-level mass-revocation guard (a run clearing a suspiciously high fraction of FAILs suppresses all such revocations → kept FAIL). New `confidence` field in rubric-verify-audit.md. HIGH/MEDIUM/LOW: thread the auditor's cited `reason` into the merged flags and render it; surface unmatched audits, washed and gated records in a new `## Audit` report section (audit_log); reject invalid audit/reverdict enums loudly on import; skip non-dict report records; export returns an honest (count, skipped); reject >1 --apply in audit mode (single independent auditor by design). merge_audit now returns (records, audit_log). Every washing path the reviewers reproduced is pinned as a regression test. Full `uv run --extra dev poe check` green (all A/B, 1192 tests, cov 95%). PR-1 aggregate behavior preserved. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rden Round-2 safety review reproduced two more washes on-branch. Both fixed with regression tests; report-only preserved; full gate green (all A/B, 1200 tests, cov 95%). BUG 1 — `_faith_after` ignored auditor-ADDED faithfulness flags when the prior faithfulness was FAIL: revoking the original flag while CONFIRMing a NEW hallucination washed to PASS (and left the confirmed flag on a green record). Fix: `_faith_after` is now symmetric and driven by STANDING faithfulness evidence — FAIL whenever any surviving/added confirmed faithfulness flag remains, or a prior FAIL was not fully revoked (covers the flagless FAIL). This makes the invariant "a PASS/REVIEW record never carries a confirmed faithfulness flag" hold by construction; a defensive `_anomalies` scan (audited records only) surfaces any violation in the `## Audit` report instead of swallowing it. BUG 2 — the mass-revocation guard was bypassable by splitting revocations across `--apply` runs (each run overwrites verify-report.json, re-reading the shrunk FAIL set; `dropped>=2` never fires at one-per-run). Fix: `verify --audit --apply` refuses a report that already contains `audited` records (single independent pass by design); `--force` overrides knowingly. Test nits: loud `confidence` validation on import (reject non-numeric / out-of-[0,1], allow omitted); REVOKE-then-CONFIRM same-key order-independence; assert the gated + unmatched `## Audit` lines render. New regression tests: test_merge_added_confirmed_faith_flag_reestablishes_fail_ after_original_revoked, test_pass_review_never_carries_surviving_faithfulness_ flag, test_merge_revoke_then_confirm_same_key_confirm_still_wins, test_import_audit_rejects_non_numeric_confidence, test_import_audit_rejects_out_ of_range_confidence, test_import_audit_allows_omitted_confidence, test_report_audit_section_renders_gated_and_unmatched_lines (unit); test_verify_ audit_apply_refuses_second_run_on_audited_report (CLI, incl. --force). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
feat(verify): verifier-audit stage (judge≠party) — PR-2 of the verification layer
Follow-up of the verification layer (#79). Adds an opt-in write path so `verify` can persist its verdicts, and a `generate` badge that surfaces a verdict only while it is still current for the output a reader sees. - models: additive, back-compat `Item.verification: dict[str, VerificationVerdict]` (keyed by target, defaults `{}` so every legacy item loads unchanged). Each verdict carries an `output_fingerprint` (sha256 of the exact judged text) + `verified_at` (UTC-validated). - verification: `fingerprint_output` (the single canonicalization shared by writer + reader) and `apply_verdicts_to_store` (defensive, in-place write). - cli: `verify --apply --write-verdicts` opt-in write (auto-snapshots `pre-verify-write-verdicts`); default `verify` stays report-only; `--write-verdicts` without `--apply` errors. - generate: `_verdict_badge` renders ❌ FAIL /⚠️ REVIEW (PASS unbadged) only when the recomputed fingerprint matches the stored one — a STALE verdict (output re-generated since) is silently NOT badged. Label localised via i18n. - docs: README (`--write-verdicts` + badge), ARCHITECTURE (field + staleness rule + generate badge), CLAUDE.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…mpute (#79) Two reviewers independently reproduced a HIGH bug: a LYING badge. The write path recomputed `output_fingerprint` from the LIVE store at `--write-verdicts` time, not from the JUDGED output. If the summary/digest/topics was regenerated in the export→judge→write window, the stored fingerprint = the NEW output the judge never saw, so `generate` matched it and badged the new output with a verdict about the old one (a fixed output could show a bogus ❌; a stale FAIL could read as current). Fix — capture the judged fingerprint at worksheet EXPORT and thread it through: - `export_verify_worksheet` stamps each entry with `fingerprint_output(item, target)`. - `import_verify_fingerprints` reads the judged map back from the worksheet(s). - `apply_verdicts_to_store(store, aggregated, fingerprints)` stores the export-time fingerprint verbatim — never a live recompute. `generate`'s compare now detects a change in EITHER window. Also (reviewer flags): - MEDIUM: partial writes are no longer silent — `apply_verdicts_to_store` returns a `VerdictWriteResult` (written + skipped-with-reasons); the CLI echoes the tally. - LOW: badge collapses internal newlines in a flag issue so it can't break the single-line blockquote; debug-log on skipped non-dict flags/judgments; type-harden `VerificationVerdict` — shared `Verdict` Literal for verdict/faithfulness/adherence, `output_fingerprint` constrained to `^[0-9a-f]{64}$`. Tests (all new/updated green): stored fingerprint == judged even when the output is regenerated before the write (unit + CLI + end-to-end no-badge); export stamps + import round-trips + rejects garbage; skip-reason tally; newline-in-issue badge. Docs synced (README, ARCHITECTURE, CLAUDE.md). Gate: 1225 tests, 95% cov, all A/B. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Last reviewer LOW: `import_verify_fingerprints` took first-seen (`setdefault`) when two applied worksheets stamped DIFFERENT `output_fingerprint`s for the same (item_id, target) — off the documented workflow (N judges copy ONE export → identical stamps), and not a lying badge, but the divergence was resolved silently. Now a key with conflicting valid stamps across worksheets is DROPPED entirely (logged) instead of guessed: it becomes `fingerprint-missing` at write, so no verdict is persisted and nothing is badged. A key all worksheets agree on still resolves and writes. Extracted `_worksheet_items` / `_entry_fingerprint` helpers to keep complexity A/B. Test: test_import_verify_fingerprints_drops_conflicting_stamps — two worksheets, same (item, target), different fingerprints → key dropped (not written, fingerprint-missing), while an agreed key still writes. Gate: 1226 tests, 95% cov, all A/B. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
feat(verify): staleness-aware verification-verdict badge (#79)
Integration merge bringing the verification-layer epic up to date with develop (docs-sync PR #81). Resolved README verify-row conflict by combining #81's phrasing (named worksheet file, executor-default note) with the umbrella's --write-verdicts/staleness-badge and --audit content. ARCHITECTURE.md and CLAUDE.md auto-merged as clean unions (verified: no duplicated verification sections, no content loss). Full gate green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The #81/develop verify docs were written before --write-verdicts existed; unioned against the umbrella's badge docs they left two semantic clashes git could not flag. - ARCHITECTURE.md `### verify`: scope 'never mutates/snapshots' to the default report-only mode; add the opt-in --write-verdicts (mutate+snapshot) and --audit (verifier-audit) clauses the prose omitted; add `verify --write-verdicts` to the items.json 'Mutated by' column. - CLAUDE.md verify bullet: same scoping + --write-verdicts/--audit clause. Docs-only; full `poe check` green (1226 tests, 95% cov, all A/B). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enrichment verification layer — integration merge of the epic (refs #79)
Umbrella branch bringing the full verification layer together, now merged up to date with
develop.What's in the layer
verification.py) scoring each enrichment output (summary/digest/topics) for faithfulness and rubric-adherence, aggregating N judges into a report-onlydata/verify-report.{json,md}(worst-faithfulness wins, divergence flagged). Keyless worksheet flow, mirrorscv-guardrail.--auditruns a judge≠party second pass over ONLY the consequential (FAIL/divergent) verdicts (verification_audit.py): an independent auditor CONFIRM/REVOKEs each flag, then a deterministic re-verdict under one invariant (a verdict lowers only when the specific cited evidence is explicitly revoked; guards only escalate). Three backstops: confidence gate (≥0.7), axis scoping, mass-revocation guard. Single-pass,--forceto re-audit.verify --apply --write-verdictspersists each verdict onto the additiveItem.verificationfield with a sha256 fingerprint of the exact judged output (stamped at worksheet EXPORT, threaded through to the writer).generaterecomputes the current output's fingerprint and badges (❌ FAIL /pre-verify-write-verdicts) and echoes a written/skipped tally.Integration merge
This branch was merged with
origin/develop(docs-sync PR #81). Conflicts resolved by combining both sides:verifyrow — kept the umbrella's--write-verdicts/staleness +--auditcontent, ported in docs: document video-digest + verify, sync stale video-digest render #81's named worksheet file (data/verify-worksheet.json) and theDefaults to [enrich].executor; worksheet tracks only (no api)note. docs: document video-digest + verify, sync stale video-digest render #81's absolute "never mutates/snapshots" was superseded by the umbrella's nuanced "report-only by default" (accurate post---write-verdicts).### verifysection / updateddevelop-based branch model) — verified no duplicated sections and no content loss.Gate
uv run --extra dev poe checkis entirely green: Ruff (lint+format), Mypy, Radon All A/B, Bandit, Detect-secrets, Deptry, Vulture, Interrogate 89.3% all pass — 1226 tests passed, coverage 95.01% (min 78%).🤖 Generated with Claude Code