Cross-source rfc_message_id dedupe and mixed-sources banner retirement (Phase D)

[zbuc]

Closes #32 — the final committed phase (Phase D) of #26: exact cross-source dedupe via rfc_message_id, and retirement of the interim mixed-sources banner.

What's here

Schema (migration 2, user_version 2). seen_mails gains a nullable rfc_message_id column with a non-unique index. mid: rows are populated during the migration itself from the Message-ID embedded in their key; bare Gmail rows stay NULL until backfilled. A database from a newer binary is still refused. Message-IDs are normalized by one shared helper (trim whitespace, strip one enclosing <>, case preserved) used by the scanner, the importer, the migration, and the backfill.

Dual-direction check-before-increment. Both ingesters record the normalized Message-ID when marking a message seen; inside the same single-writer transaction, a newly inserted row whose Message-ID already exists on any other seen_mails row is marked seen (idempotency intact) but not counted — neither the sender count nor the run's messages_new moves (there was no room in the run row schema for a distinct deduped counter, so deduped messages read as seen-not-new). Works in both directions: scan-after-import and import-after-scan. Messages without a Message-ID keep the old behavior (counted, NULL column).

scan --backfill-message-ids. A first-class ingest run: takes the flock, writes a run row (source: backfill, with heartbeats and total_estimate), stops cleanly on SIGTERM/SIGINT, honors --quiet/--verbose, and shows up in the web viewer's progress UI like any other run. It iterates bare-Gmail-id rows with NULL rfc_message_id in keyset-paginated batches (cursor over mail_id, so writer-lag can never re-select in-flight rows), fetches format=metadata restricted to Message-ID/From/Return-Path through the scan's existing rate-limited, retrying fetch machinery, and populates the column.

The collapse of historical double counts is fused into the populate rather than run as a separate phase: each row's duplicate check and sender decrement (never below zero) commit atomically with its populate. That is what makes partial completion + resume exact — a killed run leaves every row either fully repaired or untouched, re-running the same command continues with precisely the remainder, and a second run over a repaired database finds nothing to do and never touches the network. Rows fetched and found to have no Message-ID get an '' marker so they are not re-fetched (forward ingestion keeps NULL for such messages, per the issue; the marker exists only so the backfill terminates). No --resume flag: resumption is inherent, and combining the two is rejected.

Banner retirement. mixed_sources in /api/status now means "both keyspaces present AND at least one bare-Gmail-id row has NULL rfc_message_id", computed from seen_mails itself, with unrepaired_count exposed for the UI copy. A pre-Phase-D database (no column yet) counts every scan row as unrepaired and does not 5xx the read-only viewer. The banner explains that counts may include cross-source duplicates and shows the exact backfill command in a copyable code block (textContent only); it disappears once repaired. README's "don't mix both modes" warning is replaced with dedupe + backfill documentation.

Honest limitation

The live Gmail fetch inside the backfill was not run — it needs real OAuth credentials, which this change never touches. It shares fetch_message's machinery (now parameterized to add format=metadata + metadataHeaders) and the scan's auth path, and the fetch loop is generic over the fetcher, so everything downstream of the network — iteration/pagination, header extraction, populate, collapse, resume — is covered by tests against fabricated rows and messages. stats.db/credentials.json/tokencache.json in the repo root are untouched.

Tests

• Migration: fresh DB, v0 upgrade, v1 upgrade with mid: population, newer-version refusal, re-run no-op
• Normalization: <id@host> / id@host / whitespace / case / unbalanced-bracket variants
• Forward dedupe both directions with fabricated rows: overlapping scan+import sets count the true union; id-less messages still counted; replays add zero
• Backfill: fixture DB with historical double counts repaired exactly once; never below zero; interrupted-midway resume; second run no-op; loop selects only pending bare rows; cancel between batches
• Status: single source / mixed unrepaired (+count) / mixed repaired / import-only / pre-Phase-D column-less DB
• Front-end (headless-Chrome harness, 8 scenarios incl. 2 new): banner shows the command when unrepaired, absent when repaired

Gates: cargo build, cargo test (49+41+13 pass), cargo clippy --all-targets -- -D warnings, cargo fmt --check, python3 demo/build_demo.py into a temp dir (all anchors intact), python3 tests/frontend/harness.py (all scenarios pass).