Takeout mbox importer + ingestion coordination (Phase A of #26)

[zbuc]

Closes #25

This is Phase A of the ingestion design in #26: the CLI-only importer plus the coordination primitives (run tracking, locking, schema versioning, clean shutdown, path flags). Deliberately no web/viewer changes and no new endpoints — but a terminal scan already writes live run rows, and Ctrl-C yields a clean cancelled row with resume state, ready for Phase B's read-only progress UI.

What's here

• gmail_stats import <path.mbox> — streaming, header-only mbox importer (src/mbox.rs): bounded memory regardless of file size (capped reusable line buffer over a BufReader; bodies skipped without interpretation), mboxrd >From escaping, CRLF, over-long lines truncated but fully consumed so framing survives, malformed regions skipped to the next valid From separator and counted (as are messages missing a Message-ID). Dedupe key is mid:<Message-ID> in seen_mails — the permanent namespace rule from Design proposal: drive scans and imports from the web UI #26 — so re-imports add zero counts and RFC ids can't collide with Gmail API ids. Sender extraction reuses the scan's From→Return-Path priority (shared pick_sender) and cleanup_sender, so counts are consistent across sources.
• Resume: bytes_done on the run row is the committed parse offset (only ever advanced behind the FIFO writer channel, so it never points past uncommitted work). import <path> --resume <run_id> validates a fingerprint (size + mtime + FNV-1a of the first 64 KiB) and seeks to the offset; on mismatch it falls back to a full re-parse, which dedupe keeps correct.
• ingest_runs (src/ingest.rs): one row per run — source, state (running/done/failed/cancelled/abandoned), pid, started/updated/finished timestamps with a ~2s heartbeat, messages_seen/messages_new, bytes_done/bytes_total, persisted scan resume_token, error_kind/error. Written by both modes through the existing single-writer channel (new WriteMsg::Progress variant next to Seen); messages_new increments inside the same transaction as the insert, so it's exact across writer restarts. The next ingester marks rows orphaned by a dead process as abandoned — safe because liveness is judged by the flock, never pids/heartbeats.
• Schema versioning: PRAGMA user_version + guarded migrations. Existing tables are the version-0 baseline (still applied as idempotent DDL); ingest_runs is migration 1. Old DBs upgrade transparently on startup; DBs newer than the binary are refused with a clear message.
• flock mutual exclusion: both modes take an exclusive kernel flock on <db>.ingest.lock before touching the DB; a second ingester exits promptly with "another ingester is already running…". Crate choice: rustix (fs feature only) — safe wrappers so this crate stays free of unsafe, much smaller than nix, and maintained; Unix-only per maintainer decision 6.
• Clean shutdown: SIGTERM/SIGINT stops fetching/parsing, drains the writer channel, persists resume state, marks the row cancelled, exits 0 (a second signal aborts immediately). The scan also stops advancing its resume token past a partially-fetched page.
• Path flags: --db, --credentials, --tokens with GMAIL_STATS_DB/_CREDENTIALS/_TOKENS env fallbacks and the previous cwd-relative defaults; bare cargo run is still the OAuth scan. Arg parsing is hand-rolled (~90 lines incl. --flag=value) — a clap dependency didn't seem justified for two subcommands and six flags.
• Output: current main printed a sender: … line per message — noisy and it sprays inbox metadata into the terminal, so per the design note this is now off by default. Default output is a periodic progress line; --verbose restores per-message detail; --quiet leaves only errors + the final summary.
• Scan hardening along the way: missing/unreadable credentials.json and authenticator failures now produce a failed run row with error_kind (missing_credentials, auth_required, policy_enforced detected by message) instead of a panic, plus a hint pointing APP users at Takeout; best-effort total_estimate from users.getProfile.
• README: both ingestion modes and when to choose which, the Takeout how-to (takeout.google.com → Mail → mbox) with the ~2-day Advanced Protection delay warning, the new flags, and the interim caveat that mixing scan + import in one DB double-counts until Design proposal: drive scans and imports from the web UI #26's Phase D dedupe lands.

The first commit is a whitespace-only rustfmt pass on src/bin/web.rs (it was already failing cargo fmt --check on main); no functional web change, and easy to drop if unwanted.

Testing

cargo build, cargo test (55 tests), cargo clippy --all-targets -- -D warnings, and cargo fmt --check all pass.

• Parser fixtures: normal/CRLF messages, >From escaping (and the unescaped-separator case), missing Message-ID, malformed-region recovery with skip counts, folded headers, case-insensitivity/first-wins, 24 KiB lines, header-only messages, resume-from-offset; plus a ~50 MB on-disk mbox and a ~64 MB generated stream (a Read impl that synthesizes messages on the fly, so the input provably never exists in memory).
• Import end-to-end (temp DBs): populate + re-import adds zero new counts with mid: namespacing verified; resume with matching fingerprint imports only the tail; stale fingerprint falls back to a full parse; resuming a scan run is rejected; pre-cancelled import yields a cancelled row; non-mbox files are refused by the From sniff.
• Coordination: second flock acquisition refused in-process; fresh-DB and version-0-with-data migrations (data preserved, duplicates collapsed, user_version bumped), newer-DB refusal; run-row lifecycle (running → abandoned/failed); writer dedupe + progress accounting.
• Manual smoke test against a scratch directory: imported a synthetic 349 MB / 120k-message mbox — importer RSS stayed ~19 MB while streaming; a concurrent second ingester was refused by the flock (verified cross-process); SIGTERM mid-run exited 0 with a cancelled row at 87,921 messages; --resume then imported exactly the remaining 32,079 (500 senders, 120,000 total — no overlap, no gap); a scan with missing credentials recorded failed/missing_credentials and exited nonzero without touching OAuth.

Not exercised: the real OAuth flow / live Gmail API (no credentials in this environment). The scan path was refactored mechanically (channel message type, run row, cancellation checks) and compiles/passes the existing transient-error tests, but a live end-to-end scan should get a once-over before/after merge. total_estimate via get_profile is best-effort and ignored on failure, so it can't break a scan.

Deferred by design (per the #26 phase plan): consuming the persisted scan resume_token on a later scan (decision 4 is still open — it's persisted, not yet read back), awaiting_auth/auth_url states and the auth timeout (Phase C), and cross-source dedupe (Phase D).