Watch ingestion runs from the web viewer (observe-only, Phase B)

[zbuc]

Closes #28

Phase B of the #26 design ("watch from the browser"): the viewer surfaces ingestion state while staying strictly read-only — no DB writes, no state-changing endpoints, no CORS, zero new attack surface.

Server

GET /api/status — the single cheap poll target:

• db: ready|missing|empty (drives onboarding; a database file whose tables were never created reads as empty). Never 5xx for a missing/empty DB; only busy/unexpected errors use the existing 503/500 contract.
• active_run / last_run from the read-only ingest_runs view; a missing ingest_runs table (pre-Phase-A database) is simply "no runs".
• Flock probe of Phase A's lockfile (ingest::ingest_lock_path, reused via a new thin src/lib.rs rather than duplicated): open with create(true).truncate(false) exactly like the ingester, non-blocking try-lock, immediate release. Never truncates, never holds, detects "something is running" even before the first run row exists (verified by a test that plants sentinel bytes in the lockfile and locks around the probe).
• owns_active_run: false always (the viewer spawns nothing this phase), mixed_sources (interim caution, retired by Phase D).
• rate_per_sec / eta_seconds: computed in viewer memory from a 30s sliding window of successive status observations, reset on run change — never persisted anywhere. ETA prefers byte progress (exact for imports) and falls back to total_estimate for scans.
• csrf_token: 32 bytes from the OS CSPRNG, hex, minted at startup, per the Design proposal: drive scans and imports from the web UI #26 CSRF design. No endpoint consumes it until Phase C; shipping the slot now means it is only ever obtainable via this same-origin GET.

GET /api/runs?limit=20 — history straight from ingest_runs, newest first, limit clamped to 1..=100; missing table/file ⇒ {"runs": []}.

Both routes sit behind the unchanged Host guard + CSP default-src 'self' + nosniff + no-referrer middleware; tests assert the headers are present on the new routes and that no Access-Control-* header is ever emitted. The pool stays read_only(true) + query_only=ON; a test asserts /api/status against a nonexistent path does not create the file. The viewer also honors GMAIL_STATS_DB (default ./stats.db unchanged), matching Phase A's path flexibility.

Front-end (all new DOM via createElement/textContent; CSP untouched)

• Onboarding (db missing/empty, idle): the two cards — OAuth scan with the Advanced Protection policy_enforced callout, and Takeout import with the ~2-day export warning — each with the exact CLI command and a Copy button. No launch buttons (Phase C).
• Observe-only progress panel (collapsible via native <details>): source badge, state chip, seen/new counts, msg/s, determinate bar + ETA when a total exists (bytes for imports, total_estimate for scans), indeterminate sweep otherwise; display-only "stalled?" hint when the heartbeat is older than 15s (compared against server now_unix, not the client clock); non-owned runs say "started from the terminal — cancel it with Ctrl-C there".
• Live-filling stats: header pill ("Importing · 12,004 messages · 840/s · 61%"); /api/summary re-fetched every ~3.5s during active runs plus once on completion, preserving search/page/page-size/hidden-senders (all client-side); busy 503 keeps the previous table and dims the freshness stamp.
• History strip from /api/runs (failed runs show error_kind: error as inert text) and the interim mixed_sources caution banner.
• Polling cadence: 1s while active, doubling to a 10s idle ceiling, fully paused on document.hidden (immediate poll on return).
• Demo/static degradation: a 404 (or non-JSON 200) from /api/status hides every piece of ingestion UI and stops status polling for good; demo/build_demo.py builds with all anchors intact — the fetch("/api/summary") anchor still occurs exactly once, and the new /api/status + /api/runs fetches are separate call sites.

src/lib.rs exposes ingest/mbox as a library (the extraction #26 anticipated) so the viewer reuses Phase A's lock naming and DDL/migrations instead of copying them; src/main.rs now consumes them through the crate — no behavior change.

Testing

• cargo build / cargo test (66 tests: 22 lib, 31 scanner, 13 viewer) / cargo clippy --all-targets -- -D warnings / cargo fmt --check all clean; python3 demo/build_demo.py <tmp> succeeds.
• New endpoint tests run the real router via tower::ServiceExt::oneshot against temp databases created with Phase A's actual ingest::migrate DDL: status for missing DB (200, db:"missing"), empty DB, pre-Phase-A DB without ingest_runs, populated DB with an active running row + a finished failed row (field-level assertions incl. error_kind), mixed_sources flip, the non-truncating lock probe under a real held flock, /api/runs ordering/limit/garbage-limit/missing-DB, security headers + no-CORS on both new routes, host-guard 403s, CSRF token stability/uniqueness, and unit tests for the rate-window/ETA math.
• Front-end, headless Chrome against a static dir with stubbed status/runs/summary JSON (the Viewer: search filtering, pagination, and per-sender hide with show-hidden toggle #22 harness pattern, extended with a state-stubbing driver): onboarding renders on db:"missing" (cards, commands, callouts; stats/panel/pill hidden); an active mbox run renders the pill ("Importing · 12,004 messages"), determinate 61% bar, "840.5 msg/s", "~23s left", running chip, terminal-origin note, mixed-sources banner, and history incl. a failed run whose <script>-bearing error string renders inert; a stalled heartbeat shows the display-only hint; the built Pages demo (no /api/status) shows stats from summary.json with every ingestion element hidden. 28/28 assertions pass.
• End-to-end exit criterion (no credentials, scratch dir only): generated a synthetic 60,000-message / 12.3 MB mbox, started the viewer against a not-yet-existing DB (status: db:"missing", lock free), then ran gmail_stats import from a terminal-style background process. Polling /api/status observed the run 36 times: bytes_done advanced monotonically 0 → 12,099,390 with ingest_lock_held:true throughout and viewer-derived rates (~2,500–3,800 msg/s) and ETAs; final status showed active_run:null, last_run.state:"done", 60,000 seen/new, lock released, and /api/summary reported 400 senders / 60,000 messages. A live headless-Chrome DOM dump mid-import showed the pill ("Importing · 44,924 messages · 75%"), the running chip, the 74.9% bar, and the terminal-origin note.

Deferred / adjusted (relative to #26's full UI-flow text)

• Everything mutating (launch/cancel/log, awaiting_auth authorize-link flow, resume/retry buttons, remediation buttons on failures) is Phase C by design; failed runs surface here via the history strip with error_kind + error text.
• resume_token and mbox_fingerprint are internal resume bookkeeping and are not exposed in the API payloads; everything else on the run row is.
• The panel's "collapsible banner" is a native <details> element rather than custom JS.

Part of the incremental plan in #26 (Phase B).