Conversation
…esume token The scanner now installs a custom InstalledFlow delegate that writes state='awaiting_auth' plus the consent URL onto its ingest_runs row (via a new AuthState writer-channel message, so single-writer discipline holds) and returns the row to 'running' once tokens arrive. The delegate holds only a weak sender: the authenticator outlives the scan loop, and a strong clone would keep the writer channel open forever and deadlock the end-of-scan drain. Token acquisition happens eagerly, bounded by GMAIL_STATS_AUTH_TIMEOUT (default 600s); an unattended consent becomes failed/auth_required (policy_enforced keeps its dedicated kind). Terminal UX is unchanged: the URL is still printed unless --quiet, and no schema migration was needed (the auth_url column shipped with migration 1). scan --resume <run_id> (issue #26 Q4, best effort) starts listing from the run's persisted page token. A token Gmail rejects falls back to page one silently (logged in verbose mode); seen_mails dedupe keeps the counts correct either way. --resume on a bare scan is therefore no longer an error.
The viewer application moves from src/bin/web.rs into a library module (gmail_stats::webapp) so integration tests can drive the exact router the binary serves; src/bin/web.rs is now a thin main. Phase B behavior is unchanged. New surface (issue #30, per the #26 design): - POST /api/runs spawns the ingester as a supervised child: fixed argv (never a shell), stdin null, stdout null, stderr piped into a 200-line in-memory ring buffer, kill_on_drop(false) so runs survive viewer restarts. The binary is located as a sibling of current_exe() (GMAIL_STATS_INGEST_BIN override); spawned runs get --quiet and the viewer's --db path. Responses: 202 {run_id} (the child mints the id by inserting its run row; the viewer waits for it, matched by child pid), 409 run_active (friendly flock pre-check, plus mapping the child's own authoritative flock refusal), 422 bad_path/bad_resume/bad_source (absolute + regular + readable + mbox magic sniff), 500 spawn_failed with a cargo build hint. - POST /api/runs/{id}/cancel: owned runs only. SIGTERM through the supervisor that owns the child handle (never a pid read from the database), SIGKILL after a 10s grace; a SIGKILLed child's open row is abandoned by the next ingester's janitor - the viewer still never writes. 409 not_owned otherwise. - GET /api/runs/{id}/log: the owned child's stderr ring buffer, memory-only, gone on viewer restart. - CSRF stack on every mutating route, all three layers required: strict Content-Type application/json (415), X-Gmail-Stats-Csrf equal to the per-process token from /api/status (403, constant-time compare), and Origin / Sec-Fetch-Site validation when present (absent passes - curl is the same local user). No CORS headers anywhere. Host guard and the existing security headers stay on every route. - /api/status now reports owns_active_run truthfully and exposes auth_url on run rows (the client refuses to hyperlink anything that is not https://accounts.google.com/...). src/bin/fake_ingester.rs is a test-only stand-in driven by tests: it takes the real flock and writes real run rows, with modes for completing, crashing mid-run, honoring SIGTERM, ignoring SIGTERM, emitting stderr, and simulating a lost flock race. tests/web_mutating.rs covers the full 415/403/409/422/202 matrix via tower::ServiceExt::oneshot (asserting no Access-Control-* headers) and the supervision paths: normal completion, crash then janitor abandonment, cancel honored, SIGTERM-ignored SIGKILL escalation, viewer restart losing ownership while the run continues, log ring capping, and both flock-conflict shapes.
…rize link Onboarding cards go live against the real viewer: Start scan, and an absolute-path text input plus Import button (422 reasons render inline). The copyable CLI commands remain, and every launch control sits in a .live-only block that starts hidden and is only revealed once /api/status responds - the static Pages demo (404 status) never shows ingestion UI or fires a mutating request. Every POST sends Content-Type: application/json and the X-Gmail-Stats-Csrf token from the latest status payload. Cancel and Show log appear only when owns_active_run is true; non-owned runs keep the cancel-from-terminal hint. The log renders the stderr ring buffer via textContent into a pre block. awaiting_auth runs show an Authorize in Google link only when auth_url parses with origin https://accounts.google.com (target=_blank, rel=noopener noreferrer); anything else renders as inert text. History entries gain Retry (failed runs), Resume import (interrupted imports, fingerprint-validated by the ingester), and Resume - may restart from the beginning (failed scans, best-effort per #26 Q4); policy_enforced failures deep-link to the Takeout card, which stays pinned open until a run starts. All dynamic strings are rendered with textContent only. tests/frontend/ is a headless-Chrome harness (python3 tests/frontend/harness.py): a stub server serves the real web/ assets and records every request, so both sides are asserted - the DOM (launch controls hidden without a status endpoint, Cancel/Log visibility by ownership, link vs inert text for good vs hostile auth_url) and the wire (launch and cancel POSTs carry the CSRF token and JSON content type; the demo scenario fires no requests at all). demo/build_demo.py anchors are untouched and the build still passes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #30 — Phase C of the #26 design: the viewer gains its state-changing surface behind the full CSRF stack, onboarding buttons go live, and OAuth consent is drivable from the browser.
What's here
Viewer (now
gmail_stats::webapp, withsrc/bin/web.rsa thin main so tests drive the exact served router):POST /api/runs—{"source":"gmail_api"},{"source":"mbox","path":...}, or{"source":..., "resume_run_id":N}. Spawns the ingester as a supervised child: fixed argv (never a shell), stdin null, stderr into a 200-line in-memory ring buffer,kill_on_drop(false); binary located as a sibling ofcurrent_exe()withGMAIL_STATS_INGEST_BINoverride; spawned runs get--quietand the viewer's--db.202 {run_id}(the child mints the id by inserting its run row; the viewer waits for it, matched by the child's pid),409 run_active(friendly flock pre-check, plus mapping the child's own authoritative flock refusal after a lost TOCTOU race),422 bad_path/bad_resume/bad_source(absolute + regular + readable +Frommagic sniff),500 spawn_failedwith acargo buildhint.POST /api/runs/{id}/cancel— owned runs only; SIGTERM via the supervisor that owns the child handle (never a DB-supplied pid), SIGKILL after 10s, abandoned marking left to the existing janitor (the viewer still performs zero DB writes).409 not_ownedotherwise.GET /api/runs/{id}/log— owned child's stderr ring buffer, memory-only, gone on viewer restart.Content-Type: application/json(415),X-Gmail-Stats-Csrfequal to the per-process token from/api/status(403, constant-time compare), Origin/Sec-Fetch-Site validated when present, absent passes. No CORS headers anywhere; Host guard and security headers unchanged. One deliberate reading: the Origin allow-list checks schemehttp+ host127.0.0.1/localhostwithout pinning the port (mirroring the Host guard — any loopback origin is the same local user, and the token is the unforgeable layer)./api/statusnow reportsowns_active_runtruthfully and exposesauth_urlon run rows.Scanner:
InstalledFlowDelegatewritesawaiting_auth+auth_urlonto the run row (newAuthStatewriter-channel message, single-writer discipline intact) and returns it torunningonce tokens arrive; token acquisition is eager and bounded byGMAIL_STATS_AUTH_TIMEOUT(default 600s) →failed/auth_required(policy_enforcedkeeps its kind). No migration 2 needed —auth_urlshipped with migration 1. Terminal UX unchanged (URL still printed unless--quiet). The delegate holds a weak channel sender: the authenticator outlives the scan loop, and a strong clone would have deadlocked the end-of-scan writer drain (caught by a test hang).scan --resume <run_id>(Design proposal: drive scans and imports from the web UI #26 Q4, best-effort): starts listing from the persisted page token; a rejected/stale token falls back to page one silently (logged in verbose mode), correct either way viaseen_mailsdedupe.Front-end: launch buttons on the onboarding cards (mbox absolute-path input, inline 422 errors), Cancel + Log only when
owns_active_run, validated Authorize in Google link (https://accounts.google.com/origin only,target=_blank rel="noopener noreferrer", inert text otherwise), Retry on failed runs, Resume import on interrupted imports, Resume — may restart from the beginning on failed scans,policy_enforceddeep-link to the Takeout card. All POSTs carry the CSRF header from the latest status payload; all dynamic strings viatextContent. Launch controls live in.live-onlyblocks that start hidden and are revealed only after/api/statusresponds, so the static demo never shows ingestion UI or fires a mutating request.demo/build_demo.pyanchors intact.Tests (all passing locally: 90 Rust tests + 6 harness scenarios + live E2E)
tests/web_mutating.rs, viatower::ServiceExt::oneshoton both mutating routes): form/multipart/text-plain and missing content types → 415; JSON without token → 403; wrong token → 403; hostile Origins (https://evil.example,http://evil.example,https://127.0.0.1,null) → 403;Sec-Fetch-Site: cross-site/same-site→ 403; valid trio → 202; second start while active → 409run_active; relative/missing/directory/non-mbox paths and bad resume ids → 422; not-owned cancel → 409not_owned; malformed JSON → 400. Every response asserted to carry noAccess-Control-*headers.fake_ingesterbinary (takes the real flock, writes real run rows; modes: complete / crash / hang+trap / hang+ignore / stderr / flockfail / norow): normal completion; crash mid-run → open row, freed flock, abandoned by the next real ingester's janitor; cancel SIGTERM honored → cleancancelledrow; SIGTERM ignored → SIGKILL escalation (shortened grace), then janitor abandonment; viewer restart →owns_active_run:false, cancel/log refused, run continues and the original owner still cancels it; log ring capped at the last 200 lines; both flock-conflict shapes (pre-check 409, and the child losing the race post-pre-check still mapping to 409).awaiting_auth+ URL to the row and the post-consent message returns it torunning(no real OAuth — the delegate is invoked directly); consent timeout maps toauth_required, apolicy_enforcedmessage keeps its kind.python3 tests/frontend/harness.py, headless Chrome + a stub server that records every request): launch clicks POST with the CSRF token and JSON content type (asserted on the wire); anaccounts.google.comauth_url renders a_blank/noopener noreferrerlink while a hostile lookalike (accounts.google.com.evil.example) renders as inert text; Cancel/Log hidden when not owned (and shown + functional when owned, cancel POST captured with the token); the no-status-endpoint scenario shows zero ingestion UI and fires zero requests.target/debug/web, fetched the real token from/api/status,POST /api/runs(mbox, 2000-message synthetic file) with curl →202 {"run_id":1}→ watched/api/statustodone(2000 seen / 2000 new,bytes_done == bytes_total) →/api/summaryreported 2000 messages across 17 senders. Forged cross-site-style requests against the same live server were all rejected (415/403 across six probe shapes). Scan-source E2E, honestly scoped since no real OAuth consent was run: with a fake client secret andGMAIL_STATS_AUTH_TIMEOUT=8,POST {"source":"gmail_api"}→ 202, status showedawaiting_authwith anhttps://accounts.google.com/o/oauth2/auth?...auth_url andowns_active_run:true, thenfailed/auth_requiredafter the timeout. The full consent happy path and the stale-resume-token fallback against the live Gmail API were not exercised.cargo build,cargo test(90 tests),cargo clippy --all-targets -- -D warnings,cargo fmt --check,python3 demo/build_demo.pyinto a temp dir — all clean.Notes
stats.db/credentials.json/tokencache.jsonwere never touched; all E2E ran against scratch paths.GET /api/runs/{id}/logfor a non-owned run returns 404not_owned(409 felt wrong for a GET; the issue's 409 contract applies to cancel).