Design proposal: Gmail filter generation from sender stats

[zbuc]

Turn what gmail-stats already knows — the ranked sender list from /api/summary and the hidden-senders intent signal in localStorage — into Gmail filters the user can actually apply. Follows the phased design-issue → issue-N branch → one-PR convention established in #26.

Prior art: gmailctl (MIT), whose Gmail filter-import XML format, v1alpha3 config schema, and OR-splitting limits inform this design. No gmailctl code is reused; only its documented/observed formats.

Summary

A Filter Studio built entirely into the existing web viewer (web/app.js / index.html / app.css), generating two downloadable artifacts client-side:

1. Gmail-native filter XML (primary) — applied manually via Gmail Settings → Filters and Blocked Addresses → Import filters. This is the universal baseline — the only apply path that works for an Advanced-Protection-enrolled account for which Google rejects all Gmail OAuth scopes outright (policy_enforced, see Google Takeout mbox importer as an alternative to the Gmail API scan #25). Zero credentials, zero network, and the import is additive (never deletes existing filters).
2. gmailctl-compatible v1alpha3 config (secondary) — plain JSON written to a .jsonnet file (Jsonnet is a strict superset of JSON, so JSON.stringify suffices). Power users adopt it into gmailctl's diff/apply lifecycle with their own OAuth setup; gmailctl export reproduces the same XML from it, which doubles as a cross-check.

No new server surface, no Rust changes, no new dependencies, no Gmail API calls — ever, from gmail-stats itself.

Chosen architecture and why

Pure client-side generation. All inputs already live in the browser: /api/summary ships the full ranked sender list in one payload, and the hidden-senders list (localStorage["gmail-stats.hiddenSenders"]) exists only client-side. Generation is a pure function ending in a Blob download via <a download>.

Consequences:

• Zero new attack surface. No mutating routes, so the three-layer CSRF stack is not implicated; no CORS considerations; the viewer's connection-level read-only DB invariant is untouched; the error contract gains no cases.
• The static Pages demo runs the feature at full fidelity. No new fetch targets means demo/build_demo.py needs no new replace_exactly deltas and no ingest.available gating — the demo becomes the feature's showcase instead of hiding it.
• Nothing to maintain in Rust. The generator is a few hundred lines of JS exercised by golden-fixture tests.

Rejected alternatives:

• Server-side export endpoint (POST /api/filters/export): adds a CSRF-guarded mutating route, a demo-degradation branch, and build_demo anchors — for a pure function whose inputs the server doesn't even have (the hidden list would need to be POSTed up). No capability gained.
• Shelling out to a user-installed gmailctl binary for compile/diff/apply: the only credential-free gmailctl operation (export) adds nothing over emitting the XML ourselves, and gmailctl apply is a full reconcile — it deletes any upstream filter absent from the config — a filter-destroying footgun if wired to a standalone generated config. Diff/apply serve only the secondary persona, at the cost of new supervised-subprocess endpoints and version-skew surface. Users who want that lifecycle get it by taking the jsonnet artifact to gmailctl directly, outside gmail-stats.
• In-viewer Gmail API apply: rejected in the literal sense — the viewer process never holds a write scope or talks to Google. But API apply itself is committed for OAuth-capable users (maintainer decision, 2026-07-24: the APP-enrolled maintainer is not the only user) as phase F4, following the Phase C spawn model: a new ingester-binary subcommand owns the OAuth and the API calls, spawned as a supervised child behind check_csrf. Additive-only — never a gmailctl-style reconcile.

Filter model

Criteria: from: only. The DB stores senders(sender, mails_sent) and dedupe ids — no subjects, labels, dates, or List-IDs — so sender-address criteria are the only ones with data behind them. Senders are already normalized to the bare address part by the ingesters (get_sender/cleanup_sender).

• Defensive validation before inclusion: each sender must match a conservative address shape (roughly ^[^\s"<>@,{}()]+@[^\s"<>@,{}()]+\.[^\s"<>@,{}()]+$). Non-conforming strings (sender values are attacker-controlled From: headers) are excluded from generation and listed in the review UI as "skipped — not a valid address". This is the correctness guard; the security guard is structural serialization (below).
• Domain roll-up (opt-in, suggested): when ≥3 selected senders share a registrable domain, offer "replace with one rule for the whole domain", enumerating exactly which other known senders from the stats list the domain rule would additionally sweep in (the DB knows every sender at that domain — show it honestly). Freemail/shared domains (gmail.com, outlook.com, yahoo.*, icloud.com, proton.me, …) are hard-excluded from suggestion. Never automatic.

Actions: presets mapping to Gmail XML properties / gmailctl actions:

Preset	XML properties	gmailctl actions
Archive + label (default)	shouldArchive, label	archive: true, labels: […]
Mute (archive + mark read, optional label)	shouldArchive, shouldMarkAsRead, label	archive, markRead, labels
Label only	label	labels: […]
Trash	shouldTrash	delete: true

Plus an explicit "never send to spam" toggle (shouldNeverSpam) for protecting wanted senders. Trash is confirmation-gated with copy stating it silently trashes all future mail from those senders. Deliberately omitted: forward (a generator emitting forwardTo from UI state influenced by hostile data is an exfiltration primitive we don't need), mark-spam, star/importance/category (clutter; the internal rule model is generic so adding them later is UI-only).

Consolidation: group selected senders by identical action-set; each group becomes one logical OR-of-from rule; chunk at 15 addresses per emitted <entry> (margin under gmailctl's default 20-leaf split and Gmail's ~1500-char per-filter ceiling). One label per filter (Gmail's constraint), enforced as one label per group. Review shows chunk counts ("this rule exports as 3 Gmail filters") and a soft warning near Gmail's 1000-filter account cap.

Volume preview: per rule, Σ mails_sent and share of total_messages ("These 4 rules cover 12,381 of 48,920 messages, 25.3%") — labeled as historical volume from the scanned corpus, not a promise about future mail.

Artifact formats

1. gmail-filters-YYYY-MM-DD.xml (primary). The Atom-feed dialect Gmail's importer accepts (xmlns="http://www.w3.org/2005/Atom", xmlns:apps="http://schemas.google.com/apps/2006"; one <entry> per chunk carrying <apps:property> pairs: from, label, shouldArchive, shouldMarkAsRead, shouldTrash, shouldNeverSpam). Built exclusively via DOM APIs — document.implementation.createDocument + createElementNS/setAttribute + XMLSerializer — never string concatenation, so hostile sender/label strings are escaped structurally. Because we serialize through the platform rather than porting gmailctl's marshaller, no code is copied (if any gmailctl logic is ever ported verbatim, it carries the MIT attribution).

2. gmail-filters.jsonnet (secondary). A v1alpha3 document via JSON.stringify(doc, null, 2) — one rule per logical group, no manual chunking (gmailctl's compiler splits itself). Offered in two variants: standalone config, and snippet form (bare {rules, labels} with a comment-header merge recipe) — because gmailctl apply reconciles the entire filter list, existing gmailctl users must merge, not replace; the UI says so.

3. Draft backup (implicit). The in-progress draft is exportable/importable JSON so a profile wipe doesn't lose curation.

API surface

New endpoints: none. New localStorage keys, same idiom as hiddenSenders/pageSize:

• gmail-stats.filterDraft — the in-progress builder state (selection, groups, actions, labels); survives refresh and the silent summary refresh during active runs; cleared on export or reset.
• gmail-stats.filteredSenders — { [sender]: { exportedAt, preset } }, written on download. Powers per-row "filter exported" badges and the "new senders only" export scope — important because Gmail's importer appends, so re-importing overlapping files creates duplicate filters. Includes a clear affordance for users who deleted filters in Gmail.

UI flows

1. Capture. Each sender row's existing action cell gains an "add to filter" button; a select mode swaps in checkboxes with select-all-on-current-search. Bulk seed: "Add all hidden senders (N)" — the hidden list is already "senders I want gone"; this is its payoff. Rows already exported show a badge.
2. Hide-nudge. Hiding a sender shows a small inline toast: "Hidden here. Also filter in Gmail?" with add/dismiss and a don't-ask-again flag. Non-modal, auto-dismissing.
3. Builder. A persistent "Filter draft (N)" pill opens a panel: senders grouped by preset, per-group label input (datalist of prior labels, Parent/Child hint), domain roll-up cards with the also-matches enumeration, running volume total.
4. Review. Rule cards (expandable sender lists, action badges, historical volume, chunk counts), skipped-address list with reasons, roll-up totals, and the exact serialized artifact text in a preview pane (all rendered via textContent).
5. Export & handoff. Primary Download Gmail filter XML, secondary Download gmailctl config (standalone/snippet toggle). Post-download: exact import instructions (Settings → See all settings → Filters and Blocked Addresses → Import filters → choose file → optionally tick "Apply new filters to matching conversations" for retroactive cleanup → Create filters), honest notes (re-importing overlapping files duplicates filters; prefer "new senders only" on later exports), a copyable equivalent Gmail search string for manual retro-sweeps, and an optional "also hide these senders in stats?" bridge.
6. Demo. Everything works on GitHub Pages against summary.json; the existing synthetic-data banner suffices.

Security

• Threat model unchanged: hostile same-browser pages and hostile sender strings; no remote users; loopback bind, Host guard, CSP default-src 'self', headers middleware all untouched.
• XML injection (the one genuinely new risk): a string-concatenated serializer would let a crafted "sender" close an attribute and inject properties — e.g. forwardTo (mail exfiltration) or shouldTrash into a filter the user believed was label-only, inside a file they then upload to Gmail. Two independent guards: the address-shape validator excludes anything containing quotes/angle-brackets/whitespace/braces, and serialization goes exclusively through DOM APIs + XMLSerializer. Fixture tests include senders containing <, &, ", ]]>.
• Query-operator smuggling: addresses passing the validator cannot contain spaces, braces, or quotes, so no gmailctl-style quoting of arbitrary strings is needed — we never emit arbitrary strings.
• jsonnet path is injection-proof by construction (JSON.stringify); the only non-JSON bytes are a fixed header comment.
• DOM: all new dynamic text via textContent only, per the audited house rule. Blob downloads are a download navigation, not a CSP-governed subresource fetch; no CSP change.
• No mutation anywhere: nothing for the CSRF stack to cover; read-only DB pool untouched; DDL remains ingester-owned. Sender data never leaves the machine — filters reach Google only when the user personally uploads the file in Gmail's own UI.
• User-trust: the review pane shows the exact criteria/actions before download, and Gmail's import UI provides a second per-filter checkbox review.

APP user vs. non-APP user

APP-enrolled (primary): the complete loop with zero OAuth and zero extra software — import Takeout → see top senders → hide/select → review → download XML → one manual import in Gmail Settings, with retroactive application via Gmail's own "apply to matching conversations" checkbox. On the next Takeout cycle, filteredSenders badges + "new senders only" make follow-up exports incremental. This column gets 100% of the committed feature.

OAuth-capable: everything above, plus committed in-app apply (F4): an Apply via Gmail API action on the export step spawns filters apply as a supervised run — incremental consent for gmail.settings.basic + gmail.labels through the existing awaiting_auth flow, pre-listing existing filters so re-applies are idempotent (created/skipped reported on the run row), creating labels as needed, strictly additive (deletion stays out of scope). An APP-enrolled account that tries it gets the familiar policy_enforced failure kind steering back to the XML path — same pattern as the scanner. The jsonnet artifact remains for users who prefer gmailctl's full diff/apply lifecycle, with the snippet/merge variant and the reconcile warning. The viewer process itself still never holds a write scope.

Incremental delivery plan

One issue → issue-N branch → one PR per phase, per #26 precedent:

• F1 — capture + XML export (ships standalone value; the entire APP-user story): per-row add + select mode, hidden-senders seeding, Archive+label and Mute presets, address validator, group/chunk consolidation, DOM-based XML serializer, Blob download, review with volume totals, handoff instructions, filteredSenders recording. Pure web/ change; demo-safe by construction. Tests: golden XML fixtures (authored fresh; shape verified against gmailctl's exporter output for the same rules), hostile-sender escaping cases, validator cases.
• F2 — bulk & intent polish: hide-nudge toast, remaining presets incl. confirmation-gated Trash and never-spam toggle, "new senders only" delta scope, badges + clear affordance, draft persistence/backup, copyable retro-sweep search string, post-export hide bridge.
• F3 — consolidation + gmailctl artifact: domain roll-up with freemail blocklist and also-matches enumeration, chunk-count display, jsonnet download (standalone + snippet), README section documenting the gmailctl workflow, apply-reconcile warning, and the gmailctl export cross-check.
• F4 — supervised API apply (committed; maintainer decision 2026-07-24): filters apply <plan> as a new ingester-binary subcommand per the Phase C model — flock, run row, heartbeat, awaiting_auth consent for gmail.settings.basic + gmail.labels, spawned behind check_csrf from an Apply button on the export step. Additive-only with idempotent re-apply (list-then-skip existing); policy_enforced steers to the XML path. The one apply-side open question: whether the plan artifact is the XML file itself or a small JSON plan both exporters share — decide at F4 implementation time.

Open questions

1. Label auto-creation on XML import: does Gmail's importer still create labels named in <apps:property name="label"> that don't exist? Believed yes; must be verified on a real account before F1's instruction copy promises it (fallback: "create the label first").
2. OR form inside the from value: brace group {a@x.com b@y.com} (gmailctl's compiled form) vs a@x.com OR b@y.com — pick one after an empirical import test in F1; both are believed equivalent.
3. Domain rule spelling: y.com vs @y.com in the from field — verify against a live account before F3 finalizes roll-up output.
4. Chunk size 15 vs computing against the ~1500-char budget: start with the fixed margin; revisit only if filter counts annoy.
5. JS test harness: the repo has none — is a dev-only Node script over the pure generator functions acceptable?
6. Should "add all hidden senders" auto-seed on entering filter mode (making hide → filter two clicks), or stay an explicit button?
7. List-ID capture at ingest: list:-based rules beat from: for newsletters (survive sender rotation) but require an ingester-owned schema change — worth queueing as its own future issue?
8. Draft scoping: localStorage is origin-scoped, not DB-scoped; should the draft record the DB identity from /api/status and warn on mismatch, or is that overkill for one user?

[zbuc]

Amendment per maintainer feedback: the Advanced-Protection constraint describes the maintainer's account, not the user base — so in-app API apply for OAuth-capable users is promoted from "deferred indefinitely" to committed F4 scope. Shape unchanged from what was recorded: a supervised filters apply ingester subcommand per the Phase C model (own OAuth with incremental consent, run rows, CSRF-gated spawn), strictly additive with idempotent re-apply — never a reconcile, so no filter-deletion hazard. XML export remains the universal baseline and the only path for APP accounts; policy_enforced failures steer there, mirroring the scanner.