Skip to content

feat: configurable ingest-time privacy controls for hook payloads (#148) #166

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
linkvapeluckyman wants to merge 3 commits into
base: master
Choose a base branch
from
Open

feat: configurable ingest-time privacy controls for hook payloads (#148) #166

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
eb49f45
feat(server,client): configurable ingest-time privacy controls for ho…
linkvapeluckyman Jun 10, 2026
3961ad2
fix: address PR #166 review feedback
linkvapeluckyman Jun 10, 2026
a9fe6c6
Merge remote-tracking branch 'upstream/master' into feat/privacy-cont…
linkvapeluckyman Jun 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 15 additions & 2 deletions ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,12 +294,17 @@ graph TD
PRICING[routes/pricing.js<br/>Cost calculation + pricing CRUD]
SETTINGS[routes/settings.js<br/>System info + data management]
WORKFLOWS[routes/workflows.js<br/>Workflow visualizations]
PRIVACYR[routes/privacy.js<br/>Privacy policy + preview]
PRIVACY[lib/privacy.js<br/>Ingest-time sanitizer]

INDEX --> DB
INDEX --> WS
INDEX --> HOOKS & SESSIONS & AGENTS & EVENTS & STATS & PRICING & SETTINGS & WORKFLOWS
INDEX --> HOOKS & SESSIONS & AGENTS & EVENTS & STATS & PRICING & SETTINGS & WORKFLOWS & PRIVACYR

HOOKS --> DB & WS & TC
HOOKS --> PRIVACY
PRIVACYR --> PRIVACY
PRIVACY --> DB
SETTINGS --> DB & TC
INDEX --> TC
SESSIONS --> DB & WS
Expand Down Expand Up @@ -330,7 +335,9 @@ graph TD
| `routes/stats.js` | Single aggregate query returning total/active counts + status distributions |
| `routes/analytics.js` | Extended analytics — token totals, tool usage counts, daily event/session trends, agent type distribution. The client-side analytics heatmap grid is aligned to a Sunday start for correct day-of-week positioning |
| `routes/pricing.js` | Model pricing CRUD (list/upsert/delete) and per-session / global cost calculation with pattern-based model matching. Cost is computed per token bucket — keyed by (model, speed, inference_geo, service_tier) — applying fast-mode premium, US data-residency (1.1x), and Batch (0.5x) modifiers, the 5m/1h cache-write split, plus server-tool surcharges (web search $10/1k; code execution estimated by container-time with the monthly free-hours allowance; web fetch free). Feature rates + modifier math live in `lib/pricing-constants.js`; usage normalization in `lib/token-usage.js` |
| `routes/settings.js` | System info (DB size, hook status, server uptime, transcript cache stats), data export as JSON, session cleanup (abandon stale, purge old), clear all data, reset pricing, reinstall hooks |
| `routes/settings.js` | System info (DB size, hook status, server uptime, transcript cache stats), data export as JSON (includes the active `privacy_policy` so a restore can re-apply the same rules; events export as stored — post-policy), session cleanup (abandon stale, purge old), clear all data, reset pricing, reinstall hooks |
| `routes/privacy.js` | HTTP surface for ingest-time privacy controls: `GET /api/privacy` (active policy + action/match-type vocabulary + defaults), `PUT /api/privacy` (replace policy, validated server-side — invalid regexes, unknown actions, and `drop_field` on value rules are rejected with structured 400s), and `POST /api/privacy/preview` (before/after transformation of a sample payload + optional draft policy, persisting nothing) |
| `lib/privacy.js` | Ingest-time payload sanitizer. The policy (persisted as JSON in `app_settings`, cached in memory, invalidated on PUT) holds six built-in detector toggles — `secret_keys` (key-name match, same regex family as the Config Explorer redaction), `bearer_tokens`, `api_key_formats`, `private_key_blocks`, plus opt-in `email_addresses` and `home_paths` — a `default_action` (`mask` or stable truncated SHA-256 `hash`), and up to 100 custom rules (`match_type: key\|value` regex with `mask` / `hash` / `drop_field` / `drop_event_payload` actions; `drop_event_payload` reduces the payload to a metadata-only stub preserving `session_id`/`tool_name`/`transcript_path`/`cwd`). `sanitizeEventData()` deep-walks payloads (depth cap 32, node cap 20k; strings are fully scanned up to 2 MB — above the express body cap — and anything larger is masked wholesale rather than skipped) and stamps a `_privacy` counters block only when something was redacted — clean payloads are stored byte-identical. `sanitizeText()` covers event summaries and agent task strings. Called from `routes/hooks.js` at every event-insert site **after** routing decisions ran on the raw payload; fail-safe — a sanitizer crash stores a conservative metadata-only stub (never raw data) and never breaks ingestion. Import/re-import intentionally bypasses the policy (records are written as found on disk); the Settings panel and docs call this out explicitly |
| `routes/workflows.js` | Aggregate workflow visualization data (agent orchestration graphs, tool transition flows, collaboration networks, workflow pattern detection, model delegation, error propagation, concurrency timelines, session complexity metrics, compaction impact). Accepts `?status=active\|completed` query parameter to filter all data by session status. Per-session drill-in endpoint with agent tree, tool timeline, and event details |
| `lib/transcript-cache.js` | Stat-based JSONL transcript cache with incremental byte-offset reads. Shared between `hooks.js` (token extraction on every event) and the periodic compaction scanner (`index.js`). Extracts tokens, compaction entries, API errors (`isApiErrorMessage` + raw error responses), turn durations (`system` subtype `turn_duration`), thinking block counts, and usage extras (service_tier, speed, inference_geo). Uses `(path, mtime, size)` cache key — unchanged files return cached results instantly, grown files only parse new bytes, shrunk files (compaction) trigger full re-read. Each cache entry stores **only** `{mtimeMs, size, bytesRead, result}` — the previous shape that duplicated every growable array at both the top level and inside `result` is gone, halving steady-state memory per entry. Per-entry growable arrays (`turnDurations`, `errors`, `compaction.entries`, `usageExtras.*`) are bounded to `TRANSCRIPT_CACHE_MAX_ARRAY_LEN` (default `1000`, tail-kept) — older items remain in the `events` table thanks to hook dedup, so the cap only affects the in-memory view. Trimming runs both during parse (when an array reaches `2 * MAX_ARRAY_LEN`, amortized O(N)) and at finalize, so even a fresh full-file parse on a multi-day session cannot accumulate an unbounded transient before returning. **Chunked sync byte-stream reader** (`_streamRange`, 4 MiB chunks split on `0x0A` bytes — safe across UTF-8 multibyte sequences — with a growable per-line byte buffer capped at 64 MiB) replaces the previous `readFileSync("utf8")` so transcripts larger than V8's max JS string length (~512 MiB on 64-bit Node 20) parse without aborting Node with `FATAL ERROR: v8::ToLocalChecked Empty MaybeLocal`. Both full and incremental reads share the same line-level state machine (`_initParseState` / `_consumeLine` / `_finalizeState`). LRU eviction caps at 200 entries. Entries evicted on SessionEnd and abandoned session cleanup |
| `scripts/import-history.js` | Batch history importer used by (a) server startup auto-import, (b) the `/api/import/*` routes, (c) the `import-history` CLI, and (d) live `SubagentStop` ingestion via the exported `scanAndImportSubagents(dbModule, sessionId, transcriptPath)`. Exposes `importAllSessions(dbModule)` for the default `~/.claude/projects` tree and the generalized `importFromDirectory(dbModule, rootDir, {onProgress})` which walks any directory recursively, classifies each `.jsonl` as session vs subagent (with `findSessionSubagents` probing both `<proj>/<sid>/subagents/*` and `<proj>/subagents/<sid>/*` layouts), and funnels everything through the shared `parseSessionFile` + `importSession` pipeline. `parseSubagentFile` extracts ordered `toolEvents` (tool_use + tool_result paired by `tool_use_id`) so `importSubagentFromJsonl` can emit per-tool `PreToolUse` + `PostToolUse` rows under each subagent's own `agent_id`. The importer dedups against live hook-created subagent rows via `findLiveSubagentForJsonl` (session + subagent_type + start-time within 30 s) so backfill never produces parallel `<sid>-jsonl-*` rows. **Re-import is fully incremental**: for each existing session a per-event-type high-water mark (`MAX(created_at) GROUP BY event_type`) is read up-front and only JSONL entries with `ts > cutoff[type]` are inserted for Stop / PostToolUse / TurnDuration / ToolError — so long-running sessions whose transcripts grow across multiple days continue to receive new events on every re-run instead of being blocked by the old "if zero of type X then dump all" check. `sessions.ended_at` is rolled forward to the JSONL's last activity when it surpasses the stored value, and `metadata.user_messages` / `assistant_messages` / `turn_count` are refreshed on every pass. Other idempotency keys are unchanged: `data LIKE '%"tool_use_id":"X"%'` skips any tool event already inserted, compaction agents/events dedup by uuid, API errors dedup by summary, and `baseline_*` columns preserve pre-compaction token totals. Token totals, per-model cost, compactions, subagents, tool events, API errors, and turn durations are identical to live ingestion. Creates `APIError`, `TurnDuration`, and `ToolError` event types during import; subagent tool events carry `imported: true, source: "subagent_jsonl"` in their data payload so analytics can distinguish backfilled rows when needed |
Expand Down Expand Up @@ -783,6 +790,12 @@ erDiagram
TEXT auth "Auth secret"
TEXT created_at "ISO 8601"
}

app_settings {
TEXT key PK "Setting name e.g. privacy_policy"
TEXT value "JSON string"
TEXT updated_at "ISO 8601"
}
```

### Indexes
Expand Down
9 changes: 9 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -265,6 +265,7 @@ The dashboard offers a comprehensive set of features to monitor and analyze your
| **Cost Tracking** | Per-model cost estimation with configurable pricing rules and per-session breakdowns. Compaction-aware token accounting preserves totals across context compressions. Transcript reads are cached with incremental byte-offset updates for efficient token extraction |
| **Transcript Cache** | Real-time extraction from JSONL transcripts: tokens, compactions, API errors (`isApiErrorMessage` entries stored as `APIError` events), turn durations (stored as `TurnDuration` events), thinking block counts, and usage extras (service_tier, speed, inference_geo). Per-entry growable arrays are tail-capped at `TRANSCRIPT_CACHE_MAX_ARRAY_LEN` (default `1000`, configurable) — both during parse and at finalize — so even a session that runs for days cannot grow a single cache entry without bound. Each entry stores only `{mtimeMs, size, bytesRead, result}`, so there's no shadow copy of the same data at both the top level and inside `result`. Session metadata is enriched with these fields in real-time |
| **Notifications** | Full Web Push (VAPID) pipeline for reliable delivery. Arrive even when the tab is backgrounded or the browser is closed. Explicitly configured for macOS audio support. Configurable per-event toggles with subscription management |
| **Privacy Controls** | Configurable ingest-time privacy policy that redacts, hashes, or drops sensitive data from hook payloads **before** they are written to SQLite or broadcast over WebSocket. Six built-in detectors — secret-named keys (`token`/`secret`/`password`/`api_key`/`auth`/`credential`), `Bearer` tokens, common API-key formats (`sk-ant-…`, `ghp_…`, `AKIA…`, `xox…`, `AIza…`), private-key blocks, and opt-in email-address and home-directory-path detectors — plus custom regex rules matching key names or value text with `mask`, `hash` (stable truncated SHA-256 for correlation), `drop_field`, or `drop_event_payload` (metadata-only stub) actions. Conservative default: obvious secrets are masked out of the box. Redacted events carry a `_privacy` metadata block (rule count, masked/hashed/dropped counters) without exposing original values, and clean payloads are stored byte-identical so existing analytics, filters, and cost views are unaffected. Settings panel with master toggle, per-detector toggles, rule CRUD, and a live **before/after preview** (`POST /api/privacy/preview`) that never persists the sample. Fail-safe: a sanitizer error degrades to dropping the payload — never storing raw data, never failing hook ingestion. Applies to live ingest only; import/re-import writes transcripts as found (explicitly called out in the UI) |
| **Update Notifier** | Server periodically runs a non-blocking `git fetch` and compares the local checkout to `origin/master`/`origin/main`/`origin/HEAD`. When upstream is ahead, the UI surfaces a modal with the exact `git pull && npm run setup` command and a one-click **Copy** button; the Sidebar gets a persistent "Check for updates" button with live badge. The dashboard never pulls or restarts itself — the user runs the command in a terminal — so the mechanism cannot break dev sessions, pm2/systemd/Docker supervision, or leave orphaned processes |
| **Settings** | System info, hook status, model pricing management, notification preferences, data export, session cleanup. The Model Pricing section exposes an info popover (the `i` icon next to the title) explaining how rule lookup works (first matching pattern wins), the SQL-style `%` wildcard syntax with concrete examples (`claude-opus-4-7%`, `claude-%-haiku`, exact ids), and that prices must be updated manually when Anthropic publishes new rates — already-stored sessions keep the price applied at ingest time. The CLAUDE_HOME box and Import History panel are fully i18n-driven across en/vi/zh |
| **MCP Server (Local)** | Enterprise-grade local MCP server in `mcp/` with three transport modes (stdio, HTTP+SSE, interactive REPL), 25 typed tools across 6 domains, strict input schemas, retry/backoff, localhost-only API enforcement, and tiered mutation/destructive safety gates. HTTP mode serves Streamable HTTP (2025-11-25) and legacy SSE (2024-11-05) on configurable port. REPL mode provides tab-completed interactive tool invocation with colored output |
Expand Down Expand Up @@ -901,6 +902,14 @@ The OpenAPI document is generated from `server/openapi.js`, and Swagger UI is se
| `GET` | `/api/workflows` | Aggregate workflow data (orchestration, tools, patterns). Optional `?status=active\|completed` query param filters all 11 data sections by session status |
| `GET` | `/api/workflows/session/:id` | Per-session drill-in (agent tree, tool timeline, events) |

### Privacy

| Method | Path | Description |
| ------ | ---------------------- | ------------------------------------------------------------------------ |
| `GET` | `/api/privacy` | Active ingest-time privacy policy + action/match-type vocabulary |
| `PUT` | `/api/privacy` | Replace the policy (validated server-side; invalid regexes are rejected) |
| `POST` | `/api/privacy/preview` | Before/after preview of a sample payload — nothing is persisted |

### Settings

| Method | Path | Description |
Expand Down
Loading
Loading