Skip to content

Stamp the secret scanner so old sessions get checked (#335) - #339

Merged
mando merged 2 commits into
mando-issue-336-timeline-secret-markersfrom
mando-issue-335-secret-scan-version
Aug 18, 2026
Merged

Stamp the secret scanner so old sessions get checked (#335)#339
mando merged 2 commits into
mando-issue-336-timeline-secret-markersfrom
mando-issue-335-secret-scan-version

Conversation

@mando

@mando mando commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Closes #335. Stacked on #337 (which is stacked on #332) — rebase the base to main once those land.

The gap

Secret scanning (#327) runs inline at materialize, and the incremental pipeline only materializes touched sessions. Nothing about upgrading to a build that has the scanner marked old sessions as needing a look, so a user who upgraded got findings only for sessions that happened to change afterwards — their back catalogue silently never scanned. Same for a rule-set refresh: improving secret-scan-rules.ts changed nothing for sessions already indexed. The only remedies were a full argus index refresh (minutes on a large store, and you had to know to do it) or waiting for each session to change on its own.

What this does

  • SECRET_SCAN_VERSION (src/indexing/secret-scan.ts) — the scanner's implementation version. Unlike the interpreter's version, this one is part of eligibility: a bump is precisely how a rules refresh reaches sessions already in the store.
  • Schema v25resolved_sessions.secret_scan_version, NULL for never scanned. Existing rows migrate to NULL, which is exactly the "upgraded, never scanned" state this issue is about. Materialize stamps it from the scan it just ran, and deliberately does not carry it forward when a materialize didn't scan: the wholesale replace cascades the findings away, so keeping the stamp would claim a scan with nothing to show. NULL hands the session back to the drain, which self-heals.
  • SECRET_SCAN_ELIGIBLE_SQL in store.ts, alongside INTERPRETATION_ELIGIBLE_SQL and shared by the drain query and the argus status counts so "waiting" can't desync from the work. Plus readPendingSecretScanSessions, writeSessionSecretFindings (always stamps, even for an empty finding set, so a clean session de-queues; leaves secret_scan_dismissed alone so the digest comparison still governs), and secretScanProgress.
  • The drain (src/indexing/secret-scan-drain.ts), run from runIndex right after the structural index and before the interpretation drain. Shaped like the interpretation drain minus what made that one expensive: no model call, so no rate limiter. 500 sessions per pass, yielding every 25 so a large backlog can't make argus run stop responding, with the same self-recovering failure cooldown.
  • Rules-refresh procedure — bumping the constant is now step 5 in secret-scan-rules.ts, with the note that a bump re-warns in bulk (dismissal is anchored to the finding-set digest), so it's only for changes that can actually change findings.

The retainText constraint

Option 1 from the issue: sessions indexed with text retention off (#120) have no resolved_interaction_text to read, so a stamp-driven rescan is structurally impossible for them (interpretation has the same limitation). Eligibility excludes them, and argus status reports the state rather than leaving it silent:

Checked 1204 sessions for exposed credentials (340 waiting).
2 sessions can't be checked for exposed credentials without re-reading their transcripts · run `argus index refresh`

argus index refresh remains the explicit "rescan everything now" path, and the only one that covers those sessions. It is no longer the only way a rules improvement lands.

Test plan

  • bun run typecheck clean
  • bun test — 800 pass / 0 fail
  • New test/secret-scan-drain.test.ts (10 tests): stamping at materialize, eligibility, newest-first ordering, a version bump re-queuing scanned sessions, the drain scanning from retained text and de-queuing, clean sessions still stamped, dismissal surviving an identical rescan and lapsing on a changed one, wholesale replace, and a re-materialize without a scan handing the session back
  • test/store.test.ts — v24 → v25 migration test: existing sessions migrate to "never scanned" and show up as pending
  • test/cli.test.ts — pins the two new argus status lines
  • End to end against a scratch store seeded with an unstamped session containing a synthesized AWS key: the drain found it and status reported the check

Note for reviewers

The v25 migration means a store touched by this branch can't be read by an older build.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TTu8aFTkXgk2SVjkAw8rWn

@mando
mando marked this pull request as ready for review August 17, 2026 19:48
@mando

mando commented Aug 17, 2026

Copy link
Copy Markdown
Contributor Author

Pushed follow-up commit 2e295278 to address the review finding.

  • writeSessionSecretFindings() now hardens the SQLite database, WAL, and SHM files with the existing 0600 policy while the store remains open.
  • Added a regression test that loosens file permissions before the drain write and verifies they are restored.

Verification: bun test (801 passed), targeted secret-scan drain tests (11 passed), root typecheck, and web typecheck.

mando added 2 commits August 18, 2026 15:50
Inline scanning only covers sessions the incremental pipeline touched, so
upgrading to a build with the scanner left a user's back catalogue silently
unscanned, and a rules refresh changed nothing for sessions already indexed.

Record which scanner version last looked at each session
(resolved_sessions.secret_scan_version, schema v25; existing rows migrate to
NULL) and drain the backlog after each index pass: sessions the current version
hasn't stamped get their retained text read back, rescanned, and stamped. No
model call, so no rate limiter, but the pass is bounded and yields so `argus run`
stays responsive. Bumping SECRET_SCAN_VERSION is now step 5 of the gitleaks
rules-refresh procedure.

Sessions indexed with text retention off can't be reached from the store at all.
`argus status` says how many, and that re-reading their transcripts is what
fixes it, rather than leaving the gap silent.
@mando
mando force-pushed the mando-issue-335-secret-scan-version branch from 2e29527 to e4b7f14 Compare August 18, 2026 20:50
@mando
mando merged commit c3cff70 into main Aug 18, 2026
2 checks passed
@mando
mando deleted the mando-issue-335-secret-scan-version branch August 18, 2026 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stamp the secret scanner so already-indexed sessions get scanned incrementally

1 participant