Stamp the secret scanner so old sessions get checked (#335)

CLAUDE.md

@@ -135,7 +135,7 @@ inventory so the surfaces don't drift apart. The `docs/contributing/` guides (an
 
 The pipeline is a one-way data flow, and `src/` is laid out by stage (full map in
 `docs/internals/architecture.md`): **`src/indexing/`** (the pipeline — discover, parse, reconcile,
-materialize, plus the interpret drain), **`src/store/`** (the `argus.db` layer + its parse→store
+materialize, plus the interpret and secret-scan drains), **`src/store/`** (the `argus.db` layer + its parse→store
 contract), **`src/reporting/`** (per-session and plugin aggregation), and **`src/api/`** (the serve
 layer). Cross-cutting modules (`types.ts`, `config.ts`, `paths.ts`, `pricing.ts`, `tool-categories.ts`,
 **`src/llm/`** [the shared LLM access layer], **`secrets.ts`** [BYO API-key storage]) and the

docs/internals/architecture.md

@@ -120,7 +120,10 @@ the "save the answer" step that makes reads cheap and reconcile-free.
 Materialize is also where the **secret scan** (#327) lands: the pipeline runs a deterministic regex
 pass over each session's in-memory prompt/response text (`src/indexing/secret-scan.ts`) and persists
 redacted findings — never secret values — to `resolved_secret_findings`. Because the scan runs on
-in-memory text at write time, it works even when conversation-text retention is off. Full design:
+in-memory text at write time, it works even when conversation-text retention is off. Materialize also
+stamps `secret_scan_version`, so a second pass after indexing (`src/indexing/secret-scan-drain.ts`,
+#335) can catch up sessions the current scanner hasn't seen — the back catalogue after an upgrade, or
+everything after a rules refresh — reading their retained text back from the store. Full design:
 [secret-scanning.md](./secret-scanning.md).
 
 ### What "interpret" means (default-on)

docs/internals/database-schema.md

@@ -271,7 +271,9 @@ timestamps, `message_count`, the `archived` flag (1 = retained but no longer on
 friction signals (`friction_interruptions` / `_rejections` / `_compactions` / `_turns`,
 `last_interruption_ms`; NULL where the source can't observe friction), the local-only UI state columns
 (`is_hidden`, `secret_scan_dismissed` — user state carried forward across re-materializes, never
-synced), and `meta_json` (the authoritative `SessionMeta`). The root of the read model.
+synced), `secret_scan_version` (#335 — which scanner version last scanned this session, NULL for never;
+**not** carried forward, since the re-materialize cascades the findings away), and `meta_json` (the
+authoritative `SessionMeta`). The root of the read model.
 
 ### `resolved_usage`
 One row per usage-bearing turn (the provider's metering grain: Claude `message.usage`, Codex
@@ -353,7 +355,9 @@ at materialize time. **Redacted locators only**: `category` + `interaction_seq`/
 `hint` (first/last few characters), never the secret value. `findings_digest` (the scanner's stable
 hash of the session's whole finding set) is denormalized onto every row so a dismissal — the matching
 digest stored on `resolved_sessions.secret_scan_dismissed` — can be compared in SQL, and lapses when a
-re-scan produces different findings. **Local-only, never synced** (the push path never reads it).
+re-scan produces different findings. Two writers: materialize (from the scan it just ran) and the
+version-stamped rescan drain's `writeSessionSecretFindings` (#335), which catches up sessions the
+current scanner hasn't stamped. **Local-only, never synced** (the push path never reads it).
 PK `(session_id, seq)`, FK → `resolved_sessions`. See [secret-scanning.md](./secret-scanning.md).
 
 ## Tier 3 — freshness & ownership
@@ -385,7 +389,7 @@ unchanged sessions and pick up reindexed ones. PK `(hub_url, client_id, session_
 
 ## Schema version & migrations
 
-`PRAGMA user_version` holds the schema version (currently **24**) and `PRAGMA application_id`
+`PRAGMA user_version` holds the schema version (currently **25**) and `PRAGMA application_id`
 (`0x41524753`, "ARGS") tags the file as an Argus store. Upgrades run forward-only `MIGRATIONS` in
 `src/store/store.ts`, each a `{ to, sql }` step applied in a transaction that bumps `user_version`, so
 a partial upgrade never leaves a half-migrated store. Fresh stores are created from `CREATE_SCHEMA_SQL`

docs/internals/secret-scanning.md

@@ -19,12 +19,16 @@ The scanner engine (`src/indexing/secret-scan.ts`) is pure regex plus per-rule e
 LLM call, no network, no throttle — over rules defined in `src/indexing/secret-scan-rules.ts`
 (adapted from gitleaks). It runs **inline at materialize time**, inside `toMaterializeSessions` in
 `src/indexing/pipeline.ts`, over the reconciled interactions' in-memory prompt/response text. That
-placement was chosen over a drain (like interpret) for two reasons:
+is the primary path, chosen over a drain (like interpret) for two reasons:
 
 - It's cheap enough to run on every materialized session, so there's nothing to throttle.
 - It scans text *in memory*, so it works even when conversation-text retention (`retainText`) is
   off — a drain reading `resolved_interaction_text` back from the store would find nothing there.
 
+Inline scanning only ever covers sessions the incremental pipeline **touched**, though, which leaves
+everything already in the store unscanned. A second path closes that gap: the version-stamped
+**backlog drain** (`src/indexing/secret-scan-drain.ts`). See "Rescanning" below.
+
 Scope is deliberately the same text the Interpret stage reads (`prompt`/`response` chunks), per the
 issue's sketch. Tool *result* text (e.g. the output of `cat .env`) is not covered: it is never
 retained anywhere in the read model today, so there is nothing to scan at write time. Covering it
@@ -70,7 +74,9 @@ are capped per session so a dumped key list can't produce unbounded rows.
 
 ## Storage and dismissal
 
-One table, `resolved_secret_findings` (schema v24), FK-chained to `resolved_sessions` with
+One table, `resolved_secret_findings` (schema v24), plus one column on `resolved_sessions` recording
+which scanner version last looked at each session (`secret_scan_version`, schema v25; see
+"Rescanning"). The findings table is FK-chained to `resolved_sessions` with
 `ON DELETE CASCADE` like every other leaf: re-materializing a session replaces its findings
 wholesale, and retracting a session removes them. Every row carries `findings_digest` — the
 scanner's stable hash of the session's whole finding set — denormalized so SQL can compare the
@@ -127,7 +133,44 @@ derived from transcript text, so agents without transcript access don't get them
 
 ## Rescanning
 
-Findings re-derive on every materialize of a session, and `argus index refresh` re-reads every
-transcript — so a rule-set improvement lands everywhere on the next refresh. There is no scanner
-version stamp in v1 (unlike the interpreter's): the scan is deterministic and free, and a refresh
-is the explicit re-run path.
+Findings re-derive on every materialize of a session, but the incremental pipeline only materializes
+**touched** sessions. Inline scanning alone would therefore mean a user who upgrades gets findings only
+for sessions that happen to change afterwards, leaving their back catalogue silently never scanned. The
+same gap applies to a rule-set refresh: improving `secret-scan-rules.ts` would change nothing for
+sessions already in the store.
+
+`SECRET_SCAN_VERSION` (`src/indexing/secret-scan.ts`) plus a drain closes it, mirroring how
+interpretation is decoupled from the structural index (#153) without the model calls that make that
+drain expensive:
+
+- **The stamp.** `resolved_sessions.secret_scan_version` records the scanner version that last
+  scanned each session; NULL means never scanned. Materialize writes it from the scan it just ran. A
+  materialize that *didn't* scan writes NULL, because the wholesale replace cascades the old findings
+  away and keeping the stamp would claim a scan with nothing to show. Existing rows migrate to NULL,
+  which is exactly the "upgraded, never scanned" state.
+- **Eligibility** (`SECRET_SCAN_ELIGIBLE_SQL` in `store.ts`, alongside `INTERPRETATION_ELIGIBLE_SQL`):
+  the stamp is NULL or below the current version, **and** the session has retained text. Unlike the
+  interpreter's version, this one *is* part of eligibility, since a bump is precisely how a rules
+  refresh reaches already-indexed sessions.
+- **The drain** (`src/indexing/secret-scan-drain.ts`, run from `runIndex` right after the structural
+  index) reads each eligible session's text back via `readSessionInteractions`, rescans it, and
+  replaces its findings wholesale with `writeSessionSecretFindings`. That write always stamps, even for
+  an empty finding set, so a clean session de-queues. There's no rate limiter and no failure cooldown
+  (nothing costs anything, nothing transient to retry), but the pass is bounded and yields every 25
+  sessions so a large backlog can't make `argus run` stop responding.
+
+Two consequences worth keeping in view:
+
+- **Bump the version only when findings can actually change.** Dismissal is anchored to the finding-set
+  digest, so a rescan that finds something different clears the dismissal by design. That's correct for
+  a genuinely different finding set, but it means a gratuitous bump re-warns in bulk. The rule-refresh
+  procedure in `secret-scan-rules.ts` carries this as its last step.
+- **Text retention bounds what the drain can reach.** With `retainText` off (#120) a session's text only
+  ever existed in memory during materialize, so there is nothing to rescan and eligibility excludes
+  those sessions. Interpretation has the same limitation, so this is a known shape rather than a new
+  one. The state is reported rather than silent: `secretScanProgress` counts them separately, and
+  `argus status` says how many sessions can't be checked without re-reading their transcripts, which is
+  what `argus index refresh` does.
+
+`argus index refresh` remains the explicit "rescan everything now" path, and the only one that covers
+sessions whose text wasn't kept. It is no longer the *only* way a rules improvement lands, though.

src/cli.ts

@@ -4,6 +4,7 @@ import { defineCommand, runMain, showUsage } from "citty";
 import type { ArgsDef, CommandContext, ParsedArgs } from "citty";
 import { printBanner } from "./banner.ts";
 import { scanStore } from "./indexing/pipeline.ts";
+import { SECRET_SCAN_VERSION } from "./indexing/secret-scan.ts";
 import { STORE_FILE } from "./paths.ts";
 import { ALL_SOURCES } from "./reporting/dashboard-builder.ts";
 import { startServer } from "./api/serve.ts";
@@ -310,11 +311,15 @@ async function runStatus(): Promise<void> {
   let interpretation:
     | { interpreted: number; pending: number; outdated: number }
     | undefined;
+  let secretScan:
+    | { scanned: number; pending: number; unscannable: number }
+    | undefined;
   try {
     const store = await openStore();
     try {
       counts = await store.resolvedSessionCounts();
       interpretation = await store.interpretationProgress();
+      secretScan = await store.secretScanProgress(SECRET_SCAN_VERSION);
     } finally {
       await store.close();
     }
@@ -376,6 +381,30 @@ async function runStatus(): Promise<void> {
         `(${interpretation.pending} waiting${outdated}).`,
     );
   }
+  // Secret-scan backlog progress (#335). The waiting count only appears while there is a backlog —
+  // after an upgrade, or a rules refresh — since every newly indexed session is checked as it's
+  // written. Sessions whose conversation text wasn't kept can't be checked from the store at all, so
+  // they get their own line with the one thing that fixes it.
+  if (secretScan && secretScan.scanned > 0) {
+    const waiting = secretScan.pending
+      ? ` (${secretScan.pending} waiting)`
+      : "";
+    printResultLine(
+      `Checked ${secretScan.scanned} session${secretScan.scanned === 1 ? "" : "s"} for exposed credentials${waiting}.`,
+    );
+  } else if (secretScan?.pending) {
+    const n = secretScan.pending;
+    printResultLine(
+      `${n} session${n === 1 ? "" : "s"} waiting to be checked for exposed credentials.`,
+    );
+  }
+  if (secretScan?.unscannable) {
+    const n = secretScan.unscannable;
+    printResultLine(
+      `${n} session${n === 1 ? "" : "s"} can't be checked for exposed credentials without re-reading ` +
+        `${n === 1 ? "its transcript" : "their transcripts"} · run \`argus index refresh\``,
+    );
+  }
   if (pending)
     printResultLine("Run `argus index` to pick up new and changed sessions.");
 }

src/index-ops.ts

@@ -4,6 +4,7 @@ import { createInterface } from "node:readline";
 import { sourcesFor } from "./reporting/dashboard-builder.ts";
 import { syncStatsSummary, reindexSession } from "./indexing/pipeline.ts";
 import { runInterpretationDrain, sessionInterpretationActive } from "./indexing/interpret/index.ts";
+import { runSecretScanDrain } from "./indexing/secret-scan-drain.ts";
 import type { RepeatCollapser } from "./backoff.ts";
 import { openSessionStore } from "./store/session-store.ts";
 import { openStore, rebuildStore } from "./store/store.ts";
@@ -42,9 +43,9 @@ export async function runIndex(
   extractTasks?: boolean,
   debug = false,
   retainText?: boolean,
-  // Persisted across watch ticks by the caller (watchIndex) so the drain's throttle-pause / failure
-  // lines collapse instead of repeating every interval. Omitted for a one-shot `argus index`.
-  interpretCollapser?: RepeatCollapser,
+  // Persisted across watch ticks by the caller (watchIndex) so the post-index drains' throttle-pause /
+  // failure lines collapse instead of repeating every interval. Omitted for a one-shot `argus index`.
+  collapser?: RepeatCollapser,
 ): Promise<void> {
   // Read argus.json once and thread it into both resolvers (avoid a double parse per pass / watch tick).
   const config = loadConfig();
@@ -64,17 +65,20 @@ export async function runIndex(
   } finally {
     await store.close();
   }
-  // Decoupled, throttled interpretation (#153): after the structural index brings the store current,
-  // interpret a bounded, rate-limited batch of eligible sessions, reading retained text back from the
-  // store. A fresh handle (the pipeline closed its own) and strictly after indexing, so there's never a
-  // concurrent writer. No-op when task extraction is disabled — and we skip opening the store entirely.
-  if (sessionInterpretationActive(taskExtraction)) {
-    const store = await openStore();
-    try {
-      await runInterpretationDrain(store, taskExtraction, log, interpretCollapser);
-    } finally {
-      await store.close();
+  // Two decoupled passes run after the structural index brings the store current, both reading their
+  // inputs back from the store: the secret-scan backlog (#335) and throttled interpretation (#153).
+  // A fresh handle (the pipeline closed its own) and strictly after indexing, so there's never a
+  // concurrent writer. Both are silent no-ops when there's nothing eligible, which is the steady state.
+  const drainStore = await openStore();
+  try {
+    // Cheap and unconditional: regex over text already in the store, no setting to honor. Runs first
+    // so a just-upgraded user gets their credential warnings without waiting on the model calls below.
+    await runSecretScanDrain(drainStore, log, collapser);
+    if (sessionInterpretationActive(taskExtraction)) {
+      await runInterpretationDrain(drainStore, taskExtraction, log, collapser);
     }
+  } finally {
+    await drainStore.close();
   }
 }
 

src/indexing/pipeline.ts

@@ -34,7 +34,7 @@ import { NATIVE_PRODUCERS, nativeProducerForSource } from "./parse/producers/ind
 import type { AgentSource, MessageRecord, ParseResult } from "../types.ts";
 import type { TaskFact } from "../store/store-contract.ts";
 import { interpretSession, sessionInterpretationActive } from "./interpret/index.ts";
-import { scanSessionForSecrets, secretFindingsDigest } from "./secret-scan.ts";
+import { SECRET_SCAN_VERSION, scanSessionForSecrets, secretFindingsDigest } from "./secret-scan.ts";
 import type { ResolvedSessionInterpretation } from "../config.ts";
 
 export interface SyncStats {
@@ -322,16 +322,20 @@ function toMaterializeSessions(output: ReconcileResult): MaterializeSession[] {
     const interactions = interactionsBySession.get(sid) ?? [];
     // Secret scan (#327): regex-cheap and deterministic, so it runs inline here for every
     // materialized session — no throttle, no LLM call. Scanning the in-memory interaction text
-    // (not the store) means findings exist even when conversation-text retention is off.
+    // (not the store) means findings exist even when conversation-text retention is off. The result
+    // is always attached, even with zero findings, so materialize stamps SECRET_SCAN_VERSION (#335)
+    // and the rescan drain knows this session has been seen by the current scanner.
     const findings = scanSessionForSecrets({ interactions });
     sessions.push({
       meta,
       messages: messagesBySession.get(sid) ?? [],
       tasks: output.tasksBySession.get(sid) ?? [],
       interactions,
-      ...(findings.length
-        ? { secretFindings: { digest: secretFindingsDigest(findings), findings } }
-        : {}),
+      secretFindings: {
+        version: SECRET_SCAN_VERSION,
+        digest: secretFindingsDigest(findings),
+        findings,
+      },
     });
   }
   return sessions;