an interrupted incremental cass index keeps its committed conversations but nothing else

[bitjson]

(Fable 5:)

Title: main (0.7.0): an interrupted incremental cass index keeps its committed conversations but nothing else — no signal handling, scan watermarks only written at connector/run end (and withheld entirely while any session is active), no per-file ingest ledger (every file is re-parsed on resume), and the unbackfilled WAL is left for the next open to replay

Summary

Long incremental runs (hours on big archives) cannot be stopped and resumed cheaply, and on a busy host they never get cheaper. After SIGTERM mid-ingest:

• Kept: every conversation whose batch had committed (fsqlite transaction per chunk). No duplicates on resume.
• Lost: the scan progress. The per-connector last_scan_ts:connector:<name> is written only when a connector's stream reaches Done (src/indexer/mod.rs:12664-12670 at 557bb53) and the legacy global last_scan_ts only at run end (:10926, :11006), and there is no per-file (path, mtime, size) ledger. So the next run re-scans and re-parses every file and only skips the DB insert after parsing (message fingerprint dedupe). Measured with 40 × 25 MB synthetic Codex rollouts (97k messages): from-scratch 118 s (scan 97 s); kill at 12/40 and resume: 121 s (scan 95 s) — nothing saved; kill at 22/40 on a pre-populated index and resume: 53 s (scan 42 s, 18 inserted) — every file re-read. On a 37 GB / 1,325-file Codex archive a resume re-reads all 37 GB before it reaches the files it still needs.
• Left behind: an unbackfilled WAL (db 315 KB / wal 298 MB after the kill in the fresh-dir case; db 265 MB / wal 295 MB in the other). The next cass invocation must replay it at open (fsqlite reads and checksums every frame — tens of seconds per 500 MB, minutes for the multi-GB WALs a long run accumulates, frankensqlite#385).
• There is deliberately no SIGTERM/SIGINT handler (src/lib.rs:49991-49999 keeps SIGINT on the OS default), so nothing — not even a connector watermark write or a passive checkpoint — runs on termination.

Distinct from #380 (closed, 430ec8d): that was the lexical rebuild checkpoint being written to a path the next run never read; this is the scan/ingest side, and #380's fix does not apply to it (verified: after the kill, status was stale/ready with rebuilding:false, and the next run chose incremental_inline, never a full rebuild).

Two things that were expected to be lost were not: no deferred-authoritative-rebuild state was left behind, and the last conversations committed before the SIGTERM were searchable after the resume in both runs (own-file hits for head-of-file tokens of the final two committed conversations). If a future run shows conversations present in the DB but absent from search after an interruption, that would be the inline-lexical commit window (lexical commits every ~5 s; dedupe on resume adds no docs) — it did not reproduce here.

The same missing ledger makes uninterrupted runs re-parse everything on a busy host

The per-connector watermark is also withheld for the whole run whenever any active-session source was skipped: scan_watermark_preservation_active() (src/indexer/mod.rs:11973) is true as soon as ACTIVE_SESSION_SOURCE_SKIP_OBSERVED is set (an open session file with a writable fd, an advisory lock, or an mtime inside the recent-write window — ActiveSessionSourceFilter, :138-190), and the Done handler then skips set_connector_last_scan_ts for every connector whose scan finishes after that moment (:12664-12666). Connectors that finish before the first skip still advance (a race, not a policy).

Measured on a host where a Codex TUI session is always open (three codex processes each holding their rollout with O_WRONLY|O_APPEND): ten consecutive 30-minute timer runs (cass index --json, incremental) each reported the same 48 conversations / ~6.6k messages ingested (30 omp + 14 claude + 4 grok — every file modified since the last quiet moment), took 8–12 min and peaked at 36–40 GB RSS, while last_scan_ts:connector:{claude,codex,grok,antigravity,pi_agent} stayed at the value from 27 h earlier (gemini/cursor/factory, which finish first, kept advancing). The re-scan window therefore grows without bound for as long as any session stays open, and long-lived TUI sessions themselves are invisible to search until their process exits.

A related code-level observation for the non-default batch mode (CASS_STREAMING_INDEX=0): run_batch_index writes last_scan_ts = scan_start_ts every 10 s during ingest (src/indexer/mod.rs:13345-13352, "so that if the process is killed, the next run does a delta scan"). Because all connectors are parsed before any ingest there, a kill during ingest makes the next run skip files that were parsed but never persisted — the opposite failure. Not measured; noted from the source.

Reproduction (synthetic, no real sessions)

Generate 40 × 25 MB Codex-shaped rollouts with the generator inlined in #425 (gen_rollouts.py), then:

python3 gen_rollouts.py /tmp/cass-synth/gen --files 40 --mb-per-file 25 --seed 7
mkdir -p /tmp/cass-synth/home /tmp/cass-synth/data
cassenv() { env -i PATH=/usr/bin:/bin HOME=/tmp/cass-synth/home USER=$USER CODEX_HOME=/tmp/cass-synth/gen \
  CASS_DATA_DIR=/tmp/cass-synth/data CASS_INDEX_STALL_ABORT_SECS=0 \
  CASS_TRACE_FILE=/tmp/cass-synth/$1.trace CASS_TRACE_FILTER='warn,coding_agent_search=info' "${@:2}"; }

# run 1: SIGTERM the real cass pid after 12 conversations were ingested
( exec env -i PATH=/usr/bin:/bin HOME=/tmp/cass-synth/home USER=$USER CODEX_HOME=/tmp/cass-synth/gen \
    CASS_DATA_DIR=/tmp/cass-synth/data CASS_INDEX_STALL_ABORT_SECS=0 \
    CASS_TRACE_FILE=/tmp/cass-synth/run1.trace CASS_TRACE_FILTER='warn,coding_agent_search=info' \
    cass index --json > /tmp/cass-synth/run1.progress 2>/tmp/cass-synth/run1.err ) &
pid=$!
while kill -0 $pid 2>/dev/null; do
  n=$(grep -c '"streaming_ingest"' /tmp/cass-synth/run1.trace 2>/dev/null || echo 0)
  [ "${n:-0}" -ge 12 ] && { echo "SIGTERM at $n"; kill -TERM $pid; break; }
  sleep 0.5
done
wait $pid; echo "run1 exit=$?"
ls -la /tmp/cass-synth/data | grep -E 'wal$|db$'                         # db tiny, wal ~300 MB
sqlite3 -readonly /tmp/cass-synth/data/agent_search.db \
  "select count(*) from conversations; select key,value from meta where key like 'last_%';"   # 12 rows; no watermark
cassenv st cass status --json | jq -c '{status:.index.status, rebuilding:.index.rebuilding}'

# run 2: resume — compare scan_ms/index_ms with a from-scratch run of the same 40 files
cassenv run2 cass index --json > /dev/null; jq -c 'select(.fields.message=="streaming_indexing_complete")|.fields' /tmp/cass-synth/run2.trace

# watermark freeze: keep one generated rollout open for append in another shell
#   (python3 -c 'import time; f=open("/tmp/cass-synth/gen/sessions/2026/07/01/<file>.jsonl","a"); time.sleep(3600)') &
# then run `cassenv w cass index --json` twice: both runs re-ingest every file, and
#   `select key,value from meta where key like 'last_scan_ts:connector:codex'` never appears/advances.

Observed (Linux x86-64, cass 0.7.0 main a06d389):

run1 exit=143
agent_search.db      315392
agent_search.db-wal  298007872
12                                  <- conversations committed
                                    <- no last_scan_ts rows
{"status":"stale","rebuilding":false}
run2: {"total_conversations":40,"total_messages":97612,"scan_ms":95093,"index_ms":121199}   # from scratch: scan 97243 / index 117982

Suggested fix shape

1. Per-file ingest ledger, written in the same transaction as each committed chunk — e.g. ingested_source_files(connector, source_path, mtime_ms, size_bytes, scan_start_ts); the connector scan skips a file whose (mtime, size) row exists, with the existing file_modified_since mtime filter as the outer gate. This is safe under the current path-sorted, parallel-connector scan (unlike advancing last_scan_ts mid-run, which would skip not-yet-yielded files with mtime < scan_start_ts — exactly the batch-mode bug above), it turns a resume from "re-parse everything" into "stat everything, parse the remainder", and it makes the active-session skip cheap: a skipped file simply has no ledger row yet, so the per-connector watermark can advance for everything else instead of being withheld for the whole run.
2. Advance the per-connector watermark only when that connector's scan is complete and all its batches are committed (already the streaming behaviour) and drop the 10 s mid-ingest set_last_scan_ts in run_batch_index, or gate it on "no pending batch".
3. A termination hook for cass index (SIGTERM/SIGINT → set a cancel flag checked at batch boundaries): finish the in-flight chunk, write the ledger/watermarks, run a passive (or, when single-connection, TRUNCATE) checkpoint so the WAL is backfilled, release the lock, exit non-zero. Even without a signal handler, a --max-minutes/--max-conversations budget that stops cleanly at a batch boundary would give operators a resumable unit of work.
4. Optional: on resume, if observed_tantivy_docs != total_messages for an incremental_inline run, index messages.id > <last id recorded at the previous run's final lexical commit> instead of refusing to refresh the checkpoint (a small .lexical-inline-cursor.json {max_message_id} written after each inline commit would make this a bounded delta). Not observed to be needed in the runs above; included because the code path exists.

Workarounds

• Do not interrupt long runs; if containment is needed use memory caps, not wall-clock timeouts.
• Split big ingests into many small runs (per directory / per day) so each run's end-of-run watermark write and TRUNCATE checkpoint happen while the unit of work is small.
• After an accidental kill, expect the next run to spend the full scan time again plus a WAL replay; nothing needs repair.
• On hosts with always-open sessions, budget for every scheduled incremental run to re-parse everything modified since the last quiet moment; there is currently no knob that advances the watermark past an active file.

Severity / scope

Medium. No data loss or corruption; the cost is wasted work proportional to the size of the source tree (re-parse) and of the WAL (replay), which on multi-GB archives is hours — and, on hosts with long-lived agent sessions, is paid on every scheduled run. Combined with the WAL-growth report (#425) it makes long runs both slow and un-interruptible.

[Dicklesworthstone]

Thanks for the measurements. The most actionable part is the per-connector watermark being withheld for the whole run whenever any session is active, so a host with an always-open TUI re-parses everything modified since the last quiet moment on every scheduled run; the ledger shape you propose (per-file rows committed with each chunk, watermark advanced per connector once its scan and batches are done) is the right direction, and it also makes the active-session skip cheap instead of run-wide.

This spans signal handling, the streaming/batch watermark semantics and a new schema table, so it is being considered as a design item rather than patched piecemeal; no timeline yet. One confirmation on the batch-mode 10 s watermark write: your reading of the source is correct that it can skip parsed-but-unpersisted files after a kill, which is one more reason to move to a ledger rather than tune that timer.

[Dicklesworthstone]

Current main has fixed the unsafe watermark portions of this report, but the per-source resume capability is not complete yet:

• streaming mode advances each connector watermark only after that connector reports Done and its emitted batches have committed;
• batch mode no longer advances a timed global watermark while other pre-parsed connector batches remain pending; it advances only the fully persisted connector;
• one active/skipped source now withholds only that connector watermark, rather than freezing unrelated connectors.

The remaining exact per-file ledger needs an upstream connector seam. FAD 0.2.1 exposes discover_source_files() and conversation-level scan_with_callback(), but no pre-parse host skip predicate or source-complete event. CASS cannot safely infer completion from NormalizedConversation.source_path: one SQLite/file source can emit multiple conversations across batches, so marking it after the first committed conversation would skip uncommitted siblings after interruption.

That root contract is already tracked in franken_agent_detection#18 and #22. Once source-boundary events land, CASS can write the exact (connector, origin/source_id, normalized path, size, mtime) completion row in the same transaction as the final conversation batch and use the host predicate before parsing on resume.

I am keeping this issue open. I am deliberately not adding a disconnected ledger table or another timestamp shortcut, since neither would deliver safe resume and the latter could reintroduce silent omission.