perf: reduce watcher work and add experimental lazy MCP startup

[thedonmon]

MCP sessions can spend CPU both maintaining active watchers and indexing projects that no code request ever uses. This PR contains two independently reviewable changes: a watcher predicate reorder with unchanged matching semantics, and experimental demand-driven MCP initialization behind CODEDB_LAZY_MCP=1. Eager initialization remains the default; installers do not enable the experiment.

Experimental lazy MCP startup

When opted in, initialization, tools/list, ping, status, and projects leave the default project idle. The first code request without a project argument starts normal initialization and watching. Explicit project requests retain the existing project-cache path. Connections stay open, explicit-root precedence is preserved, and retrieval ranking/breadth is unchanged.

Tradeoffs: the first default-project query pays initialization cost (up to a 30-second readiness wait, then an explicit retry error). Status-first clients must handle scan: idle. Lazy sessions skip speculative warmup and the MCP CLI proxy; CLI calls use the existing standalone/daemon fallback. This does not consolidate processes, share worktree indexes, or reduce watcher work once sessions are actively queried.

Four fresh Git worktrees with 1,024 files each, stable ReleaseFast builds, two runs per build in AB/BA order:

Unused-session metric, four processes combined	Previous watcher build	Lazy experiment
Cumulative startup CPU	0.68–0.71 seconds	0.01 seconds
Summed RSS	182.7–183.3 MiB	46.1 MiB
First query on this fixture	20–25 ms	69–112 ms

CPU is near the measurement floor for the candidate; this is avoided startup work, not an indexing-throughput claim. Each run checks first-query additions and subsequent edits/deletions. Opt-in setup and rollback: docs/mcp.md.

The lazy comparison used Apple Silicon macOS and Zig 0.17.0-dev.813+2153f8143, with cf86ae8 as baseline. Each worktree contained 64 directories and used an isolated HOME cache; auto-update and telemetry were disabled in both builds. Two sessions used positional roots and two client roots. After sequential handshakes, wait five seconds and sum ps time and RSS across the four processes. CPU includes startup and has 0.01-second granularity per process; summed RSS is not unique physical memory. The first query pays deferred initialization, so these small fixtures do not establish large-repository latency.

Reproduce with stable ReleaseFast binaries outside zig-out (unit tests can overwrite it with Debug):

python3 scripts/bench_lazy_mcp.py --binary /tmp/base/bin/codedb
python3 scripts/bench_lazy_mcp.py --binary /tmp/head/bin/codedb --require-lazy
# Repeat in reverse order; each invocation creates fresh worktrees and caches.
python3 scripts/e2e_lazy_mcp_test.py --binary /tmp/head/bin/codedb

Validation for the lazy experiment:

• Required unit suite: 33/33 build steps succeeded; 1,110 passing tests and nine skips among executed groups, other groups cached.
• Standard MCP E2E on the exact ReleaseFast candidate: 76/76 passed with the flag absent.
• Lazy lifecycle suite: six passing scenarios covering cwd fallback, changed roots, missing roots, metadata-only idle past the fallback deadlines, stale snapshots, and explicit-root precedence. Added to macOS CI.
• Baseline fails the lazy-idle assertion as expected. Two candidate benchmark runs passed first-query and live watcher checks.

Watcher registration optimization

With watch-budget overflow, rearmDirty receives native event paths plus every fallback-polled file. For each watched file/directory, affectedByDirtyDirectory previously searched the entire watched-directory map for each dirty path before checking whether the path could affect that target. One event can therefore cause repeated nested scans across mostly unrelated fallback files.

Check equality/ancestry first, then call hasWatchedDir only for a related path. This is a two-line reorder of pure predicates: both conditions must still pass, and the chosen registrations are identical. All fallback reads and hashes, polling intervals, descriptor limits, path filtering, and retrieval behavior remain unchanged.

Evidence

Stable ReleaseFast builds on Apple Silicon macOS, pinned Zig 0.17.0-dev.813+2153f8143. Baseline production code is 4eaca89. Both fixed-work test binaries contain the same opt-in fixture: 512 files, 64 leaf directories, watch budget 256, three warmup cycles and 20 measured cycles. Eight paired runs alternate AB/BA with test seed 0x747.

Fixed-work metric, median per cycle	Baseline	Candidate
Event cycle: reconciliation + rearming	17.06 ms	6.82 ms
Rearming within that cycle	9.78 ms	0.148 ms
Quiet cycle	6.18 ms	6.02 ms

That is about 60% less event-cycle work, with 98.5% less rearming work. The paired comparator also required quiet-cycle time to remain within 10% of baseline.

A separate real MCP-process benchmark loads a prebuilt snapshot, settles, then creates/deletes an ignored root file 40 times, with 125 ms between actions. An in-place source edit after measurement must become visible through codedb_symbol.

Execution order	Baseline CPU	Candidate CPU	CPU reduction
Baseline → candidate	13.96%	7.26%	48%
Candidate → baseline	13.92%	8.33%	40%

Both versions saw the subsequent edit in approximately 0.9 seconds. These figures apply to the controlled filesystem-activity workload; they are not a claim of equivalent savings for quiet sessions or every repository.

Reproduce the fixed-work fixture with:

CODEDB_BENCH_WATCHER=1 zig build test-watcher \
  -Dtest-filter='benchmark watcher fixed cycles' -Doptimize=ReleaseFast

For a baseline comparison, use the same benchmark test on both revisions, preserve each compiled test binary and run them in alternating order. Reproduce the real-process comparison with:

python3 scripts/bench_watcher_activity.py \
  --binary /tmp/base/bin/codedb --binary /tmp/head/bin/codedb
# Repeat with the --binary arguments reversed.

Correctness and validation

• The same-metadata rewrite regression retains per-poll content verification for unwatched files and directories.
• New matching cases cover directory descendants, unrelated prefixes, file-only dirty entries and root-directory semantics.
• The dedicated test-watcher target ensures the macOS CI filter actually reaches the embedded watcher tests. Previously test-index -Dtest-filter=issue-709 selected no tests.
• zig build test -j2 passed; the dedicated watcher target has 45 passing correctness tests plus the opt-in benchmark, which is skipped by default.
• MCP E2E: 76/76 passed.
• Descriptor regression passed: 45 descriptors for both 128 and 2,048 files, with budget 32.
• Formatting and whitespace checks passed.

The earlier timestamp-only optimization and its 71–86% claim remain withdrawn. The review correctly identified its stale-index bug. This revision restores the original hashing behavior and measures a separate optimization in watch-registration selection.

[github-actions]

👋 Thanks for the contribution! Quick heads-up: this repo lands changes on the current release/* branch, not main.

Please retarget this PR via Edit → base branch to the active release/* branch shown in the repository branch list.

(Automated hint — reply here if you need a hand.)

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-06T00:01:20.294978Z	7cdaa19	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7cdaa19d34

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".