STR-03: local file retrieval (shared engine, lookup index, benchmark)

[kaanozhan]

The search hook and find-module only knew files that belong to a concept group: 121 of 435 files in this repository. This adds one shared retrieval engine, a compact index the worker keeps current, a frozen benchmark, and the instrument for measuring real agent effort.

What changes

• scripts/structure-retrieval.js (new) — one engine for the CLI and the hook.
  • Evidence tiers: path, concept, synonym, file name, symbol and partial concept (the hook stops here), then path words and description words (CLI only).
  • Every identifier word must be explained by a match ("webpack config" doesn't hint because of "config"); non-ASCII words count as prose.
  • Turkish folding (including İ), camelCase/snake/kebab splitting, regex and glob syntax stripped.
  • legacyRetrieve reproduces today's behavior exactly; it is the rollback path.
• lookup.json — published after every working-view update (130 KB against the 662 KB map), recording source and curation signatures. A stale index is never trusted.
• find-module — --json (frame.lookup/1; unavailable exits 1, no-match exits 0), --limit, and --retrieval=legacy|v2 or project.retrieval.engine. A lookup no longer runs Git or a full --check.
• The search hook — v2 reads the index (≤ 2 MiB) and stays quiet when the map is dirty or incomplete. It labels candidates from an unverified map, hints only files that exist inside the project, and caps output at 8 files / 1,800 characters. Dedup is per map revision. A transitive import-closure test keeps out processes, the network and builders.
• Benchmark — scripts/eval/retrieval-cases.json: 194 frozen queries, development/held-out split by family, 35 Turkish, 39 negatives. run-retrieval.js runs them through the real CLI and the Claude and Codex hook adapters.
• S8 instrument — run-eval.js --retrieval-arms (no-hint / legacy / v2, repeats, seeded order, cell validity); score.js counts tokens including cache, from the final result only, with unknown ≠ 0.

Results (held-out, run once)

	legacy	v2	gate
exact recall	64.9%	100%	100% ✓
recall@5	57.4%	89.4%	≥ 90% ✗
precision@1	55.3%	92.0%	≥ 90% ✓
hint precision	56.6%	96.5%	≥ 98% ✗
false hints	10.7%	3.6%	≤ 2% ✗
hook p95 (10k files)	68 ms	55 ms	≤ 50 ms ✗

The default stays legacy, as decided before the run: v2 is better everywhere but misses 4 gates. It is opt-in through project.retrieval.engine: "v2". Failure analysis and next-round candidates (validated on a new held-out split) are in scripts/eval/README.md. The paid S8 run is pending, and no token saving is claimed.

Tests

npm test: 1169 passing. New: structureRetrieval, findModule, retrievalEval. Extended: module-hint, structureLifecycle, projectAgnostic, scriptsProjectRoot.

Spec: .frame/specs/str-03-local-file-retrieval/. Separate fix: #168 (the commit-hook hang).