From db703308c8a93866773adbc7a366823d9c5bc86f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kaan=20=C3=96zhan?= Date: Thu, 1 Oct 2026 15:53:32 +0300 Subject: [PATCH 1/9] docs(spec): re-plan STR-03 on the merged structure contracts and generate its tasks --- .../specs/str-03-local-file-retrieval/plan.md | 223 ++++++++++++++++++ .../specs/str-03-local-file-retrieval/spec.md | 40 ++++ .../str-03-local-file-retrieval/status.json | 18 ++ .../str-03-local-file-retrieval/tasks.md | 9 + 4 files changed, 290 insertions(+) create mode 100644 .frame/specs/str-03-local-file-retrieval/plan.md create mode 100644 .frame/specs/str-03-local-file-retrieval/spec.md create mode 100644 .frame/specs/str-03-local-file-retrieval/status.json create mode 100644 .frame/specs/str-03-local-file-retrieval/tasks.md diff --git a/.frame/specs/str-03-local-file-retrieval/plan.md b/.frame/specs/str-03-local-file-retrieval/plan.md new file mode 100644 index 00000000..f77280f4 --- /dev/null +++ b/.frame/specs/str-03-local-file-retrieval/plan.md @@ -0,0 +1,223 @@ +# Plan — STR-03 — Local File Retrieval + +## Architecture + +### Resolved plan-time decisions + +Re-planned 2026-10-01 on top of the merged STR-01, STR-02, STR-02b and STR-02c; the 2026-09-25 plan predates all four and is replaced. + +- **D1 · Scope of the evidence (business, asked).** This spec ships local retrieval, a deterministic held-out benchmark (S7) and the matched-agent instrument (S8): runner arms, transcript accounting and paired reporting, validated on synthetic transcripts. The paid matched-agent run is a separate, explicitly commanded operation; until it has run, the eval README records the S8 result as pending and no token saving is claimed. +- **D2 · Rollout (business, asked).** The new engine (`v2`) becomes the default for `find-module` and the search hook within this spec, once the held-out benchmark passes the A5 gates. `legacy` stays selectable (`--retrieval=legacy`, or `project.retrieval.engine` in `.frame/config.json`) as the rollback path. If a gate fails, the default stays `legacy` and the README records the failed gate. +- **D3 · Delivery stays additive (business, carried).** Hooks remain optional, local, non-blocking and exit 0 with empty output on any failure; they never cancel the original search, run Git, rebuild the map or touch the network. No graph hints, no new CLI integrations, no change to hook registration, trust or user overrides (`audit-q3-deterministic-graph-hints`, `codex-parity`). +- **D4 · One compact lookup index (technical, asked).** After every working-view publication the writer also publishes `.frame/runtime/structure/lookup.json`: paths, basenames, symbols, curated concepts/synonyms and short descriptions only. The hook reads it only when it is at most 2 MiB; above that it stays quiet while the CLI keeps working from the map. Sharded buckets were rejected: more code and activation machinery for scales (50k files) this project's users have not reached. +- **D5 · Test posture (technical, asked).** Everything testable with Node's built-in runner and subprocess fixtures — the project's standing convention (`.frame/PROJECT_NOTES.md` → Testing): ranking, normalization, index build/staleness, hook and CLI adapters, benchmark metrics and transcript scoring. +- **D6 · Fallback without an index (technical, silent).** A missing or stale `lookup.json` is never rebuilt in a hook. The hook then compiles the same index in memory from the read view (`structure-read.resolveReadPath`) only if that file is at most 2 MiB, else stays quiet. This keeps today's behavior for a clone without a worker; the CLI always may compile in memory. +- **D7 · Staleness by signature (technical, silent).** `lookup.json` records the signature (`structure-read.artifactSignature`) of the map file it was built from, the signature of `intent-map.json`, the map's `generation.revision` and an algorithm version. Any mismatch with the current files makes the index stale (D6 applies). No hashing of the map in the hook. +- **D8 · Evidence hierarchy (technical, silent; carried from the old D5 with today's measurements).** Tiers, highest first: + 1. exact repo-relative path; + 2. exact curated concept; + 3. curated synonym; + 4. exact basename, with or without extension; + 5. exact symbol (exported name, function name, IPC channel); + 6. partial curated concept (substring either way, today's hook tier 3, kept because transcript replay found it useful — `audit-q3-core-value-efficacy`); + 7. path-token match; + 8. description-token overlap. + + Hooks emit tiers 1–6; tiers 7–8 are CLI-only. A query that exactly names a file is a file request: tiers 1/4 beat a coincidental concept alias. Curated order is preserved; ties sort by path. +- **D9 · Query normalization (technical, silent).** The raw operand is tried first as a path/basename. Tokens are then built with Unicode NFKC, case folding plus Turkish aliases (ı/i, İ/i, ş/s, ğ/g, ç/c, ö/o, ü/u), camelCase/snake_case/kebab-case splitting, the existing noise list, and length ≥ 3 for concept tokens. Raw path identities are never merged: two files differing only in case stay two candidates. Queries are bounded to 512 characters and 8 tokens. +- **D10 · No separate result cache (technical, silent).** The revision-stamped `lookup.json` is the precomputed cache: a query costs one bounded read plus in-memory term lookups, so a candidate cache would add invalidation risk without measurable gain at the D4 cap. The old plan's `structure-retrieval-cache.js` is dropped. +- **D11 · Revision-aware dedup (technical, silent).** The hook's session state (`.frame/runtime/module-hint/.json`, 7-day TTL, unchanged location) keys deliveries by `revision + candidate-set fingerprint` instead of the concept name. A map, curation or engine change therefore permits a new hint in the same session (S4). Without a session id nothing is persisted, so no global bucket is shared across sessions. +- **D12 · Freshness wording (technical, silent).** + - Fresh map: normal hint. + - Map `dirty`, or inventory not `complete`: the hook stays quiet, as today for `dirty`. + - Map `stale` or `unknown`: the hook still hints, but as "candidates — map not verified recently", never as "already answers" (S3). + - CLI: prints the descriptor as today, plus a structured `freshness` field in `--json` output. +- **D13 · Lookup stays read-only and fast (technical, silent).** `find-module` drops its date heuristic (`stalenessBanner`, which runs `git log` and a full `--check` for up to 30 s) and keeps the STR-02 descriptor only. An unknown map says "Map: unverified (no lifecycle record)". `check-freshness` keeps the drift check. +- **D14 · Path validation (technical, silent).** Before output, each of at most 8 candidates is checked: + - its relative path must stay inside the root (no `..`, absolute or symlink-escaping path); + - it must be a regular file now. + + A missing candidate is dropped from hooks and shown with "⚠ missing on disk" by the CLI (S5). Projects and worktrees never share results, because the index lives in each checkout's own runtime directory. +- **D15 · Settings home (technical, silent; verified).** `project.structure` rejects unknown keys (`structure-discovery.js:139`) and feeds the generation policy digest, so retrieval settings live under a new `project.retrieval` key with one field: `engine: "v2" | "legacy"`. An invalid value falls back to the default with a stderr note in the CLI, and silently in the hook. +- **D16 · Collisions (technical, silent).** STR-04 (planned) shares `scripts/eval/run-eval.js`, `score.js` and `structureBootstrap.js`; `audit-q3-performance-resources` (one measurement task left) shares `structureBootstrap.js`. Work stays sequential; STR-04 receives this spec's frozen corpus and measured local baseline. + +### A1. Retrieval core + +New `scripts/structure-retrieval.js`: pure, dependency-free and shipped to `.frame/bin/`. + +- `compileIndex(structure, curation, source)` returns the D4 index object: + - `version`, `algorithm`, `revision`, `source: { path, signature }`, `curation: { signature }`; + - `files: [{ path, key, description }]`, with descriptions clipped to 160 characters; + - `terms`: postings `{ term → [[fileId, tier]] }` for paths, basenames, symbols and description tokens, capped at 64 postings per term and marked truncated; + - `concepts: { name → { fileIds, synonyms } }`. +- `normalizeQuery(raw)` applies D9. +- `retrieve(index, query, { mode: 'hook' | 'cli', limit })` returns `{ status: 'resolved' | 'ambiguous' | 'no-match', candidates: [{ path, key, tier, evidence, description }], truncated }`. + - Status is `resolved` only for a single tier-1/2/3/4 answer, or a single curated group. + - Several exact basenames or symbols are `ambiguous`, and the candidate set is listed. + - Mode `hook` applies the D8 tier cut; the limit is at most 8. +- `legacyRetrieve(structure, query)` keeps today's tiers exactly (find-module's four, the hook's three) behind the same result shape, for D2 rollback and the A5 baseline. + +### A2. Index publication + +`scripts/structure-lifecycle.js` (`reconcile`) and `scripts/update-structure.js` (`runFull`, `runDelta`) call one helper after a successful working-view publication, or an unchanged one when the index is missing or stale. The helper is `publishLookup(root)`, new in `structure-retrieval.js`: + +- It reads the working view's bytes and `intent-map.json`, compiles the index, and writes `lookup.json` atomically (temp + rename). +- A failure is recorded in the result and never fails the map publication. +- An index above 2 MiB is still written, flagged `oversize: true`, so the CLI can use it and the hook knows to stay quiet. + +Discovery, snapshot, writer ownership and refresh timing are not changed (STR-01/02 own them). + +### A3. Search hook + +`scripts/module-hint.js` keeps its fast non-search bail, segment/heredoc guards, tool vocabulary, activity notes and output contract (`hookSpecificOutput.additionalContext`). + +**Changes:** +- Pattern extraction also takes `Glob` patterns, `find -path`, and the full operand for path/basename lookup. +- The engine comes from `project.retrieval.engine`, default per D2. Index loading follows D6/D7, freshness D12, ranking D8, dedup D11 and path checks D14. + +**Output:** +- At most 8 files and 1,800 characters. Descriptions are clipped first, and output stops at a whole-candidate boundary. +- It names the evidence ("file name", "symbol", "concept") and the `find-module` command. +- It never claims the search was suppressed. + +**New quiet reasons**, with texts in `src/shared/activityEvents.js` `SEARCH_REASON_TEXT`: +- `index-oversize`; +- `map-incomplete`; +- `ambiguous-weak`: only tiers 7–8 matched. + +The header comment's obsolete "no sibling imports" rule is rewritten to the real constraint: only read-only helpers. A test enforces it on the import closure (no `child_process`, `net`, `http(s)`, builder or writer modules). + +### A4. Explicit lookup + +`scripts/find-module.js` uses A1 and keeps positional queries and `--list`. New flags: +- `--json`: one bounded envelope on stdout, with `{ schema: 'frame.lookup/1', status, engine, freshness, candidates, truncated }`. Diagnostics go to stderr. A missing or corrupt map gives `status: 'unavailable'` with exit 1, distinct from `no-match` with exit 0. +- `--limit `: default 8, at most 20. +- `--retrieval=legacy|v2`. + +Human output shows evidence and ambiguity, plus ⚠ for missing files. D13 removes the Git/`--check` banner. + +### A5. Benchmark, gates and matched-agent instrument + +**Corpus.** New `scripts/eval/retrieval-cases.json` holds 180 labelled queries, frozen before tuning, with a SHA-256 of each split recorded in the file: +- Splits: 60 development and 120 held-out, split by file family so paraphrases do not leak. +- Required coverage: at least 30 Turkish queries, 30 no-match/negative queries, 20 singleton files outside every intent group, plus same-name files, symbols, shell/regex noise, a renamed/removed file and a stale map. +- Labels are acceptable file sets in this repository at a pinned commit, and in a small synthetic multilingual fixture generated by the benchmark. + +**Runner.** New `scripts/eval/run-retrieval.js`: +1. Exports the pinned commit (`git archive`) into a temp directory. +2. Runs `update-structure.js --full` there (this publishes `lookup.json`). +3. Evaluates both engines through the CLI (`--json`) and the real hook adapter (stdin payloads in the Claude and Codex shapes). +4. Generates synthetic 1k- and 10k-file projects for latency. + +It reports, with sample counts per stratum: +- recall@5 (answerable queries), precision@1, emitted-hint precision, false-hint rate on negatives and abstention rate; +- p50/p95 cold-process latency for hook and CLI; +- additionalContext characters and bytes; +- index size and build time. + +**Gates for D2,** all on the held-out split: +- 100% expected-file recall on exact path/basename/symbol/curated cases; +- recall@5 ≥ 90% and not below legacy; +- precision@1 ≥ 90%; +- emitted-hint precision ≥ 98%; +- false hints on negatives ≤ 2%; +- hook p95 ≤ 50 ms on this repo and on the 10k fixture, measured on the machine recorded in the README; +- CLI p95 ≤ 150 ms; +- payload within A3. + +Tests assert the metric math and gate evaluation, never wall-clock numbers. + +**Matched-agent instrument (S8, D1).** `scripts/eval/run-eval.js` gains `--retrieval-arms`: three arms (`no-hint`, `legacy`, `v2`) with identical worktrees, prompts, model and permissions. Only the search-hook registration and engine differ. A cell is invalid if its hook did not emit (or withhold) as the arm intends. `scripts/eval/score.js` adds: +- files found/read; +- search and read calls; +- input tokens including cache-creation and cache-read tokens where reported; +- output tokens, separately; +- elapsed time and failures. + +Missing telemetry is `unknown`, not 0, and per-message usage is not double-counted with the final result usage. `tasks.json` gets at least 12 navigation tasks for those arms. The README records the protocol, the gates, the benchmark result and the D2 promotion decision, with S8 marked "pending — run with `node scripts/eval/run-eval.js --retrieval-arms`". + +### Coverage + +- G1 → A1, A3, A4. +- G2 → A1 (one engine, two modes), D8. +- G3 → A5. +- C1 → D1–D2 (built on the STR-02c read view), A2. +- C2 → D8. +- C3 → D3, A3. +- C4 → D3, D15, A4 (flags are additive). +- C5 → D4, D9, A3 limits, A5 budgets. +- S1 → D8, A1. +- S2 → A1 status/ambiguity, D8 hook cut. +- S3 → D12, A4 `unavailable`. +- S4 → D7, D10, D11. +- S5 → D14. +- S6 → A3, A5 adapter runs. +- S7 → A5 benchmark. +- S8 → A5 instrument, D1. + +## Files + +- `scripts/structure-retrieval.js` — **New** — index compilation and publication, normalization, tiered retrieval (v2) and the legacy engine. +- `scripts/module-hint.js` — **Modified** — shared engine, index loading/staleness, freshness wording, revision-aware dedup, bounded payload. +- `scripts/find-module.js` — **Modified** — shared engine, `--json`/`--limit`/`--retrieval`, evidence output, no Git banner. +- `scripts/structure-lifecycle.js` — **Modified** — publish `lookup.json` after a working-view publication. +- `scripts/update-structure.js` — **Modified** — publish `lookup.json` after `--full`/file updates. +- `src/main/structureBootstrap.js` — **Modified** — ship `structure-retrieval.js`; entry requirements for `find-module`, `module-hint`, `update-structure`, `structure-lifecycle`. +- `src/shared/activityEvents.js` — **Modified** — texts for the new hook quiet reasons. +- `src/shared/frameTemplates.js` — **Modified** — REFERENCE wording for lookup evidence, `--json`, and `project.retrieval.engine`. +- `package.json` — **Modified** — add `scripts/structure-retrieval.js` to the packaged files. +- `scripts/eval/retrieval-cases.json` — **New** — frozen development/held-out query corpus with split hashes. +- `scripts/eval/run-retrieval.js` — **New** — deterministic quality/latency/payload benchmark and gate evaluation. +- `scripts/eval/run-eval.js` — **Modified** — `--retrieval-arms` (no-hint/legacy/v2) with emission checks. +- `scripts/eval/score.js` — **Modified** — files found/read, search/read calls, full token accounting, unknown-vs-zero. +- `scripts/eval/tasks.json` — **Modified** — navigation tasks for the retrieval arms. +- `scripts/eval/README.md` — **Modified** — protocol, gates, benchmark record, promotion decision, S8 pending note. +- `test/structureRetrieval.test.js` — **New** — tiers, curated precedence, singletons, ambiguity, Turkish/camelCase normalization, limits, legacy parity, index compile/staleness. +- `test/findModule.test.js` — **New** — CLI compatibility, `--json` envelope, `unavailable` vs `no-match`, missing files, engine selection. +- `test/module-hint.test.js` — **Modified** — real adapter payloads, freshness wording/quiet, revision-aware dedup, payload limits, import closure, oversize/missing index fallback. +- `test/structureLifecycle.test.js` — **Modified** — `lookup.json` published with the working view and refreshed on curation change. +- `test/projectAgnostic.test.js` — **Modified** — `update-structure` publishes `lookup.json`; a failed index never fails the map. +- `test/scriptsProjectRoot.test.js` — **Modified** — copied `.frame/bin/` runs lookup and hook without Frame's repo. +- `test/retrievalEval.test.js` — **New** — corpus split integrity, metric and gate math, transcript scoring with synthetic transcripts, arm validity. + +## Footprint + +- scripts/structure-retrieval.js +- scripts/module-hint.js +- scripts/find-module.js +- scripts/structure-lifecycle.js +- scripts/update-structure.js +- src/main/structureBootstrap.js +- src/shared/activityEvents.js +- src/shared/frameTemplates.js +- package.json +- scripts/eval/retrieval-cases.json +- scripts/eval/run-retrieval.js +- scripts/eval/run-eval.js +- scripts/eval/score.js +- scripts/eval/tasks.json +- scripts/eval/README.md +- test/structureRetrieval.test.js +- test/findModule.test.js +- test/module-hint.test.js +- test/structureLifecycle.test.js +- test/projectAgnostic.test.js +- test/scriptsProjectRoot.test.js +- test/retrievalEval.test.js + +## Dependencies + +None. The paid matched-agent run uses an already configured agent CLI and runs only on an explicit command (D1). + +## Sequencing + +1. **Corpus and baseline.** Write `retrieval-cases.json` (frozen splits with hashes) and `run-retrieval.js` measuring the current behavior through the existing CLI and hook. Record the legacy baseline in the eval README. Author `retrievalEval` tests for split integrity and metric/gate math. +2. **Retrieval core.** Add `structure-retrieval.js`: + - `compileIndex`, `normalizeQuery`, `retrieve` (v2); + - `legacyRetrieve` reproducing today's tiers; + - author `structureRetrieval` tests (tiers, curated precedence, singletons, ambiguity, normalization, limits, legacy parity). +3. **Index publication.** `publishLookup` with signatures and the oversize flag; call it from the lifecycle worker and `update-structure.js`; ship the helper (bootstrap, package). Extend the `structureLifecycle`, `projectAgnostic` and `scriptsProjectRoot` tests. +4. **Explicit lookup.** Move `find-module.js` onto the core with `--json`/`--limit`/`--retrieval`, evidence output and the D13 banner removal; author `findModule` tests. +5. **Search hook.** Move `module-hint.js` onto the core with D6/D7 loading, D11 dedup, D12 wording, D14 checks and payload bounds; add the quiet-reason texts. Extend `module-hint` tests (adapters, import closure, fallbacks). +6. **Gate and promotion.** Run the held-out benchmark against both engines, record results in the README, and set the default engine per D2. Update the REFERENCE wording in `frameTemplates.js`. +7. **Matched-agent instrument.** `--retrieval-arms` in `run-eval.js`, token/read/search accounting in `score.js`, navigation tasks in `tasks.json`, and the README S8 protocol marked pending. Extend `retrievalEval` tests with synthetic transcripts and invalid-cell cases. diff --git a/.frame/specs/str-03-local-file-retrieval/spec.md b/.frame/specs/str-03-local-file-retrieval/spec.md new file mode 100644 index 00000000..afb5b48e --- /dev/null +++ b/.frame/specs/str-03-local-file-retrieval/spec.md @@ -0,0 +1,40 @@ +--- +keywords: STR, structure, file retrieval, module hints, local search, cache, evaluation, token savings +related: reliable-structure-generation, str-02-structure-lifecycle, str-04-jev-evaluation, audit-q3-core-value-efficacy, audit-q3-deterministic-graph-hints, codex-parity +--- + +# STR-03 — Local File Retrieval + +## Problem + +Being present in STRUCTURE does not guarantee a file can be found through a hint. Current hooks depend on intent groups, leaving single-file concepts invisible, while broad content matching previously produced noisy hints. Session deduplication does not account for map changes. Moreover, injecting a hint does not itself prevent the original search or prove token savings. + +## Goal + +Give agents a fast local route from a filename, path, symbol, or concept to a small set of relevant files. Share retrieval semantics between explicit lookup and supported search hooks, while allowing hooks stricter evidence thresholds. Measure actual navigation effort and establish the baseline for STR-04. + +## Constraints + +- Series: **STR — Structure Reliability and Retrieval**. Order: STR-01 → STR-02 → STR-03 → STR-04. Depends on `reliable-structure-generation` and `str-02-structure-lifecycle` for coverage, identity, and revision contracts. +- Retain curated intent precedence from `audit-q3-core-value-efficacy`. Candidate discovery must also reach individual modules without requiring an intent group. Do not restore broad noisy hook matching without measured evidence. +- Preserve the additive, non-blocking hook contract recorded in `audit-q3-deterministic-graph-hints` and `codex-parity`: search remains available, malformed input degrades quietly, and no network request or map rebuild occurs in the hook. +- Keep existing CLI adapters and user settings intact. This is STRUCTURE retrieval; the graph-specific work in `audit-q3-deterministic-graph-hints` remains separate. +- Use bounded local work, payloads, and caches. Set latency, payload, and false-positive budgets during planning before comparing implementations; preserve the existing approximately 50 ms hook target unless evidence justifies an explicit revision. + +## Success Criteria + +1. When a query names an indexed path, filename, or extracted symbol, then the relevant file is retrievable even without an intent group. When a curated concept or synonym matches, then curated ownership remains authoritative. +2. When a query is ambiguous, then results expose a bounded candidate set and match evidence rather than falsely asserting one definitive answer. Weak automatic matches remain silent. +3. When the map is incomplete, stale, or missing, then explicit lookup reports that condition, hooks avoid authoritative claims, and ordinary filesystem search remains usable. +4. When queries repeat, then caching and hint deduplication avoid redundant work within the same revision. When the map, curation, or retrieval configuration changes, then stale hits and cached misses are invalidated, including within an existing session. +5. When a file is removed or renamed between indexing and lookup, then invalid paths are not presented as verified answers; projects and worktrees cannot reuse one another's results. +6. When supported CLI search inputs run through their real hook adapters, then hints remain bounded, non-blocking, and compatible with each adapter's output contract. +7. When representative held-out queries run against baseline and improved retrieval, then the report measures relevant-file recall, top-result precision, false hints, p50/p95 latency, and payload size, including singleton files, multilingual queries, and no-match cases. +8. When matched agent tasks run with and without the change, then the report records files found, search/read calls, total context tokens, and elapsed time. Deployment requires the predeclared quality budgets; token savings are claimed only when observed, not inferred from a hint firing. + +## Out of Scope + +- Generation and lifecycle changes owned by STR-01 and STR-02. +- Jev, remote reranking, embeddings, and paid semantic enrichment: STR-04. +- Graph construction or graph-specific hints in `audit-q3-deterministic-graph-hints`. +- Blocking or cancelling agent tools, and new CLI integrations beyond existing adapters. diff --git a/.frame/specs/str-03-local-file-retrieval/status.json b/.frame/specs/str-03-local-file-retrieval/status.json new file mode 100644 index 00000000..733cb8c1 --- /dev/null +++ b/.frame/specs/str-03-local-file-retrieval/status.json @@ -0,0 +1,18 @@ +{ + "slug": "str-03-local-file-retrieval", + "title": "STR-03 — Local File Retrieval", + "phase": "tasks_generated", + "generated_task_ids": [ + "task-spec-str-03-local-file-retrieval-T01", + "task-spec-str-03-local-file-retrieval-T02", + "task-spec-str-03-local-file-retrieval-T03", + "task-spec-str-03-local-file-retrieval-T04", + "task-spec-str-03-local-file-retrieval-T05", + "task-spec-str-03-local-file-retrieval-T06", + "task-spec-str-03-local-file-retrieval-T07" + ], + "ai_tool": "claude-code", + "created_at": "2026-09-17T23:58:30Z", + "updated_at": "2026-10-01T12:53:22.489Z", + "last_phase_at": "2026-10-01T12:53:22Z" +} diff --git a/.frame/specs/str-03-local-file-retrieval/tasks.md b/.frame/specs/str-03-local-file-retrieval/tasks.md new file mode 100644 index 00000000..93da17a9 --- /dev/null +++ b/.frame/specs/str-03-local-file-retrieval/tasks.md @@ -0,0 +1,9 @@ +# Tasks — STR-03 — Local File Retrieval + +- T01 · Create `scripts/eval/retrieval-cases.json` (180 labelled queries, 60 development / 120 held-out split by file family with per-split SHA-256, ≥30 Turkish, ≥30 negatives, ≥20 singleton files, same-name files, symbols, shell/regex noise, a renamed/removed file, a stale map) and `scripts/eval/run-retrieval.js` (export the pinned commit with `git archive`, run `update-structure.js --full`, query the current `find-module` CLI and the real `module-hint` hook with Claude and Codex payloads, generate synthetic 1k/10k-file projects, compute recall@5, precision@1, emitted-hint precision, false-hint and abstention rates, p50/p95 latency, payload size, index size/build time, and evaluate the A5 gates); record the legacy baseline in `scripts/eval/README.md`; author `test/retrievalEval.test.js` for split integrity and metric/gate math +- T02 · Add `scripts/structure-retrieval.js` with `compileIndex`, `normalizeQuery` (NFKC, case folding with Turkish aliases, camel/snake/kebab splitting, raw path identities kept, 512 characters / 8 tokens), `retrieve` with the eight evidence tiers and the hook/CLI cut, and `legacyRetrieve` reproducing today's find-module and hook tiers; author `test/structureRetrieval.test.js` for tiers, curated precedence, singletons, ambiguity, normalization, limits and legacy parity +- T03 · Add `publishLookup(root)` to `scripts/structure-retrieval.js` (atomic `.frame/runtime/structure/lookup.json` with source/curation signatures, revision, algorithm version and the 2 MiB `oversize` flag; failures recorded, never failing the map), call it after working-view publications in `scripts/structure-lifecycle.js` and `scripts/update-structure.js`, and ship the helper through `src/main/structureBootstrap.js` entry requirements and `package.json`; extend `test/structureLifecycle.test.js`, `test/projectAgnostic.test.js` and `test/scriptsProjectRoot.test.js` +- T04 · Move `scripts/find-module.js` onto the shared engine with `--json` (`frame.lookup/1` envelope, `unavailable` exit 1 vs `no-match` exit 0), `--limit` (default 8, max 20), `--retrieval=legacy|v2` and `project.retrieval.engine`, evidence/ambiguity output with ⚠ for missing files, contained-path checks, and the Git/`--check` staleness banner removed; author `test/findModule.test.js` +- T05 · Move `scripts/module-hint.js` onto the shared engine: Glob/`find -path`/full-operand extraction, index loading with signature staleness and the ≤2 MiB in-memory fallback, quiet on dirty/incomplete/oversize/weak-only, "candidates — map not verified recently" wording for stale/unknown maps, hook tiers 1–6 only, dedup keyed by revision + candidate fingerprint (no persistence without a session id), contained regular-file checks, ≤8 files and ≤1,800 characters, rewritten header contract; add the new quiet-reason texts to `src/shared/activityEvents.js`; extend `test/module-hint.test.js` with Claude/Codex adapter payloads, import-closure, fallback, dedup and payload-limit cases +- T06 · Run the held-out benchmark for both engines, record the results and the promotion decision in `scripts/eval/README.md`, set the default engine per the gates (v2 when all pass, otherwise legacy with the failed gate named), and update the REFERENCE wording in `src/shared/frameTemplates.js` for lookup evidence, `--json` and `project.retrieval.engine` +- T07 · Add `--retrieval-arms` (no-hint / legacy / v2, identical worktrees, invalid cells when a hook did not emit or withhold as intended) to `scripts/eval/run-eval.js`, files found/read, search/read calls, full input-token accounting with cache tokens, output tokens, elapsed time and unknown-vs-zero to `scripts/eval/score.js`, at least 12 navigation tasks to `scripts/eval/tasks.json`, and the S8 protocol marked pending to `scripts/eval/README.md`; extend `test/retrievalEval.test.js` with synthetic transcripts and invalid-cell cases From cfd20eca86a4a66ace5c0438f781def8c8e4dfa4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kaan=20=C3=96zhan?= Date: Thu, 1 Oct 2026 16:01:00 +0300 Subject: [PATCH 2/9] feat(eval): frozen retrieval corpus, benchmark runner and legacy baseline (STR-03 T01) --- .frame/STRUCTURE.json | 233 +- .../str-03-local-file-retrieval/outcome.md | 19 + .../str-03-local-file-retrieval/status.json | 7 +- scripts/eval/README.md | 82 + scripts/eval/retrieval-cases.json | 2353 +++++++++++++++++ scripts/eval/run-retrieval.js | 450 ++++ test/retrievalEval.test.js | 145 + 7 files changed, 3278 insertions(+), 11 deletions(-) create mode 100644 .frame/specs/str-03-local-file-retrieval/outcome.md create mode 100644 scripts/eval/retrieval-cases.json create mode 100644 scripts/eval/run-retrieval.js create mode 100644 test/retrievalEval.test.js diff --git a/.frame/STRUCTURE.json b/.frame/STRUCTURE.json index fa33d45f..f600cf82 100644 --- a/.frame/STRUCTURE.json +++ b/.frame/STRUCTURE.json @@ -1,7 +1,7 @@ { "version": "1.1", "description": "Auto-generated module structure", - "lastUpdated": "2026-09-30", + "lastUpdated": "2026-10-01", "architecture": {}, "modules": { ".claude/settings.json": { @@ -16382,7 +16382,7 @@ "exports": [], "depends": [], "functions": {}, - "sizeBytes": 3923, + "sizeBytes": 7654, "extraction": { "status": "parsed" }, @@ -16390,6 +16390,18 @@ "description": "deafc552be20c9b3" } }, + "scripts/eval/retrieval-cases.json": { + "file": "scripts/eval/retrieval-cases.json", + "description": "", + "exports": [], + "depends": [], + "functions": {}, + "sizeBytes": 55037, + "extraction": { + "status": "unsupported", + "reason": "no-extractor" + } + }, "scripts/eval/run-eval": { "file": "scripts/eval/run-eval.js", "description": "", @@ -16436,6 +16448,189 @@ "status": "parsed" } }, + "scripts/eval/run-retrieval": { + "file": "scripts/eval/run-retrieval.js", + "description": "", + "exports": [ + "handle${i" + ], + "depends": [ + "fs", + "os", + "path", + "crypto", + "child_process" + ], + "functions": { + "digest": { + "line": 53, + "params": [ + "cases" + ] + }, + "loadCases": { + "line": 58, + "params": [ + "file = CASES_FILE" + ], + "purpose": "Load the corpus; a split whose hash does not match was edited after freezing." + }, + "parseCli": { + "line": 71, + "params": [ + "stdout" + ], + "purpose": "Candidate files from find-module output: the --json envelope, or the human listing." + }, + "parseHook": { + "line": 89, + "params": [ + "stdout" + ], + "purpose": "Hinted files and the raw context from a hook's stdout ('' = no hint)." + }, + "quantile": { + "line": 109, + "params": [ + "values", + "q" + ] + }, + "summarize": { + "line": 122, + "params": [ + "results" + ], + "purpose": "results: [{ case, cli: [files], cliMs, hooks: { claude: { files, context, ms }, codex: {…} } }]" + }, + "evaluateGates": { + "line": 186, + "params": [ + "summary", + "baseline = null", + "scale = null", + "gates = GATES" + ], + "purpose": "The A5 gates for one engine's held-out summary. `baseline` is legacy's" + }, + "timed": { + "line": 212, + "params": [ + "args", + "options" + ] + }, + "baseEnv": { + "line": 218, + "params": [ + "project", + "activityHome" + ] + }, + "setEngine": { + "line": 222, + "params": [ + "project", + "engine" + ] + }, + "availableEngines": { + "line": 235, + "purpose": "Engines this checkout can run: legacy always; v2 once the shared engine exists." + }, + "cliSupportsJson": { + "line": 241, + "params": [ + "project", + "env" + ] + }, + "hookPayloads": { + "line": 250, + "params": [ + "c", + "sessionId", + "project" + ] + }, + "runCase": { + "line": 264, + "params": [ + "c", + "engine", + "project", + "env", + "json" + ] + }, + "applyMutation": { + "line": 280, + "params": [ + "base", + "mutate" + ] + }, + "buildMap": { + "line": 289, + "params": [ + "project", + "env" + ] + }, + "exportPinned": { + "line": 308, + "params": [ + "commit" + ] + }, + "syntheticProject": { + "line": 315, + "params": [ + "count" + ], + "purpose": "A synthetic project of `count` JavaScript files in nested packages." + }, + "runScale": { + "line": 329, + "params": [ + "engine", + "activityHome", + "sizes = [1000", + "10000]" + ] + }, + "parseArgs": { + "line": 355, + "params": [ + "argv" + ] + }, + "printReport": { + "line": 373, + "params": [ + "report" + ] + }, + "main": { + "line": 398 + } + }, + "sizeBytes": 20432, + "extraction": { + "status": "parsed" + }, + "provenance": { + "purposes": { + "availableEngines": "d7a3b42e8e1d0229", + "evaluateGates": "948f7fa8250c71c8", + "loadCases": "c4906eb41cfc462c", + "parseCli": "4090dfdc9064bc07", + "parseHook": "fcfc1b3dfcb70ab2", + "summarize": "38f21892ccfab5b8", + "syntheticProject": "522ebf564236d377" + } + } + }, "scripts/eval/score": { "file": "scripts/eval/score.js", "description": "", @@ -23031,6 +23226,28 @@ "description": "7899da73ee2127a3" } }, + "test/retrievalEval.test": { + "file": "test/retrievalEval.test.js", + "description": "STR-03 retrieval benchmark: the frozen corpus and the metric/gate math.", + "exports": [], + "depends": [ + "node:test", + "node:assert/strict", + "path", + "child_process", + "scripts/eval/run-retrieval", + "fs", + "os" + ], + "functions": {}, + "sizeBytes": 7907, + "extraction": { + "status": "parsed" + }, + "provenance": { + "description": "bd894ce766feb6b8" + } + }, "test/scriptsProjectRoot.test": { "file": "test/scriptsProjectRoot.test.js", "description": "Shipped-script project-root tests (non-invasive-overlay T03).", @@ -25885,8 +26102,8 @@ "extraction": { "coverage": "complete", "counts": { - "parsed": 290, - "unsupported": 82, + "parsed": 292, + "unsupported": 83, "partial": 0 } }, @@ -25932,9 +26149,9 @@ } }, "counts": { - "visitedEntries": 460, + "visitedEntries": 463, "traversedDirectories": 80, - "eligibleFiles": 372, + "eligibleFiles": 375, "unreadable": 0, "binaryFiles": 5, "symlinks": 0, @@ -25951,7 +26168,7 @@ "config": 0, "depth": 0 }, - "indexedFiles": 372, + "indexedFiles": 375, "legacyGroups": 0 }, "diagnostics": { @@ -25959,6 +26176,6 @@ "truncated": false, "samples": [] }, - "revision": "859fa6c387a5b70589dad88c312990b84f705db6989cb690fb37138df146c433" + "revision": "60fa3f10f358213741926b7c4b230997858335f69a05121ded9fe6aa21eb2417" } } diff --git a/.frame/specs/str-03-local-file-retrieval/outcome.md b/.frame/specs/str-03-local-file-retrieval/outcome.md new file mode 100644 index 00000000..ebd64f5e --- /dev/null +++ b/.frame/specs/str-03-local-file-retrieval/outcome.md @@ -0,0 +1,19 @@ +# Outcome — STR-03 — Local File Retrieval + +## T01 — Frozen corpus, benchmark runner, legacy baseline + +`scripts/eval/retrieval-cases.json` holds 194 labelled queries against pinned 262f91b: 72 development and 122 held-out, disjoint by family. It includes 35 Turkish queries (10 purely Turkish), 39 negatives, 72 whose files belong to no intent group, and three tree mutations (removed, renamed, not yet indexed). Each split carries the SHA-256 of its cases. Deviation: the plan said 180 (60/120); the corpus came out larger, and the tests assert minimums (≥60/≥120/≥180, ≥30 Turkish, ≥30 negatives, ≥20 singletons). Labels were checked against the pinned map while authoring, and `test/retrievalEval.test.js` re-checks them against `git ls-tree` of the pinned commit. + +`scripts/eval/run-retrieval.js` exports the pinned commit, builds its map with `--full`, and runs every case through `find-module` and the real hook in the Claude (Grep/Glob/Bash) and Codex (Bash, `search codex`) payload shapes. It measures synthetic 1k/10k-file projects too. Engine selection is via `project.retrieval.engine`, with `--json` used when the CLI supports it. Activity records go to a temporary `FRAME_ACTIVITY_HOME`. Metric and gate functions are exported and tested. + +Legacy baseline on held-out (recorded in `scripts/eval/README.md`): +- CLI: recall@5 57.4%, exact recall 64.9%, P@1 55.3%. +- Hook: precision 56.6%, false hints 10.7%, p95 33 ms. +- 10k files: hook p95 60 ms. +- 8 gates fail. + +Files touched: `scripts/eval/retrieval-cases.json`, `scripts/eval/run-retrieval.js`, `scripts/eval/README.md`, `test/retrievalEval.test.js`. + +_Captured: 2026-10-01 · 4 file change(s)_ + +--- diff --git a/.frame/specs/str-03-local-file-retrieval/status.json b/.frame/specs/str-03-local-file-retrieval/status.json index 733cb8c1..3cb878ee 100644 --- a/.frame/specs/str-03-local-file-retrieval/status.json +++ b/.frame/specs/str-03-local-file-retrieval/status.json @@ -1,7 +1,7 @@ { "slug": "str-03-local-file-retrieval", "title": "STR-03 — Local File Retrieval", - "phase": "tasks_generated", + "phase": "implementing", "generated_task_ids": [ "task-spec-str-03-local-file-retrieval-T01", "task-spec-str-03-local-file-retrieval-T02", @@ -13,6 +13,7 @@ ], "ai_tool": "claude-code", "created_at": "2026-09-17T23:58:30Z", - "updated_at": "2026-10-01T12:53:22.489Z", - "last_phase_at": "2026-10-01T12:53:22Z" + "updated_at": "2026-10-01T12:54:14Z", + "last_phase_at": "2026-10-01T12:54:14Z", + "implement_mode": "custom" } diff --git a/scripts/eval/README.md b/scripts/eval/README.md index 6f5a5b88..b30936b4 100644 --- a/scripts/eval/README.md +++ b/scripts/eval/README.md @@ -74,3 +74,85 @@ The pilot's methodological catches are already folded into the harness: diffs are taken against the starting sha (agents that self-commit in the worktree no longer hide their changes), and grep-based success checks measure "did the named change land", not code quality. + +--- + +# Retrieval benchmark (STR-03) + +Does `find-module` find the right file, and does the search hook speak only +when it should? A deterministic, local measurement: no agent, no network. + +## Method + +- **Corpus** — `retrieval-cases.json`: 194 labelled queries against this + repository at `pinnedCommit` (262f91b). 72 development and 122 held-out, + disjoint by file family so paraphrases cannot leak between the splits. + 35 Turkish queries (10 purely Turkish, with no identifier in them), 39 + negatives, 72 whose expected files belong to no intent group, plus shell + and regex noise, same-name files, symbols, and three cases that change the + tree after indexing (a removed file, a renamed file, a file the map has not + seen). `expect` lists the acceptable files; `[]` means nothing should be + returned. Each split carries the SHA-256 of its cases, frozen before any + tuning; `test/retrievalEval.test.js` fails if a split is edited. +- **Runner** — `run-retrieval.js`: + 1. Exports the pinned commit with `git archive` and builds its map with this + checkout's `update-structure.js --full`. + 2. Runs every case through `find-module.js` and the real `module-hint.js` + adapters, with Claude Code payloads (Grep/Glob/Bash) and Codex payloads + (Bash only, `search codex`). Every case is a cold process. + 3. Generates synthetic 1k- and 10k-file projects for latency and index size. +- **Metrics**: + - recall@5 (answerable cases, CLI); + - exact recall (path/basename/symbol/curated/synonym cases that are not Turkish-mixed); + - precision@1 (cases where the CLI returned anything); + - per adapter: emitted-hint precision, hint recall, false-hint rate on negatives, abstention; + - p50/p95 latency; additionalContext characters and bytes; + - map/lookup size and build time; + - strata per tag. + +```bash +node scripts/eval/run-retrieval.js # every split and available engine +node scripts/eval/run-retrieval.js --split heldOut --engine v2 --no-scale +node scripts/eval/run-retrieval.js --json # the full report +``` + +## Gates + +Evaluated on the held-out split only, and decided before the new engine was +written. The new engine becomes the default only when all of them pass: + +| Gate | Limit | +|---|---| +| Exact recall | 100% | +| recall@5 | ≥ 90% and not below legacy | +| precision@1 | ≥ 90% | +| Emitted-hint precision, every adapter | ≥ 98% | +| False hints on negatives, every adapter | ≤ 2% | +| Hook p95, this repository and the 10k-file fixture | ≤ 50 ms | +| CLI p95 | ≤ 150 ms | +| Hook payload | ≤ 1,800 characters | + +## Legacy baseline — 2026-10-01 + +Apple M2 · macOS (Darwin 24.6.0) · node v20.20.0 · pinned 262f91b. The map is +662 KB; `--full` builds it in 194 ms. + +| Held-out (122) | CLI | Hook (Claude) | Hook (Codex) | +|---|---|---|---| +| recall@5 / hint recall | 57.4% | 31.9% | 31.9% | +| Exact recall (77) | 64.9% | — | — | +| precision@1 / emitted precision | 55.3% | 56.6% | 56.6% | +| False hints on negatives | — | 10.7% | 10.7% | +| p50 / p95 | 41 / 44 ms | 30 / 33 ms | 30 / 33 ms | +| Max payload | — | 852 chars | 852 chars | + +Development split (72): CLI recall@5 63.9%, P@1 68.1%; hook precision 63.9%, +false hints 27.3%. Scale: 1k files gives a 607 KB map and hook p95 28 ms; +10k files gives a 6.1 MB map and hook p95 60 ms (the hook parses the whole +map on every search). + +Legacy fails 8 gates: exact recall, recall@5, precision@1, emitted precision +and false hints (both adapters), and the 10k-file hook latency. The misses are +the ones the spec describes: files outside every intent group, function names, +and partial concept matches that hint the wrong group (for example +`structureBootstrap nerede` → every file in the `structure` group). diff --git a/scripts/eval/retrieval-cases.json b/scripts/eval/retrieval-cases.json new file mode 100644 index 00000000..1f924d76 --- /dev/null +++ b/scripts/eval/retrieval-cases.json @@ -0,0 +1,2353 @@ +{ + "_comment": "STR-03 retrieval corpus. Frozen before tuning: each split carries the SHA-256 of JSON.stringify(cases); test/retrievalEval.test.js verifies it. Splits are disjoint by family (negatives aside). expect = acceptable files at pinnedCommit; [] = nothing should be returned. Run: node scripts/eval/run-retrieval.js", + "version": 1, + "pinnedCommit": "262f91bbc584e21fe464fd9adceba7a686ba6160", + "splits": { + "development": { + "sha256": "3259c37b4b6fc2b9ea9a8d741dca67a494e899aa93764a6a9c50ec6335b66b4a", + "cases": [ + { + "id": "dev-001", + "family": "github", + "query": "github", + "expect": [ + "src/main/githubManager.js", + "src/renderer/githubPanel.js" + ], + "tags": [ + "curated" + ] + }, + { + "id": "dev-002", + "family": "github", + "query": "pull-requests", + "expect": [ + "src/main/githubManager.js", + "src/renderer/githubPanel.js" + ], + "tags": [ + "synonym" + ] + }, + { + "id": "dev-003", + "family": "github", + "query": "githubPanel", + "expect": [ + "src/renderer/githubPanel.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "dev-004", + "family": "github", + "query": "rowModels", + "expect": [ + "src/renderer/github/rowModels.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "dev-005", + "family": "github", + "query": "issueBranchName", + "expect": [ + "src/renderer/github/rowModels.js" + ], + "tags": [ + "singleton", + "symbol" + ] + }, + { + "id": "dev-006", + "family": "github", + "query": "checkGhAuth", + "expect": [ + "src/main/githubManager.js" + ], + "tags": [ + "symbol" + ] + }, + { + "id": "dev-007", + "family": "github", + "query": "GitHub paneli", + "expect": [ + "src/main/githubManager.js", + "src/renderer/githubPanel.js" + ], + "tags": [ + "turkish" + ] + }, + { + "id": "dev-008", + "family": "github", + "query": "githubManager modülü", + "expect": [ + "src/main/githubManager.js" + ], + "tags": [ + "basename", + "turkish" + ] + }, + { + "id": "dev-009", + "family": "github", + "query": "GİTHUB", + "expect": [ + "src/main/githubManager.js", + "src/renderer/githubPanel.js" + ], + "tags": [ + "casing", + "turkish" + ] + }, + { + "id": "dev-010", + "family": "terminal", + "query": "terminal", + "expect": [ + "src/main/pty.js", + "src/main/ptyManager.js", + "src/renderer/multiTerminalUI.js", + "src/renderer/terminal.js", + "src/renderer/terminalManager.js", + "src/renderer/terminalTabBar.js", + "src/renderer/terminalsView.js" + ], + "tags": [ + "curated" + ] + }, + { + "id": "dev-011", + "family": "terminal", + "query": "xterm", + "expect": [ + "src/main/pty.js", + "src/main/ptyManager.js", + "src/renderer/multiTerminalUI.js", + "src/renderer/terminal.js", + "src/renderer/terminalManager.js", + "src/renderer/terminalTabBar.js", + "src/renderer/terminalsView.js" + ], + "tags": [ + "synonym" + ] + }, + { + "id": "dev-012", + "family": "terminal", + "query": "ptyManager.js", + "expect": [ + "src/main/ptyManager.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "dev-013", + "family": "terminal", + "query": "getForegroundCommand", + "expect": [ + "src/main/ptyManager.js" + ], + "tags": [ + "symbol" + ] + }, + { + "id": "dev-014", + "family": "terminal", + "query": "terminalInput", + "expect": [ + "src/renderer/terminalInput.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "dev-015", + "family": "terminal", + "query": "terminalChipNotice", + "expect": [ + "src/renderer/terminalChipNotice.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "dev-016", + "family": "terminal", + "query": "terminal yöneticisi", + "expect": [ + "src/main/pty.js", + "src/main/ptyManager.js", + "src/renderer/multiTerminalUI.js", + "src/renderer/terminal.js", + "src/renderer/terminalManager.js", + "src/renderer/terminalTabBar.js", + "src/renderer/terminalsView.js" + ], + "tags": [ + "turkish" + ] + }, + { + "id": "dev-017", + "family": "terminal", + "query": "TERMİNAL", + "expect": [ + "src/main/pty.js", + "src/main/ptyManager.js", + "src/renderer/multiTerminalUI.js", + "src/renderer/terminal.js", + "src/renderer/terminalManager.js", + "src/renderer/terminalTabBar.js", + "src/renderer/terminalsView.js" + ], + "tags": [ + "casing", + "turkish" + ] + }, + { + "id": "dev-018", + "family": "terminal", + "query": "ptyManager dosyasında", + "expect": [ + "src/main/ptyManager.js" + ], + "tags": [ + "basename", + "turkish" + ] + }, + { + "id": "dev-019", + "family": "terminal", + "query": "escapePathForShell", + "expect": [ + "src/renderer/terminalManager.js" + ], + "tags": [ + "shell", + "symbol" + ], + "hook": { + "tool": "Bash", + "command": "grep -rn \"escapePathForShell\" src/" + } + }, + { + "id": "dev-020", + "family": "terminal", + "query": "envPath", + "expect": [ + "src/main/envPath.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "dev-021", + "family": "activity", + "query": "activity", + "expect": [ + "scripts/activity-log.js", + "src/main/activityLog.js", + "src/renderer/activityPanel.js", + "src/shared/activityEvents.js" + ], + "tags": [ + "concept" + ] + }, + { + "id": "dev-022", + "family": "activity", + "query": "activityPanel", + "expect": [ + "src/renderer/activityPanel.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "dev-023", + "family": "activity", + "query": "activity-log.js", + "expect": [ + "scripts/activity-log.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "dev-024", + "family": "activity", + "query": "projectKey", + "expect": [ + "scripts/activity-log.js" + ], + "tags": [ + "singleton", + "symbol" + ] + }, + { + "id": "dev-025", + "family": "activity", + "query": "aktivite günlüğü", + "expect": [ + "scripts/activity-log.js", + "src/main/activityLog.js" + ], + "tags": [ + "turkish", + "turkish-pure" + ] + }, + { + "id": "dev-026", + "family": "activity", + "query": "redact", + "expect": [ + "scripts/redact.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "dev-027", + "family": "activity", + "query": "redactValue", + "expect": [ + "scripts/redact.js" + ], + "tags": [ + "singleton", + "symbol" + ] + }, + { + "id": "dev-028", + "family": "git", + "query": "git-branches", + "expect": [ + "src/main/gitBranchRefs.js", + "src/main/gitBranchesManager.js", + "src/renderer/statusBar/branchPicker.js" + ], + "tags": [ + "curated" + ] + }, + { + "id": "dev-029", + "family": "git", + "query": "worktree", + "expect": [ + "src/main/gitBranchRefs.js", + "src/main/gitBranchesManager.js", + "src/renderer/statusBar/branchPicker.js" + ], + "tags": [ + "synonym" + ] + }, + { + "id": "dev-030", + "family": "git", + "query": "diff", + "expect": [ + "src/main/gitDiffManager.js", + "src/main/gitStatusManager.js", + "src/renderer/diffSection.js", + "src/renderer/diffViewer.js", + "src/renderer/gitChangesPanel.js" + ], + "tags": [ + "synonym" + ] + }, + { + "id": "dev-031", + "family": "git", + "query": "gitExclude", + "expect": [ + "src/main/gitExclude.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "dev-032", + "family": "git", + "query": "isValidBranchName", + "expect": [ + "src/shared/gitRefNames.js" + ], + "tags": [ + "symbol" + ] + }, + { + "id": "dev-033", + "family": "git", + "query": "branchPickerModel", + "expect": [ + "src/renderer/statusBar/branchPickerModel.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "dev-034", + "family": "git", + "query": "git dalı değiştirme", + "expect": [ + "src/main/gitBranchRefs.js", + "src/main/gitBranchesManager.js", + "src/main/gitExclude.js", + "src/main/gitSharing.js", + "src/renderer/statusBar/branchPicker.js", + "src/shared/gitRefNames.js" + ], + "tags": [ + "turkish" + ] + }, + { + "id": "dev-035", + "family": "git", + "query": "gitSharing.js", + "expect": [ + "src/main/gitSharing.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "dev-036", + "family": "git", + "query": "parseBranchLine", + "expect": [ + "src/main/gitBranchRefs.js" + ], + "tags": [ + "shell", + "symbol" + ], + "hook": { + "tool": "Bash", + "command": "grep -rn \"parseBranchLine\" src" + } + }, + { + "id": "dev-037", + "family": "logging", + "query": "logger.js", + "expect": [ + "src/main/logger.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "dev-038", + "family": "logging", + "query": "rotateArchives", + "expect": [ + "src/main/logger.js" + ], + "tags": [ + "singleton", + "symbol" + ] + }, + { + "id": "dev-039", + "family": "logging", + "query": "promptLogger", + "expect": [ + "src/main/promptLogger.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "dev-040", + "family": "logging", + "query": "perfMonitor", + "expect": [ + "src/main/perfMonitor.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "dev-041", + "family": "logging", + "query": "opStart", + "expect": [ + "src/main/perfMonitor.js" + ], + "tags": [ + "singleton", + "symbol" + ] + }, + { + "id": "dev-042", + "family": "fs", + "query": "fsSafe", + "expect": [ + "src/main/fsSafe.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "dev-043", + "family": "fs", + "query": "writeFileAtomic", + "expect": [ + "src/main/fsSafe.js" + ], + "tags": [ + "singleton", + "symbol" + ] + }, + { + "id": "dev-044", + "family": "fs", + "query": "frameStore", + "expect": [ + "src/main/frameStore.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "dev-045", + "family": "fs", + "query": "src/main/frameStore.js", + "expect": [ + "src/main/frameStore.js" + ], + "tags": [ + "path" + ] + }, + { + "id": "dev-046", + "family": "fs", + "query": "frameStore dosyası", + "expect": [ + "src/main/frameStore.js" + ], + "tags": [ + "basename", + "turkish" + ] + }, + { + "id": "dev-047", + "family": "fs", + "query": "readJsonWithRecovery", + "expect": [ + "src/main/fsSafe.js" + ], + "tags": [ + "shell", + "singleton", + "symbol" + ], + "hook": { + "tool": "Bash", + "command": "grep -rn \"readJsonWithRecovery\" src/" + } + }, + { + "id": "dev-048", + "family": "lang", + "query": "rust extractor", + "expect": [ + "scripts/lang/rust.js" + ], + "tags": [ + "description", + "singleton" + ] + }, + { + "id": "dev-049", + "family": "lang", + "query": "lang/python.js", + "expect": [ + "scripts/lang/python.js" + ], + "tags": [ + "path", + "singleton" + ] + }, + { + "id": "dev-050", + "family": "lang", + "query": "extractFunctions", + "expect": [ + "scripts/lang/go.js", + "scripts/lang/javascript.js", + "scripts/lang/markdown.js", + "scripts/lang/python.js", + "scripts/lang/rust.js" + ], + "tags": [ + "ambiguous", + "singleton", + "symbol" + ] + }, + { + "id": "dev-051", + "family": "lang", + "query": "markdown çıkarıcı", + "expect": [ + "scripts/lang/markdown.js" + ], + "tags": [ + "singleton", + "turkish" + ] + }, + { + "id": "dev-052", + "family": "sample", + "query": "requireAuth", + "expect": [ + "src/templates/sample-project/src/auth/session.js" + ], + "tags": [ + "symbol" + ] + }, + { + "id": "dev-053", + "family": "sample", + "query": "googleStrategy", + "expect": [ + "src/templates/sample-project/src/auth/oauth.js" + ], + "tags": [ + "singleton", + "symbol" + ] + }, + { + "id": "dev-054", + "family": "sample", + "query": "TaskList", + "expect": [ + "src/templates/sample-project/src/ui/components/TaskList.jsx" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "dev-055", + "family": "telemetry", + "query": "telemetry", + "expect": [ + "src/main/telemetry.js", + "src/main/telemetryEvents.js", + "src/renderer/telemetryNotice.js" + ], + "tags": [ + "curated" + ] + }, + { + "id": "dev-056", + "family": "telemetry", + "query": "analytics", + "expect": [ + "src/main/telemetry.js", + "src/main/telemetryEvents.js", + "src/renderer/telemetryNotice.js" + ], + "tags": [ + "synonym" + ] + }, + { + "id": "dev-057", + "family": "telemetry", + "query": "telemetryEvents", + "expect": [ + "src/main/telemetryEvents.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "dev-058", + "family": "telemetry", + "query": "createRateLimiter", + "expect": [ + "src/main/telemetryEvents.js" + ], + "tags": [ + "symbol" + ] + }, + { + "id": "dev-059", + "family": "telemetry", + "query": "telemetri bildirimi", + "expect": [ + "src/renderer/telemetryNotice.js" + ], + "tags": [ + "turkish", + "turkish-pure" + ] + }, + { + "id": "dev-060", + "family": "eval", + "query": "run-eval", + "expect": [ + "scripts/eval/run-eval.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "dev-061", + "family": "eval", + "query": "scoreTranscript", + "expect": [ + "scripts/eval/score.js" + ], + "tags": [ + "singleton", + "symbol" + ] + }, + { + "id": "dev-062", + "family": "negative", + "query": "kubernetes", + "expect": [], + "tags": [ + "negative" + ] + }, + { + "id": "dev-063", + "family": "negative", + "query": "graphql resolver", + "expect": [], + "tags": [ + "negative" + ] + }, + { + "id": "dev-064", + "family": "negative", + "query": "stripe", + "expect": [], + "tags": [ + "negative" + ] + }, + { + "id": "dev-065", + "family": "negative", + "query": "ödeme sayfası", + "expect": [], + "tags": [ + "negative", + "turkish" + ] + }, + { + "id": "dev-066", + "family": "negative", + "query": "redux store", + "expect": [], + "tags": [ + "negative" + ] + }, + { + "id": "dev-067", + "family": "negative", + "query": "webpack config", + "expect": [], + "tags": [ + "negative" + ] + }, + { + "id": "dev-068", + "family": "negative", + "query": "mongodb", + "expect": [], + "tags": [ + "negative" + ] + }, + { + "id": "dev-069", + "family": "negative", + "query": "fatura", + "expect": [], + "tags": [ + "negative", + "turkish" + ] + }, + { + "id": "dev-070", + "family": "negative", + "query": "useEffect", + "expect": [], + "tags": [ + "negative" + ] + }, + { + "id": "dev-071", + "family": "negative", + "query": "console.log", + "expect": [], + "tags": [ + "negative", + "noise", + "shell" + ], + "hook": { + "tool": "Bash", + "command": "grep -rn \"console.log\" src/" + } + }, + { + "id": "dev-072", + "family": "negative", + "query": "foo|bar", + "expect": [], + "tags": [ + "negative", + "noise" + ] + } + ] + }, + "heldOut": { + "sha256": "39285f9aebf35af4e4f2f912718a6a1a691ca60ceb420c418444072cca12d23b", + "cases": [ + { + "id": "ho-001", + "family": "tasks", + "query": "tasks", + "expect": [ + "src/main/tasksManager.js", + "src/renderer/taskConfirmModal.js", + "src/renderer/taskInfoModal.js", + "src/renderer/taskRunModal.js", + "src/renderer/taskSection.js", + "src/renderer/tasksDashboard.js", + "src/renderer/tasksPanel.js" + ], + "tags": [ + "curated" + ] + }, + { + "id": "ho-002", + "family": "tasks", + "query": "todo", + "expect": [ + "src/main/tasksManager.js", + "src/renderer/taskConfirmModal.js", + "src/renderer/taskInfoModal.js", + "src/renderer/taskRunModal.js", + "src/renderer/taskSection.js", + "src/renderer/tasksDashboard.js", + "src/renderer/tasksPanel.js" + ], + "tags": [ + "synonym" + ] + }, + { + "id": "ho-003", + "family": "tasks", + "query": "tasksManager", + "expect": [ + "src/main/tasksManager.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "ho-004", + "family": "tasks", + "query": "flattenLegacyTasks", + "expect": [ + "src/main/tasksManager.js" + ], + "tags": [ + "symbol" + ] + }, + { + "id": "ho-005", + "family": "tasks", + "query": "taskConfirmModal.js", + "expect": [ + "src/renderer/taskConfirmModal.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "ho-006", + "family": "tasks", + "query": "taskRunModal", + "expect": [ + "src/renderer/taskRunModal.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "ho-007", + "family": "tasks", + "query": "görev paneli", + "expect": [ + "src/renderer/tasksPanel.js" + ], + "tags": [ + "turkish", + "turkish-pure" + ] + }, + { + "id": "ho-008", + "family": "tasks", + "query": "görev çalıştırma modalı", + "expect": [ + "src/renderer/taskRunModal.js" + ], + "tags": [ + "turkish", + "turkish-pure" + ] + }, + { + "id": "ho-009", + "family": "specs", + "query": "specs", + "expect": [ + "src/main/specManager.js", + "src/renderer/specPanel.js", + "src/renderer/specPanelResize.js", + "src/renderer/specSection.js", + "src/renderer/specsDashboard.js" + ], + "tags": [ + "curated" + ] + }, + { + "id": "ho-010", + "family": "specs", + "query": "spec-driven", + "expect": [ + "src/main/specManager.js", + "src/renderer/specPanel.js", + "src/renderer/specPanelResize.js", + "src/renderer/specSection.js", + "src/renderer/specsDashboard.js" + ], + "tags": [ + "synonym" + ] + }, + { + "id": "ho-011", + "family": "specs", + "query": "specNextAction", + "expect": [ + "src/renderer/specNextAction.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-012", + "family": "specs", + "query": "repairSpecStatus", + "expect": [ + "src/main/specManager.js" + ], + "tags": [ + "symbol" + ] + }, + { + "id": "ho-013", + "family": "specs", + "query": "renderNextActionBar", + "expect": [ + "src/renderer/specNextAction.js" + ], + "tags": [ + "singleton", + "symbol" + ] + }, + { + "id": "ho-014", + "family": "specs", + "query": "spec-index.js", + "expect": [ + "scripts/spec-index.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-015", + "family": "specs", + "query": "parseFootprint", + "expect": [ + "scripts/spec-index.js" + ], + "tags": [ + "singleton", + "symbol" + ] + }, + { + "id": "ho-016", + "family": "specs", + "query": "spec planı", + "expect": [ + "src/main/specManager.js", + "src/renderer/specPanel.js", + "src/renderer/specPanelResize.js", + "src/renderer/specSection.js", + "src/renderer/specsDashboard.js" + ], + "tags": [ + "turkish" + ] + }, + { + "id": "ho-017", + "family": "specs", + "query": "spec-context", + "expect": [ + "scripts/spec-context.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-018", + "family": "specs", + "query": "parseFrontMatter", + "expect": [ + "scripts/spec-index.js" + ], + "tags": [ + "shell", + "singleton", + "symbol" + ], + "hook": { + "tool": "Bash", + "command": "rg -n \"parseFrontMatter\" scripts/" + } + }, + { + "id": "ho-019", + "family": "specs", + "query": "specManager içinde repairSpecStatus", + "expect": [ + "src/main/specManager.js" + ], + "tags": [ + "symbol", + "turkish" + ] + }, + { + "id": "ho-020", + "family": "structure", + "query": "structure", + "expect": [ + "src/main/structureBootstrap.js", + "src/main/structureLifecycle.js", + "src/renderer/structureMap.js" + ], + "tags": [ + "curated" + ] + }, + { + "id": "ho-021", + "family": "structure", + "query": "codebase-map", + "expect": [ + "src/main/structureBootstrap.js", + "src/main/structureLifecycle.js", + "src/renderer/structureMap.js" + ], + "tags": [ + "synonym" + ] + }, + { + "id": "ho-022", + "family": "structure", + "query": "structure-commit.js", + "expect": [ + "scripts/structure-commit.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-023", + "family": "structure", + "query": "publishStaged", + "expect": [ + "scripts/structure-commit.js" + ], + "tags": [ + "singleton", + "symbol" + ] + }, + { + "id": "ho-024", + "family": "structure", + "query": "resolveReadPath", + "expect": [ + "scripts/structure-read.js" + ], + "tags": [ + "singleton", + "symbol" + ] + }, + { + "id": "ho-025", + "family": "structure", + "query": "scripts/structure-state.js", + "expect": [ + "scripts/structure-state.js" + ], + "tags": [ + "path", + "singleton" + ] + }, + { + "id": "ho-026", + "family": "structure", + "query": "structureLifecycle", + "expect": [ + "scripts/structure-lifecycle.js", + "src/main/structureLifecycle.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-027", + "family": "structure", + "query": "update-structure", + "expect": [ + "scripts/update-structure.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-028", + "family": "structure", + "query": "yapı haritası", + "expect": [ + "src/renderer/structureMap.js" + ], + "tags": [ + "turkish", + "turkish-pure" + ] + }, + { + "id": "ho-029", + "family": "structure", + "query": "structure-ignore", + "expect": [ + "scripts/structure-ignore.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-030", + "family": "structure", + "query": "artifactSignature", + "expect": [ + "scripts/structure-read.js" + ], + "tags": [ + "shell", + "singleton", + "symbol" + ], + "hook": { + "tool": "Bash", + "command": "grep -rn \"artifactSignature\" scripts" + } + }, + { + "id": "ho-031", + "family": "structure", + "query": "createScheduler", + "expect": [ + "scripts/structure-lifecycle.js" + ], + "tags": [ + "singleton", + "symbol" + ] + }, + { + "id": "ho-032", + "family": "structure", + "query": "structureBootstrap nerede", + "expect": [ + "src/main/structureBootstrap.js" + ], + "tags": [ + "basename", + "turkish" + ] + }, + { + "id": "ho-033", + "family": "structure", + "query": "**/structure-read.js", + "expect": [ + "scripts/structure-read.js" + ], + "tags": [ + "glob", + "path", + "singleton" + ], + "hook": { + "tool": "Glob", + "pattern": "**/structure-read.js" + } + }, + { + "id": "ho-034", + "family": "settings", + "query": "settings", + "expect": [ + "src/main/userSettings.js", + "src/renderer/frameSettingsModal.js", + "src/renderer/projectSettingsModal.js", + "src/renderer/settingsOverlay.js" + ], + "tags": [ + "curated" + ] + }, + { + "id": "ho-035", + "family": "settings", + "query": "preferences", + "expect": [ + "src/main/userSettings.js", + "src/renderer/frameSettingsModal.js", + "src/renderer/projectSettingsModal.js", + "src/renderer/settingsOverlay.js" + ], + "tags": [ + "synonym" + ] + }, + { + "id": "ho-036", + "family": "settings", + "query": "settingsOverlay", + "expect": [ + "src/renderer/settingsOverlay.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-037", + "family": "settings", + "query": "userSettings", + "expect": [ + "src/main/userSettings.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "ho-038", + "family": "settings", + "query": "ayarlar", + "expect": [ + "src/main/userSettings.js", + "src/renderer/frameSettingsModal.js", + "src/renderer/projectSettingsModal.js", + "src/renderer/settingsOverlay.js" + ], + "tags": [ + "turkish", + "turkish-pure" + ] + }, + { + "id": "ho-039", + "family": "settings", + "query": "projectSettingsModal dosyası", + "expect": [ + "src/renderer/projectSettingsModal.js" + ], + "tags": [ + "basename", + "turkish" + ] + }, + { + "id": "ho-040", + "family": "ui", + "query": "uiZoom", + "expect": [ + "src/main/uiZoom.js", + "src/shared/uiZoom.js" + ], + "tags": [ + "ambiguous", + "basename" + ] + }, + { + "id": "ho-041", + "family": "ui", + "query": "factorFor", + "expect": [ + "src/shared/uiZoom.js" + ], + "tags": [ + "symbol" + ] + }, + { + "id": "ho-042", + "family": "ui", + "query": "zoom", + "expect": [ + "src/main/uiZoom.js", + "src/shared/uiZoom.js" + ], + "tags": [ + "concept" + ] + }, + { + "id": "ho-043", + "family": "ui", + "query": "tooltip", + "expect": [ + "src/renderer/tooltip.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-044", + "family": "ui", + "query": "themes", + "expect": [ + "src/renderer/themes.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-045", + "family": "ui", + "query": "terminalTheme", + "expect": [ + "src/renderer/terminalManager.js", + "src/renderer/themes.js" + ], + "tags": [ + "symbol" + ] + }, + { + "id": "ho-046", + "family": "ui", + "query": "uiZoom ayarı", + "expect": [ + "src/main/uiZoom.js", + "src/shared/uiZoom.js" + ], + "tags": [ + "basename", + "turkish" + ] + }, + { + "id": "ho-047", + "family": "home", + "query": "homeData", + "expect": [ + "src/renderer/home/homeData.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-048", + "family": "home", + "query": "widgetShell", + "expect": [ + "src/renderer/home/widgetShell.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-049", + "family": "home", + "query": "agentRows", + "expect": [ + "src/renderer/home/agentRows.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "ho-050", + "family": "home", + "query": "sessionRows", + "expect": [ + "src/renderer/home/sessionRows.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "ho-051", + "family": "home", + "query": "registry.js", + "expect": [ + "src/renderer/home/registry.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-052", + "family": "home", + "query": "resolveLayout", + "expect": [ + "src/renderer/home/registry.js" + ], + "tags": [ + "singleton", + "symbol" + ] + }, + { + "id": "ho-053", + "family": "home", + "query": "ana sayfa widget kaydı", + "expect": [ + "src/renderer/home/registry.js", + "src/renderer/home/widgetShell.js" + ], + "tags": [ + "singleton", + "turkish" + ] + }, + { + "id": "ho-054", + "family": "onboarding", + "query": "onboardingGate", + "expect": [ + "src/renderer/onboarding/onboardingGate.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "ho-055", + "family": "onboarding", + "query": "tourSteps", + "expect": [ + "src/renderer/tour/tourSteps.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "ho-056", + "family": "onboarding", + "query": "shouldAutoStart", + "expect": [ + "src/renderer/tour/tourSteps.js" + ], + "tags": [ + "symbol" + ] + }, + { + "id": "ho-057", + "family": "onboarding", + "query": "guidedTour", + "expect": [ + "src/renderer/guidedTour.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "ho-058", + "family": "onboarding", + "query": "rehberli tur", + "expect": [ + "src/renderer/guidedTour.js" + ], + "tags": [ + "turkish", + "turkish-pure" + ] + }, + { + "id": "ho-059", + "family": "onboarding", + "query": "sampleBanner", + "expect": [ + "src/renderer/sampleBanner.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-060", + "family": "update", + "query": "updateChecker", + "expect": [ + "src/main/updateChecker.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-061", + "family": "update", + "query": "isNewerVersion", + "expect": [ + "src/main/updateChecker.js" + ], + "tags": [ + "singleton", + "symbol" + ] + }, + { + "id": "ho-062", + "family": "update", + "query": "güncelleme kontrolü", + "expect": [ + "src/main/updateChecker.js" + ], + "tags": [ + "singleton", + "turkish", + "turkish-pure" + ] + }, + { + "id": "ho-063", + "family": "project", + "query": "frameProject", + "expect": [ + "src/main/frameProject.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "ho-064", + "family": "project", + "query": "workspace", + "expect": [ + "src/main/workspace.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-065", + "family": "project", + "query": "dialogs.js", + "expect": [ + "src/main/dialogs.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-066", + "family": "project", + "query": "cloneGithubRepo", + "expect": [ + "src/main/dialogs.js" + ], + "tags": [ + "singleton", + "symbol" + ] + }, + { + "id": "ho-067", + "family": "project", + "query": "projectStart", + "expect": [ + "src/renderer/projectStart.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "ho-068", + "family": "project", + "query": "openProjectModal", + "expect": [ + "src/renderer/openProjectModal.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "ho-069", + "family": "project", + "query": "proje başlatma", + "expect": [ + "src/renderer/projectStart.js" + ], + "tags": [ + "turkish" + ] + }, + { + "id": "ho-070", + "family": "project", + "query": "detect-project", + "expect": [ + "scripts/detect-project.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-071", + "family": "project", + "query": "detectJavaScript", + "expect": [ + "scripts/detect-project.js" + ], + "tags": [ + "singleton", + "symbol" + ] + }, + { + "id": "ho-072", + "family": "app", + "query": "menu.js", + "expect": [ + "src/main/menu.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-073", + "family": "app", + "query": "buildThemeSubmenu", + "expect": [ + "src/main/menu.js" + ], + "tags": [ + "singleton", + "symbol" + ] + }, + { + "id": "ho-074", + "family": "app", + "query": "appHeader", + "expect": [ + "src/renderer/appHeader.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-075", + "family": "app", + "query": "appLoader", + "expect": [ + "src/renderer/appLoader.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-076", + "family": "app", + "query": "armFailsafe", + "expect": [ + "src/renderer/appLoader.js" + ], + "tags": [ + "singleton", + "symbol" + ] + }, + { + "id": "ho-077", + "family": "app", + "query": "index.js", + "expect": [ + "src/main/index.js", + "src/renderer/index.js", + "src/templates/sample-project/src/index.js", + "test/fixtures/js-src-app/src/index.js", + "test/fixtures/pnpm-monorepo/packages/core/index.js" + ], + "tags": [ + "ambiguous", + "basename", + "singleton" + ] + }, + { + "id": "ho-078", + "family": "app", + "query": "main.js", + "expect": [ + "main.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-079", + "family": "app", + "query": "renderer.js", + "expect": [ + "renderer.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-080", + "family": "hints", + "query": "docs-hint", + "expect": [ + "scripts/docs-hint.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-081", + "family": "hints", + "query": "module-hint", + "expect": [ + "scripts/module-hint.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-082", + "family": "hints", + "query": "spec-command-hint", + "expect": [ + "scripts/spec-command-hint.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-083", + "family": "hints", + "query": "docsHealth", + "expect": [ + "src/shared/docsHealth.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "ho-084", + "family": "hints", + "query": "namedPaths", + "expect": [ + "src/shared/docsHealth.js" + ], + "tags": [ + "symbol" + ] + }, + { + "id": "ho-085", + "family": "hints", + "query": "docsManagedBlock", + "expect": [ + "src/shared/docsManagedBlock.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "ho-086", + "family": "hints", + "query": "ipucu betiği", + "expect": [ + "scripts/docs-hint.js", + "scripts/module-hint.js", + "scripts/spec-command-hint.js", + "scripts/spec-hint.js" + ], + "tags": [ + "singleton", + "turkish", + "turkish-pure" + ] + }, + { + "id": "ho-087", + "family": "hints", + "query": "İPC kanalları", + "expect": [ + "src/shared/ipcChannels.js" + ], + "tags": [ + "casing", + "turkish" + ] + }, + { + "id": "ho-088", + "family": "commands", + "query": "palette", + "expect": [ + "src/renderer/cheatSheet.js", + "src/renderer/commandPalette.js", + "src/renderer/commandRegistry.js" + ], + "tags": [ + "synonym" + ] + }, + { + "id": "ho-089", + "family": "commands", + "query": "commandRegistry", + "expect": [ + "src/renderer/commandRegistry.js" + ], + "tags": [ + "basename" + ] + }, + { + "id": "ho-090", + "family": "commands", + "query": "paletteSources", + "expect": [ + "src/renderer/paletteSources.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-091", + "family": "commands", + "query": "runById", + "expect": [ + "src/renderer/commandRegistry.js" + ], + "tags": [ + "symbol" + ] + }, + { + "id": "ho-092", + "family": "commands", + "query": "komut paleti", + "expect": [ + "src/renderer/commandPalette.js" + ], + "tags": [ + "turkish", + "turkish-pure" + ] + }, + { + "id": "ho-093", + "family": "commands", + "query": "commandStaging", + "expect": [ + "src/main/commandStaging.js" + ], + "tags": [ + "basename", + "singleton" + ] + }, + { + "id": "ho-094", + "family": "commands", + "query": "stageCommandFiles", + "expect": [ + "src/main/commandStaging.js" + ], + "tags": [ + "singleton", + "symbol" + ] + }, + { + "id": "ho-095", + "family": "mutation", + "query": "updateChecker", + "expect": [], + "tags": [ + "negative", + "removed" + ], + "mutate": { + "remove": "src/main/updateChecker.js" + } + }, + { + "id": "ho-096", + "family": "mutation", + "query": "perfMonitor", + "expect": [], + "tags": [ + "negative", + "renamed" + ], + "mutate": { + "rename": [ + "src/main/perfMonitor.js", + "src/main/perfMonitorNext.js" + ] + } + }, + { + "id": "ho-097", + "family": "mutation", + "query": "brandNewWidget", + "expect": [], + "tags": [ + "negative", + "stale-map" + ], + "mutate": { + "add": [ + "src/main/brandNewWidget.js", + "// Brand new widget\n" + ] + } + }, + { + "id": "ho-098", + "family": "negative", + "query": "payment gateway", + "expect": [], + "tags": [ + "negative" + ] + }, + { + "id": "ho-099", + "family": "negative", + "query": "veritabanı göçü", + "expect": [], + "tags": [ + "negative", + "turkish" + ] + }, + { + "id": "ho-100", + "family": "negative", + "query": "jwt refresh token", + "expect": [], + "tags": [ + "negative" + ] + }, + { + "id": "ho-101", + "family": "negative", + "query": "docker compose", + "expect": [], + "tags": [ + "negative" + ] + }, + { + "id": "ho-102", + "family": "negative", + "query": "TODO", + "expect": [], + "tags": [ + "negative" + ] + }, + { + "id": "ho-103", + "family": "negative", + "query": "kullanıcı profili", + "expect": [], + "tags": [ + "negative", + "turkish" + ] + }, + { + "id": "ho-104", + "family": "negative", + "query": "e-posta doğrulama", + "expect": [], + "tags": [ + "negative", + "turkish" + ] + }, + { + "id": "ho-105", + "family": "negative", + "query": "i18n çeviri", + "expect": [], + "tags": [ + "negative", + "turkish" + ] + }, + { + "id": "ho-106", + "family": "negative", + "query": "lorem ipsum", + "expect": [], + "tags": [ + "negative" + ] + }, + { + "id": "ho-107", + "family": "negative", + "query": "xyzzy", + "expect": [], + "tags": [ + "negative" + ] + }, + { + "id": "ho-108", + "family": "negative", + "query": "elasticsearch index", + "expect": [], + "tags": [ + "negative" + ] + }, + { + "id": "ho-109", + "family": "negative", + "query": "swagger openapi", + "expect": [], + "tags": [ + "negative" + ] + }, + { + "id": "ho-110", + "family": "negative", + "query": "tailwind", + "expect": [], + "tags": [ + "negative" + ] + }, + { + "id": "ho-111", + "family": "negative", + "query": "sentry dsn", + "expect": [], + "tags": [ + "negative" + ] + }, + { + "id": "ho-112", + "family": "negative", + "query": "sepet", + "expect": [], + "tags": [ + "negative", + "turkish" + ] + }, + { + "id": "ho-113", + "family": "negative", + "query": "prisma schema", + "expect": [], + "tags": [ + "negative" + ] + }, + { + "id": "ho-114", + "family": "negative", + "query": "grpc proto", + "expect": [], + "tags": [ + "negative" + ] + }, + { + "id": "ho-115", + "family": "negative", + "query": "android manifest", + "expect": [], + "tags": [ + "negative" + ] + }, + { + "id": "ho-116", + "family": "negative", + "query": "kafka consumer", + "expect": [], + "tags": [ + "negative" + ] + }, + { + "id": "ho-117", + "family": "negative", + "query": "vue router", + "expect": [], + "tags": [ + "negative" + ] + }, + { + "id": "ho-118", + "family": "negative", + "query": "fatura kesme", + "expect": [], + "tags": [ + "negative", + "turkish" + ] + }, + { + "id": "ho-119", + "family": "negative", + "query": "pyc", + "expect": [], + "tags": [ + "negative", + "shell" + ], + "hook": { + "tool": "Bash", + "command": "find . -name '*.pyc'" + } + }, + { + "id": "ho-120", + "family": "negative", + "query": "password", + "expect": [], + "tags": [ + "negative", + "shell" + ], + "hook": { + "tool": "Bash", + "command": "grep -rn \"password\" ." + } + }, + { + "id": "ho-121", + "family": "negative", + "query": "async function", + "expect": [], + "tags": [ + "negative", + "noise", + "shell" + ], + "hook": { + "tool": "Bash", + "command": "rg \"async function\"" + } + }, + { + "id": "ho-122", + "family": "negative", + "query": "\\bfunction\\s+\\w+", + "expect": [], + "tags": [ + "negative", + "noise" + ] + } + ] + } + } +} diff --git a/scripts/eval/run-retrieval.js b/scripts/eval/run-retrieval.js new file mode 100644 index 00000000..523be97f --- /dev/null +++ b/scripts/eval/run-retrieval.js @@ -0,0 +1,450 @@ +#!/usr/bin/env node +/** + * Retrieval benchmark (STR-03) — how well do `find-module` and the search + * hook find the right files, how often does the hook speak when it should + * not, and what does it cost? + * + * Deterministic and local: no agent, no network. The pinned commit is + * exported into a temp directory, its map is built with this checkout's + * `update-structure.js --full`, and every case in `retrieval-cases.json` is + * run through the real CLI and the real hook adapters (Claude Code payloads, + * and Codex's Bash-only payloads with `search codex`). Synthetic 1k/10k-file + * projects measure latency and index size at scale. + * + * Usage: + * node scripts/eval/run-retrieval.js # all splits, every available engine + * node scripts/eval/run-retrieval.js --split heldOut # development | heldOut | all + * node scripts/eval/run-retrieval.js --engine legacy # legacy | v2 | all + * node scripts/eval/run-retrieval.js --no-scale # skip the synthetic projects + * node scripts/eval/run-retrieval.js --json # one JSON report on stdout + * + * Gates are evaluated on the held-out split only (see README.md). Tests + * exercise the metric and gate math through the exported functions; the + * numbers themselves come from running this script. + */ + +'use strict'; + +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const crypto = require('crypto'); +const { spawnSync, execSync } = require('child_process'); + +const REPO_ROOT = path.join(__dirname, '..', '..'); +const SCRIPTS = path.join(REPO_ROOT, 'scripts'); +const CASES_FILE = path.join(__dirname, 'retrieval-cases.json'); + +const GATES = Object.freeze({ + exactRecall: 1.0, // path / basename / symbol / curated / synonym cases (not Turkish-mixed) + recallAt5: 0.9, // and never below legacy + precisionAt1: 0.9, + emittedPrecision: 0.98, // of the hook's emitted hints, every adapter + falseHintRate: 0.02, // on negative cases, every adapter + hookP95Ms: 50, // this repository and the 10k fixture + cliP95Ms: 150, + payloadChars: 1800 +}); + +const EXACT_TAGS = ['path', 'basename', 'symbol', 'curated', 'synonym']; + +/* ------------------------------- corpus ------------------------------- */ + +function digest(cases) { + return crypto.createHash('sha256').update(JSON.stringify(cases)).digest('hex'); +} + +/** Load the corpus; a split whose hash does not match was edited after freezing. */ +function loadCases(file = CASES_FILE) { + const corpus = JSON.parse(fs.readFileSync(file, 'utf8')); + for (const [name, split] of Object.entries(corpus.splits)) { + if (digest(split.cases) !== split.sha256) throw new Error(`split ${name} does not match its frozen hash`); + } + return corpus; +} + +/* ------------------------------- parsing ------------------------------ */ + +const PATH_LINE = /^ {2}(\S+)/; + +/** Candidate files from find-module output: the --json envelope, or the human listing. */ +function parseCli(stdout) { + const text = String(stdout || ''); + try { + const env = JSON.parse(text); + if (env && Array.isArray(env.candidates)) { + return env.candidates.filter((c) => !c.missing).map((c) => c.path); + } + } catch { /* human output */ } + const out = []; + for (const line of text.split('\n')) { + const m = PATH_LINE.exec(line); + if (!m || m[1].endsWith(':') || m[1].startsWith('…') || /⚠ file missing|⚠ missing/.test(line)) continue; + if (!out.includes(m[1])) out.push(m[1]); + } + return out; +} + +/** Hinted files and the raw context from a hook's stdout ('' = no hint). */ +function parseHook(stdout) { + const text = String(stdout || '').trim(); + if (!text) return { files: [], context: '' }; + let context = ''; + try { + context = JSON.parse(text).hookSpecificOutput.additionalContext || ''; + } catch { + return { files: [], context: '', invalid: true }; + } + const files = []; + for (const line of context.split('\n')) { + const m = PATH_LINE.exec(line); + if (!m || m[1].endsWith(':') || m[1].startsWith('…')) continue; + if (!files.includes(m[1])) files.push(m[1]); + } + return { files, context }; +} + +/* ------------------------------- metrics ------------------------------ */ + +function quantile(values, q) { + if (!values.length) return null; + const sorted = [...values].sort((a, b) => a - b); + const index = Math.min(sorted.length - 1, Math.max(0, Math.ceil(q * sorted.length) - 1)); + return sorted[index]; +} + +const ratio = (n, d) => (d ? n / d : null); + +/** + * results: [{ case, cli: [files], cliMs, hooks: { claude: { files, context, ms }, codex: {…} } }] + * Returns the summary for one engine over the given results. + */ +function summarize(results) { + const answerable = results.filter((r) => r.case.expect.length > 0); + const negatives = results.filter((r) => r.case.expect.length === 0); + const hit = (files, r) => files.some((f) => r.case.expect.includes(f)); + + const cliReturned = results.filter((r) => r.cli.length > 0); + const exact = answerable.filter((r) => r.case.tags.some((t) => EXACT_TAGS.includes(t)) && !r.case.tags.includes('turkish')); + const summary = { + cases: results.length, + answerable: answerable.length, + negatives: negatives.length, + recallAt5: ratio(answerable.filter((r) => hit(r.cli.slice(0, 5), r)).length, answerable.length), + exactRecall: ratio(exact.filter((r) => hit(r.cli.slice(0, 5), r)).length, exact.length), + exactCases: exact.length, + precisionAt1: ratio(cliReturned.filter((r) => r.case.expect.includes(r.cli[0])).length, cliReturned.length), + cliNegativeReturns: ratio(negatives.filter((r) => r.cli.length > 0).length, negatives.length), + cliP50Ms: quantile(results.map((r) => r.cliMs), 0.5), + cliP95Ms: quantile(results.map((r) => r.cliMs), 0.95), + hooks: {}, + strata: {} + }; + + const adapters = new Set(results.flatMap((r) => Object.keys(r.hooks || {}))); + for (const adapter of adapters) { + const runs = results.filter((r) => r.hooks[adapter]).map((r) => ({ r, h: r.hooks[adapter] })); + const emitted = runs.filter(({ h }) => h.files.length > 0); + const neg = runs.filter(({ r }) => r.case.expect.length === 0); + const ans = runs.filter(({ r }) => r.case.expect.length > 0); + const chars = runs.map(({ h }) => h.context.length); + summary.hooks[adapter] = { + runs: runs.length, + emitted: emitted.length, + emittedPrecision: ratio(emitted.filter(({ r, h }) => hit(h.files, r)).length, emitted.length), + falseHintRate: ratio(neg.filter(({ h }) => h.files.length > 0).length, neg.length), + hintRecall: ratio(ans.filter(({ r, h }) => hit(h.files, r)).length, ans.length), + abstention: ratio(ans.filter(({ h }) => h.files.length === 0).length, ans.length), + invalidOutputs: runs.filter(({ h }) => h.invalid).length, + p50Ms: quantile(runs.map(({ h }) => h.ms), 0.5), + p95Ms: quantile(runs.map(({ h }) => h.ms), 0.95), + maxChars: chars.length ? Math.max(...chars) : 0, + maxBytes: runs.length ? Math.max(...runs.map(({ h }) => Buffer.byteLength(h.context, 'utf8'))) : 0 + }; + } + + const tags = new Set(results.flatMap((r) => r.case.tags)); + for (const tag of [...tags].sort()) { + const rs = results.filter((r) => r.case.tags.includes(tag)); + const ans = rs.filter((r) => r.case.expect.length > 0); + const claude = rs.filter((r) => r.hooks && r.hooks.claude); + summary.strata[tag] = { + n: rs.length, + recallAt5: ratio(ans.filter((r) => hit(r.cli.slice(0, 5), r)).length, ans.length), + hintRecall: ratio(ans.filter((r) => r.hooks && r.hooks.claude && hit(r.hooks.claude.files, r)).length, ans.length), + falseHints: claude.filter((r) => r.case.expect.length === 0 && r.hooks.claude.files.length > 0).length + }; + } + return summary; +} + +/** + * The A5 gates for one engine's held-out summary. `baseline` is legacy's + * held-out summary (recall must not drop below it); `scale` the synthetic + * latency results. Returns { pass, gates: [{ name, value, limit, pass }] }. + */ +function evaluateGates(summary, baseline = null, scale = null, gates = GATES) { + const out = []; + const check = (name, value, limit, ok) => out.push({ name, value, limit, pass: value !== null && value !== undefined && ok }); + check('exact-recall', summary.exactRecall, gates.exactRecall, summary.exactRecall >= gates.exactRecall); + check('recall@5', summary.recallAt5, gates.recallAt5, summary.recallAt5 >= gates.recallAt5); + if (baseline) { + check('recall@5-vs-legacy', summary.recallAt5, baseline.recallAt5, summary.recallAt5 >= baseline.recallAt5); + } + check('precision@1', summary.precisionAt1, gates.precisionAt1, summary.precisionAt1 >= gates.precisionAt1); + for (const [adapter, h] of Object.entries(summary.hooks)) { + // a hook that never emitted has no precision to fail; false hints still count + check(`emitted-precision:${adapter}`, h.emittedPrecision === null ? 1 : h.emittedPrecision, gates.emittedPrecision, + (h.emittedPrecision === null ? 1 : h.emittedPrecision) >= gates.emittedPrecision); + check(`false-hints:${adapter}`, h.falseHintRate, gates.falseHintRate, h.falseHintRate <= gates.falseHintRate); + check(`hook-p95:${adapter}`, h.p95Ms, gates.hookP95Ms, h.p95Ms <= gates.hookP95Ms); + check(`payload:${adapter}`, h.maxChars, gates.payloadChars, h.maxChars <= gates.payloadChars); + } + check('cli-p95', summary.cliP95Ms, gates.cliP95Ms, summary.cliP95Ms <= gates.cliP95Ms); + if (scale) { + for (const s of scale) check(`hook-p95:${s.files}-files`, s.hookP95Ms, gates.hookP95Ms, s.hookP95Ms <= gates.hookP95Ms); + } + return { pass: out.every((g) => g.pass), gates: out }; +} + +/* ------------------------------- running ------------------------------ */ + +function timed(args, options) { + const start = process.hrtime.bigint(); + const r = spawnSync(process.execPath, args, { encoding: 'utf8', maxBuffer: 64 * 1024 * 1024, ...options }); + return { stdout: r.stdout || '', stderr: r.stderr || '', status: r.status, ms: Number(process.hrtime.bigint() - start) / 1e6 }; +} + +function baseEnv(project, activityHome) { + return { ...process.env, FRAME_PROJECT_ROOT: project, FRAME_ACTIVITY_HOME: activityHome }; +} + +function setEngine(project, engine) { + const file = path.join(project, '.frame', 'config.json'); + let config = {}; + try { + config = JSON.parse(fs.readFileSync(file, 'utf8')); + } catch { /* none yet */ } + config.project = config.project || {}; + config.project.retrieval = { engine }; + fs.mkdirSync(path.dirname(file), { recursive: true }); + fs.writeFileSync(file, JSON.stringify(config, null, 2)); +} + +/** Engines this checkout can run: legacy always; v2 once the shared engine exists. */ +function availableEngines() { + const engines = ['legacy']; + if (fs.existsSync(path.join(SCRIPTS, 'structure-retrieval.js'))) engines.push('v2'); + return engines; +} + +function cliSupportsJson(project, env) { + const r = timed([path.join(SCRIPTS, 'find-module.js'), '--json', 'probe'], { cwd: project, env }); + try { + return Boolean(JSON.parse(r.stdout).schema); + } catch { + return false; + } +} + +function hookPayloads(c, sessionId, project) { + const claude = c.hook && c.hook.tool === 'Bash' + ? { tool_name: 'Bash', tool_input: { command: c.hook.command } } + : c.hook && c.hook.tool === 'Glob' + ? { tool_name: 'Glob', tool_input: { pattern: c.hook.pattern } } + : { tool_name: 'Grep', tool_input: { pattern: c.query } }; + const codexCommand = c.hook && c.hook.tool === 'Bash' ? c.hook.command : `rg -n ${JSON.stringify(c.query)}`; + return { + claude: { session_id: `${sessionId}-claude`, cwd: project, hook_event_name: 'PreToolUse', ...claude }, + codex: { session_id: `${sessionId}-codex`, cwd: project, turn_id: 'bench', model: 'bench', hook_event_name: 'PreToolUse', + tool_name: 'Bash', tool_input: { command: codexCommand } } + }; +} + +function runCase(c, engine, project, env, json) { + const args = [path.join(SCRIPTS, 'find-module.js')]; + if (json) args.push('--json', `--retrieval=${engine}`); + args.push(c.query); + const cli = timed(args, { cwd: project, env }); + const payloads = hookPayloads(c, `bench-${engine}-${c.id}`, project); + const hooks = {}; + for (const [adapter, payload] of Object.entries(payloads)) { + const hookArgs = [path.join(SCRIPTS, 'module-hint.js'), 'search']; + if (adapter === 'codex') hookArgs.push('codex'); + const r = timed(hookArgs, { cwd: project, env, input: JSON.stringify(payload) }); + hooks[adapter] = { ...parseHook(r.stdout), ms: r.ms, status: r.status }; + } + return { case: c, cli: parseCli(cli.stdout), cliMs: cli.ms, hooks }; +} + +function applyMutation(base, mutate) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'frame-retrieval-mut-')); + fs.cpSync(base, dir, { recursive: true }); + if (mutate.remove) fs.rmSync(path.join(dir, mutate.remove)); + if (mutate.rename) fs.renameSync(path.join(dir, mutate.rename[0]), path.join(dir, mutate.rename[1])); + if (mutate.add) fs.writeFileSync(path.join(dir, mutate.add[0]), mutate.add[1]); + return dir; +} + +function buildMap(project, env) { + const start = process.hrtime.bigint(); + const r = spawnSync(process.execPath, [path.join(SCRIPTS, 'update-structure.js'), '--full'], { cwd: project, env, encoding: 'utf8' }); + if (r.status !== 0) throw new Error(`update-structure --full failed: ${r.stderr}`); + const ms = Number(process.hrtime.bigint() - start) / 1e6; + const size = (rel) => { + try { + return fs.statSync(path.join(project, rel)).size; + } catch { + return null; + } + }; + return { + buildMs: ms, + mapBytes: size('.frame/runtime/structure/working.json'), + lookupBytes: size('.frame/runtime/structure/lookup.json') + }; +} + +function exportPinned(commit) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'frame-retrieval-')); + execSync(`git archive ${commit} | tar -x -C ${JSON.stringify(dir)}`, { cwd: REPO_ROOT, stdio: ['ignore', 'ignore', 'pipe'] }); + return dir; +} + +/** A synthetic project of `count` JavaScript files in nested packages. */ +function syntheticProject(count) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), `frame-retrieval-${count}-`)); + for (let i = 0; i < count; i++) { + const pkg = `pkg${String(Math.floor(i / 100)).padStart(3, '0')}`; + const name = `module${String(i).padStart(5, '0')}`; + const file = path.join(dir, 'src', pkg, `${name}.js`); + fs.mkdirSync(path.dirname(file), { recursive: true }); + fs.writeFileSync(file, `// ${name} in ${pkg}\nfunction handle${i}(input) { return input; }\nmodule.exports = { handle${i} };\n`); + } + fs.mkdirSync(path.join(dir, '.frame'), { recursive: true }); + fs.writeFileSync(path.join(dir, '.frame', 'config.json'), JSON.stringify({ version: '1.0', name: `synthetic-${count}` })); + return dir; +} + +function runScale(engine, activityHome, sizes = [1000, 10000]) { + const out = []; + for (const count of sizes) { + const dir = syntheticProject(count); + try { + const env = baseEnv(dir, activityHome); + setEngine(dir, engine); + const build = buildMap(dir, env); + const queries = []; + for (let i = 0; i < 20; i++) queries.push(`module${String(Math.floor((i * count) / 20)).padStart(5, '0')}`); + for (let i = 0; i < 10; i++) queries.push(`absent${i}thing`); + const hookMs = []; + const cliMs = []; + queries.forEach((q, i) => { + const payload = { session_id: `scale-${engine}-${count}-${i}`, cwd: dir, tool_name: 'Grep', tool_input: { pattern: q } }; + hookMs.push(timed([path.join(SCRIPTS, 'module-hint.js'), 'search'], { cwd: dir, env, input: JSON.stringify(payload) }).ms); + cliMs.push(timed([path.join(SCRIPTS, 'find-module.js'), q], { cwd: dir, env }).ms); + }); + out.push({ files: count, ...build, hookP50Ms: quantile(hookMs, 0.5), hookP95Ms: quantile(hookMs, 0.95), cliP95Ms: quantile(cliMs, 0.95) }); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } + } + return out; +} + +function parseArgs(argv) { + const get = (flag) => { + const hit = argv.find((a) => a.startsWith(`${flag}=`)); + if (hit) return hit.slice(flag.length + 1); + const i = argv.indexOf(flag); + return i >= 0 ? argv[i + 1] : null; + }; + return { + split: get('--split') || 'all', + engine: get('--engine') || 'all', + scale: !argv.includes('--no-scale'), + json: argv.includes('--json') + }; +} + +const pct = (v) => (v === null || v === undefined ? '—' : `${(v * 100).toFixed(1)}%`); +const ms = (v) => (v === null || v === undefined ? '—' : `${v.toFixed(0)} ms`); + +function printReport(report) { + console.log(`Pinned ${report.pinnedCommit.slice(0, 7)} · ${report.machine}`); + console.log(`Map: ${report.build.mapBytes} bytes · lookup: ${report.build.lookupBytes ?? '—'} bytes · built in ${ms(report.build.buildMs)}\n`); + for (const [engine, splits] of Object.entries(report.engines)) { + for (const [split, s] of Object.entries(splits)) { + console.log(`[${engine} · ${split}] ${s.cases} cases (${s.answerable} answerable, ${s.negatives} negative)`); + console.log(` CLI recall@5 ${pct(s.recallAt5)} · exact ${pct(s.exactRecall)} (${s.exactCases}) · P@1 ${pct(s.precisionAt1)} · p50/p95 ${ms(s.cliP50Ms)}/${ms(s.cliP95Ms)}`); + for (const [adapter, h] of Object.entries(s.hooks)) { + console.log(` hook:${adapter.padEnd(6)} emitted ${h.emitted}/${h.runs} · precision ${pct(h.emittedPrecision)} · recall ${pct(h.hintRecall)} · false hints ${pct(h.falseHintRate)} · p50/p95 ${ms(h.p50Ms)}/${ms(h.p95Ms)} · max ${h.maxChars} chars`); + } + } + const g = report.gates[engine]; + if (g) { + console.log(` Gates (held-out): ${g.pass ? 'PASS' : 'FAIL'}`); + for (const x of g.gates) if (!x.pass) console.log(` ✗ ${x.name}: ${typeof x.value === 'number' && x.value <= 1 && x.limit <= 1 ? pct(x.value) : x.value} (limit ${x.limit})`); + } + if (report.scale[engine]) { + for (const s of report.scale[engine]) { + console.log(` scale ${s.files} files: build ${ms(s.buildMs)} · map ${s.mapBytes} B · lookup ${s.lookupBytes ?? '—'} B · hook p50/p95 ${ms(s.hookP50Ms)}/${ms(s.hookP95Ms)} · CLI p95 ${ms(s.cliP95Ms)}`); + } + } + console.log(''); + } +} + +function main() { + const opts = parseArgs(process.argv.slice(2)); + const corpus = loadCases(); + const splits = opts.split === 'all' ? Object.keys(corpus.splits) : [opts.split]; + const engines = availableEngines().filter((e) => opts.engine === 'all' || e === opts.engine); + const activityHome = fs.mkdtempSync(path.join(os.tmpdir(), 'frame-retrieval-activity-')); + const base = exportPinned(corpus.pinnedCommit); + const cleanup = [base, activityHome]; + const report = { + pinnedCommit: corpus.pinnedCommit, + machine: `${os.cpus()[0] && os.cpus()[0].model} · ${os.platform()} ${os.release()} · node ${process.version}`, + at: new Date().toISOString(), + engines: {}, + gates: {}, + scale: {} + }; + try { + report.build = buildMap(base, baseEnv(base, activityHome)); + for (const engine of engines) { + setEngine(base, engine); + const json = cliSupportsJson(base, baseEnv(base, activityHome)); + report.engines[engine] = {}; + for (const split of splits) { + const results = []; + for (const c of corpus.splits[split].cases) { + let project = base; + if (c.mutate) { + project = applyMutation(base, c.mutate); + cleanup.push(project); + } + results.push(runCase(c, engine, project, baseEnv(project, activityHome), json)); + } + report.engines[engine][split] = summarize(results); + } + if (opts.scale) report.scale[engine] = runScale(engine, activityHome); + } + const legacyHeld = report.engines.legacy && report.engines.legacy.heldOut; + for (const engine of engines) { + const held = report.engines[engine].heldOut; + if (!held) continue; + const scale = report.scale[engine] ? report.scale[engine].filter((s) => s.files === 10000) : null; + report.gates[engine] = evaluateGates(held, engine === 'legacy' ? null : legacyHeld, scale); + } + } finally { + for (const dir of cleanup) fs.rmSync(dir, { recursive: true, force: true }); + } + if (opts.json) process.stdout.write(`${JSON.stringify(report, null, 2)}\n`); + else printReport(report); +} + +if (require.main === module) main(); + +module.exports = { loadCases, digest, parseCli, parseHook, quantile, summarize, evaluateGates, GATES, EXACT_TAGS, CASES_FILE }; diff --git a/test/retrievalEval.test.js b/test/retrievalEval.test.js new file mode 100644 index 00000000..8156ca1b --- /dev/null +++ b/test/retrievalEval.test.js @@ -0,0 +1,145 @@ +/** + * STR-03 retrieval benchmark: the frozen corpus and the metric/gate math. + * The numbers themselves come from running scripts/eval/run-retrieval.js; + * these tests make sure what it reports means what the README says. + */ + +const { test } = require('node:test'); +const assert = require('node:assert/strict'); +const path = require('path'); +const { spawnSync } = require('child_process'); + +const bench = require('../scripts/eval/run-retrieval'); + +const REPO_ROOT = path.join(__dirname, '..'); + +/* ------------------------------- corpus ------------------------------- */ + +test('each split matches its frozen hash', () => { + const corpus = bench.loadCases(); + for (const split of Object.values(corpus.splits)) assert.equal(bench.digest(split.cases), split.sha256); +}); + +test('an edited split is refused', () => { + const fs = require('fs'); + const os = require('os'); + const corpus = JSON.parse(fs.readFileSync(bench.CASES_FILE, 'utf8')); + corpus.splits.heldOut.cases[0].expect = ['tuned.js']; + const file = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'frame-cases-')), 'cases.json'); + fs.writeFileSync(file, JSON.stringify(corpus)); + assert.throws(() => bench.loadCases(file), /heldOut does not match its frozen hash/); +}); + +test('the corpus has the planned size and coverage, with families disjoint across splits', () => { + const { splits } = bench.loadCases(); + const dev = splits.development.cases; + const held = splits.heldOut.cases; + const all = [...dev, ...held]; + const count = (tag) => all.filter((c) => c.tags.includes(tag)).length; + assert.ok(dev.length >= 60 && held.length >= 120 && all.length >= 180, `${dev.length}/${held.length}`); + assert.ok(count('turkish') >= 30); + assert.ok(count('negative') >= 30); + assert.ok(count('singleton') >= 20); + for (const tag of ['symbol', 'basename', 'path', 'curated', 'synonym', 'ambiguous', 'noise', 'removed', 'renamed', 'stale-map']) { + assert.ok(count(tag) > 0, `a ${tag} case exists`); + } + const devFamilies = new Set(dev.map((c) => c.family).filter((f) => f !== 'negative')); + assert.deepEqual(held.filter((c) => devFamilies.has(c.family)).map((c) => c.id), []); + assert.equal(new Set(all.map((c) => c.id)).size, all.length, 'ids are unique'); + for (const c of all) { + assert.equal(c.tags.includes('negative'), c.expect.length === 0, `${c.id}: negatives and only negatives expect nothing`); + } +}); + +test('every expected file exists at the pinned commit', (t) => { + const corpus = bench.loadCases(); + const r = spawnSync('git', ['ls-tree', '-r', '--name-only', corpus.pinnedCommit], { cwd: REPO_ROOT, encoding: 'utf8', maxBuffer: 64 * 1024 * 1024 }); + if (r.status !== 0) return t.skip('pinned commit not available (shallow clone)'); + const files = new Set(r.stdout.split('\n')); + for (const c of [...corpus.splits.development.cases, ...corpus.splits.heldOut.cases]) { + for (const f of c.expect) assert.ok(files.has(f), `${c.id}: ${f}`); + if (c.mutate && c.mutate.remove) assert.ok(files.has(c.mutate.remove)); + if (c.mutate && c.mutate.rename) assert.ok(files.has(c.mutate.rename[0])); + } +}); + +/* ------------------------------- parsing ------------------------------ */ + +test('CLI output parses from the JSON envelope and from the human listing; missing files never count', () => { + const env = { schema: 'frame.lookup/1', candidates: [{ path: 'a.js' }, { path: 'gone.js', missing: true }, { path: 'b.js' }] }; + assert.deepEqual(bench.parseCli(JSON.stringify(env)), ['a.js', 'b.js']); + const human = [ + 'Map: fresh · working tree', '', 'Feature: github', + ' src/main/githubManager.js — GitHub Manager Module', + ' src/renderer/gone.js — Gone ⚠ file missing on disk — run: npm run structure', + ' IPC: LOAD_GITHUB_ISSUES', '' + ].join('\n'); + assert.deepEqual(bench.parseCli(human), ['src/main/githubManager.js']); + assert.deepEqual(bench.parseCli('No modules found for "x"\n'), []); +}); + +test('hook output parses hinted files; silence and garbage are no hint', () => { + const context = 'Frame\'s module map already answers "github":\nFeature: github\n src/a.js — A\n src/b.js\n … +3 more\n IPC: X, Y\nFull query: …'; + const out = bench.parseHook(JSON.stringify({ hookSpecificOutput: { hookEventName: 'PreToolUse', additionalContext: context } })); + assert.deepEqual(out.files, ['src/a.js', 'src/b.js']); + assert.equal(out.context, context); + assert.deepEqual(bench.parseHook(''), { files: [], context: '' }); + assert.equal(bench.parseHook('{not json').invalid, true); +}); + +/* ------------------------------- metrics ------------------------------ */ + +const c = (id, expect, tags = []) => ({ id, expect, tags: expect.length ? tags : ['negative', ...tags] }); +const hook = (files, ms = 10, context = files.join('\n')) => ({ files, ms, context }); + +test('summaries compute recall, precision, false hints, abstention and quantiles as documented', () => { + const results = [ + { case: c('1', ['a.js'], ['basename']), cli: ['a.js', 'x.js'], cliMs: 10, hooks: { claude: hook(['a.js'], 10) } }, + { case: c('2', ['b.js'], ['symbol']), cli: ['x.js', 'y.js', 'z.js', 'w.js', 'b.js'], cliMs: 20, hooks: { claude: hook(['x.js'], 20) } }, + { case: c('3', ['c.js'], ['turkish', 'basename']), cli: [], cliMs: 30, hooks: { claude: hook([], 30) } }, + { case: c('4', []), cli: ['n.js'], cliMs: 40, hooks: { claude: hook(['n.js'], 40) } }, + { case: c('5', []), cli: [], cliMs: 50, hooks: { claude: hook([], 50) } } + ]; + const s = bench.summarize(results); + assert.equal(s.answerable, 3); + assert.equal(s.negatives, 2); + assert.equal(s.recallAt5, 2 / 3); + assert.equal(s.exactRecall, 1, 'the Turkish-mixed case is not an exact case'); + assert.equal(s.exactCases, 2); + assert.equal(s.precisionAt1, 1 / 3, 'top result right in 1 of the 3 cases that returned anything'); + assert.equal(s.cliNegativeReturns, 1 / 2); + assert.equal(s.cliP50Ms, 30); + assert.equal(s.cliP95Ms, 50); + const h = s.hooks.claude; + assert.equal(h.emitted, 3); + assert.equal(h.emittedPrecision, 1 / 3); + assert.equal(h.falseHintRate, 1 / 2); + assert.equal(h.hintRecall, 1 / 3); + assert.equal(h.abstention, 1 / 3); + assert.equal(s.strata.basename.n, 2); + assert.equal(s.strata.negative.falseHints, 1); +}); + +test('gates pass only when every limit holds, recall never drops below legacy, and scale latency counts', () => { + const good = { + exactRecall: 1, recallAt5: 0.93, precisionAt1: 0.95, cliP95Ms: 90, + hooks: { claude: { emittedPrecision: 0.99, falseHintRate: 0, p95Ms: 40, maxChars: 900 }, codex: { emittedPrecision: null, falseHintRate: 0, p95Ms: 41, maxChars: 0 } } + }; + assert.equal(bench.evaluateGates(good, { recallAt5: 0.6 }, [{ files: 10000, hookP95Ms: 45 }]).pass, true, 'a hook that never emitted has no precision to fail'); + + const failed = (patch, baseline = null, scale = null) => bench.evaluateGates({ ...good, ...patch }, baseline, scale).gates.filter((g) => !g.pass).map((g) => g.name); + assert.deepEqual(failed({ exactRecall: 0.99 }), ['exact-recall']); + assert.deepEqual(failed({}, { recallAt5: 0.95 }), ['recall@5-vs-legacy']); + assert.deepEqual(failed({ precisionAt1: 0.89 }), ['precision@1']); + assert.deepEqual(failed({ hooks: { claude: { ...good.hooks.claude, falseHintRate: 0.03 } } }), ['false-hints:claude']); + assert.deepEqual(failed({ hooks: { claude: { ...good.hooks.claude, maxChars: 1801 } } }), ['payload:claude']); + assert.deepEqual(failed({}, null, [{ files: 10000, hookP95Ms: 51 }]), ['hook-p95:10000-files']); + assert.deepEqual(failed({ cliP95Ms: 151 }), ['cli-p95']); +}); + +test('quantiles use the nearest-rank definition', () => { + assert.equal(bench.quantile([], 0.5), null); + assert.equal(bench.quantile([5], 0.95), 5); + assert.equal(bench.quantile([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 0.95), 10); + assert.equal(bench.quantile([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 0.5), 5); +}); From 212c42dea24775877e7582cc37a9076130142ba4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kaan=20=C3=96zhan?= Date: Thu, 1 Oct 2026 16:07:25 +0300 Subject: [PATCH 3/9] feat(structure): shared retrieval core with evidence tiers and the legacy engine (STR-03 T02) --- .frame/STRUCTURE.json | 168 ++++++- .../str-03-local-file-retrieval/outcome.md | 21 + scripts/structure-retrieval.js | 461 ++++++++++++++++++ test/structureRetrieval.test.js | 251 ++++++++++ 4 files changed, 896 insertions(+), 5 deletions(-) create mode 100644 scripts/structure-retrieval.js create mode 100644 test/structureRetrieval.test.js diff --git a/.frame/STRUCTURE.json b/.frame/STRUCTURE.json index f600cf82..e152d376 100644 --- a/.frame/STRUCTURE.json +++ b/.frame/STRUCTURE.json @@ -19061,6 +19061,130 @@ } } }, + "scripts/structure-retrieval": { + "file": "scripts/structure-retrieval.js", + "description": "STRUCTURE retrieval (STR-03) — from a path, file name, symbol or concept", + "exports": [ + "compileIndex", + "normalizeQuery", + "retrieve", + "legacyRetrieve", + "fold", + "splitWords", + "TIER", + "HOOK_MAX_TIER", + "LIMITS", + "INDEX_VERSION", + "ALGORITHM", + "NOISE" + ], + "depends": [], + "functions": { + "fold": { + "line": 62, + "params": [ + "text" + ], + "purpose": "Case- and accent-insensitive search form; Turkish letters fold to ASCII." + }, + "splitWords": { + "line": 72, + "params": [ + "text" + ], + "purpose": "Identifier words: camelCase, snake_case, kebab-case, dotted and path parts." + }, + "normalizeQuery": { + "line": 89, + "params": [ + "raw" + ], + "purpose": "A raw query → its search units. Regex syntax is dropped, alternations" + }, + "basenameOf": { + "line": 120, + "params": [ + "file" + ] + }, + "stripExt": { + "line": 124, + "params": [ + "name" + ] + }, + "compileIndex": { + "line": 138, + "params": [ + "structure", + "curation = {}", + "meta = {}" + ], + "purpose": "Compile the lookup index from a map (STR-02c read view) and the curated" + }, + "matchUnit": { + "line": 223, + "params": [ + "index", + "unit", + "phraseConcepts" + ], + "purpose": "Every match one unit has: Map(fileId → best tier) plus the concepts it named." + }, + "retrieve": { + "line": 284, + "params": [ + "index", + "query", + "options = {}" + ], + "purpose": "Rank the files that explain the query." + }, + "retrieveUnits": { + "line": 296, + "params": [ + "index", + "allUnits", + "mode", + "limit", + "options" + ] + }, + "legacyRetrieve": { + "line": 379, + "params": [ + "structure", + "curation", + "options = {}" + ], + "purpose": "Today's behavior, kept selectable as the rollback path (STR-03 D2):" + }, + "legacyHookLookup": { + "line": 432, + "params": [ + "index", + "intentMap", + "keyword" + ] + } + }, + "sizeBytes": 19438, + "extraction": { + "status": "parsed" + }, + "provenance": { + "description": "aa71905c9716ab4b", + "purposes": { + "compileIndex": "60c20164d8c3a443", + "fold": "475b654bf901e6d9", + "legacyRetrieve": "01c5d2059f3b0d6f", + "matchUnit": "02ae5ca770fd9391", + "normalizeQuery": "31b57c6910e130c5", + "retrieve": "acf77b261ff3fd1e", + "splitWords": "858bcfb0070cb9f5" + } + } + }, "scripts/structure-snapshot": { "file": "scripts/structure-snapshot.js", "description": "STRUCTURE snapshot — content identity for the working tree (STR-02).", @@ -24101,6 +24225,40 @@ } } }, + "test/structureRetrieval.test": { + "file": "test/structureRetrieval.test.js", + "description": "structure-retrieval tests (STR-03): evidence tiers, curated precedence,", + "exports": [], + "depends": [ + "node:test", + "node:assert/strict", + "fs", + "os", + "path", + "child_process", + "scripts/structure-retrieval" + ], + "functions": { + "mod": { + "line": 19, + "params": [ + "file", + "description", + "extra = {}" + ] + }, + "legacyProject": { + "line": 222 + } + }, + "sizeBytes": 13860, + "extraction": { + "status": "parsed" + }, + "provenance": { + "description": "d9d8aeaf4888558c" + } + }, "test/structureSnapshots.test": { "file": "test/structureSnapshots.test.js", "description": "structure-snapshot tests (STR-02 T03): content identity for the working", @@ -26102,7 +26260,7 @@ "extraction": { "coverage": "complete", "counts": { - "parsed": 292, + "parsed": 294, "unsupported": 83, "partial": 0 } @@ -26149,9 +26307,9 @@ } }, "counts": { - "visitedEntries": 463, + "visitedEntries": 465, "traversedDirectories": 80, - "eligibleFiles": 375, + "eligibleFiles": 377, "unreadable": 0, "binaryFiles": 5, "symlinks": 0, @@ -26168,7 +26326,7 @@ "config": 0, "depth": 0 }, - "indexedFiles": 375, + "indexedFiles": 377, "legacyGroups": 0 }, "diagnostics": { @@ -26176,6 +26334,6 @@ "truncated": false, "samples": [] }, - "revision": "60fa3f10f358213741926b7c4b230997858335f69a05121ded9fe6aa21eb2417" + "revision": "8e89c6fc955d86d991383345d2160ec4c97f950c910236d65ca47c6d6472744b" } } diff --git a/.frame/specs/str-03-local-file-retrieval/outcome.md b/.frame/specs/str-03-local-file-retrieval/outcome.md index ebd64f5e..7e2acf82 100644 --- a/.frame/specs/str-03-local-file-retrieval/outcome.md +++ b/.frame/specs/str-03-local-file-retrieval/outcome.md @@ -17,3 +17,24 @@ Files touched: `scripts/eval/retrieval-cases.json`, `scripts/eval/run-retrieval. _Captured: 2026-10-01 · 4 file change(s)_ --- + +## T02 — Retrieval core + +`scripts/structure-retrieval.js` (pure, dependency-free) provides: +- `compileIndex`: files `[[path, description]]`; postings keyed `:` for path, basename (with and without extension), symbol (functions, exports, IPC channels), path words and description words, capped at 64 per term with truncation recorded; concepts `[[name, fileIds, synonyms]]` in intentIndex order. +- `normalizeQuery`: NFKC, Turkish folding (including İ), accent stripping, camel/snake/kebab splitting, regex escapes and glob prefixes stripped, `|` alternatives tried in order, at most 512 characters and 8 units. +- `retrieve`, with the eight tiers. +- `legacyRetrieve`, reproducing find-module's four tiers and the hook's three; a parity test runs the current `find-module.js`. + +Rules added during development-split tuning (the held-out split has not been run against v2): +- Every identifier word must be explained by some match. Words with a non-ASCII letter are prose and may stay unexplained. +- When no file carries every word, files rank by how many they carry ("GitHub paneli"). +- The hook shows only the leading group (same coverage, tier and score). +- Partial concepts need ≥ 4 letters on both sides (the case that prompted it: "foo|bar" hinted the sidebar group). +- Comment markers (TODO/FIXME/XXX/HACK) are ignored by the hook. + +Development split, in process on the pinned map: recall@5 95.1%, P@1 98.3%, hook precision 100%, false hints 0/11. The remaining misses are 2 purely Turkish queries. The pinned map's index is 130 KB, against the 662 KB map. Honest note: the corpus and these rules share an author, so the held-out split guards against tuning, not against that bias. Files touched: `scripts/structure-retrieval.js`, `test/structureRetrieval.test.js`. + +_Captured: 2026-10-01 · 2 file change(s)_ + +--- diff --git a/scripts/structure-retrieval.js b/scripts/structure-retrieval.js new file mode 100644 index 00000000..1c61cd3d --- /dev/null +++ b/scripts/structure-retrieval.js @@ -0,0 +1,461 @@ +/** + * STRUCTURE retrieval (STR-03) — from a path, file name, symbol or concept + * to a small set of files, with the evidence for each. + * + * One engine serves both `find-module.js` (explicit lookup) and + * `module-hint.js` (the automatic search hint); the hook only takes the + * stronger evidence tiers. Pure and dependency-free: no Git, no subprocess, + * no network, nothing that builds or rewrites the map. Ships to + * `.frame/bin/`. + * + * Evidence tiers, strongest first: + * 1 path exact repo-relative path, or a path suffix ("lang/python.js") + * 2 concept a curated or generated intentIndex concept, exactly + * 3 synonym a curated synonym of a concept (intent-map.json) + * 4 basename a file name, with or without its extension + * 5 symbol an exported name, function name or IPC channel + * 6 partial concept a word containing a concept, or contained in one (both ≥ 4 letters) + * 7 path token a word of a file's directories or name + * 8 description a word of a file's description + * Hooks emit tiers 1–6 only; 7–8 are for explicit lookup. + * + * Every content word of a query must be explained by some match: a file + * that matches "config" does not answer "webpack config". A word carrying a + * non-ASCII letter ("dosyası", "yöneticisi") is natural-language prose — + * code identifiers here are ASCII — so it may stay unexplained. + */ + +'use strict'; + +const INDEX_VERSION = 1; +const ALGORITHM = 'str03-v2.1'; + +const LIMITS = Object.freeze({ + descriptionChars: 160, + postingsPerTerm: 64, + queryChars: 512, + queryUnits: 8, + hookFiles: 8, + cliFiles: 20, + hookIndexBytes: 2 * 1024 * 1024 +}); + +const TIER = Object.freeze({ PATH: 1, CONCEPT: 2, SYNONYM: 3, BASENAME: 4, SYMBOL: 5, PARTIAL: 6, PATH_TOKEN: 7, DESCRIPTION: 8 }); +const HOOK_MAX_TIER = TIER.PARTIAL; +const TIER_EVIDENCE = { 1: 'path', 2: 'concept', 3: 'synonym', 4: 'file name', 5: 'symbol', 6: 'partial concept', 7: 'path word', 8: 'description' }; + +// Words that carry no concept: file extensions, shell/code noise (the hook's +// historical list). Short on purpose — over-filtering costs a hit. +const NOISE = new Set(('js ts jsx tsx json md css html node npm git src test tests dist out lib bin tmp log ' + + 'true false null const let var function return async await require module exports import export ' + + 'the and for with from that this not all any new type name file path line text data code').split(' ')); +// Comment markers: a grep for them looks for comments, never for a feature. +const MARKERS = new Set(['todo', 'fixme', 'xxx', 'hack']); + +const hasOwn = (obj, key) => Object.prototype.hasOwnProperty.call(obj, key); + +/* ----------------------------- normalization ----------------------------- */ + +const TURKISH = { 'İ': 'i', 'I': 'i', 'ı': 'i', 'Ş': 's', 'ş': 's', 'Ğ': 'g', 'ğ': 'g', 'Ç': 'c', 'ç': 'c', 'Ö': 'o', 'ö': 'o', 'Ü': 'u', 'ü': 'u' }; + +/** Case- and accent-insensitive search form; Turkish letters fold to ASCII. */ +function fold(text) { + return String(text) + .normalize('NFKC') + .replace(/[İIıŞşĞğÇçÖöÜü]/g, (ch) => TURKISH[ch]) + .toLowerCase() + .normalize('NFD') + .replace(/\p{M}+/gu, ''); +} + +/** Identifier words: camelCase, snake_case, kebab-case, dotted and path parts. */ +function splitWords(text) { + return String(text) + .replace(/([\p{Ll}\p{N}])(\p{Lu})/gu, '$1 $2') + .replace(/(\p{Lu})(\p{Lu}\p{Ll})/gu, '$1 $2') + .split(/[^\p{L}\p{N}]+/u) + .map(fold) + .filter(Boolean); +} + +const NON_ASCII_LETTER = /[^\x00-\x7F]/; + +/** + * A raw query → its search units. Regex syntax is dropped, alternations + * become separate alternatives, globs keep their literal tail. Each unit + * keeps its raw spelling (for display), its folded form (for exact tiers) + * and its identifier words (for the word tiers). + */ +function normalizeQuery(raw) { + const text = String(raw == null ? '' : raw).slice(0, LIMITS.queryChars); + const alternatives = text.split('|').map((alt) => { + const cleaned = alt + .replace(/\\[bBsSwWdD]/g, ' ') + .replace(/\\(.)/g, '$1') + .replace(/(^|\s)(?:\*\*\/|\.\/)+/g, '$1') + .replace(/[\^$()[\]{}+?*'"`,;<>=!&]/g, ' '); + const units = []; + for (const piece of cleaned.split(/\s+/)) { + const token = piece.replace(/^[.:/\-_]+|[.:/\-_]+$/g, ''); + if (!token) continue; + const folded = fold(token); + const words = splitWords(token); + units.push({ + raw: token, + folded, + words, + prose: NON_ASCII_LETTER.test(token), + noise: words.length > 0 && words.every((w) => NOISE.has(w) || w.length < 3), + marker: MARKERS.has(folded) + }); + if (units.length >= LIMITS.queryUnits) break; + } + return units; + }).filter((units) => units.length > 0); + return { raw: text, alternatives }; +} + +/* ------------------------------- the index ------------------------------- */ + +function basenameOf(file) { + return file.slice(file.lastIndexOf('/') + 1); +} + +function stripExt(name) { + const dot = name.lastIndexOf('.'); + return dot > 0 ? name.slice(0, dot) : name; +} + +/** + * Compile the lookup index from a map (STR-02c read view) and the curated + * concept file (intent-map.json). Deterministic: files sorted by path, + * postings in file order, each term capped and marked when truncated. + * + * files [[path, description]] + * terms { ":": [fileId, …] } tiers 1, 4, 5, 7, 8 + * concepts [[name, [fileId, …], [synonym, …]]] intentIndex order + */ +function compileIndex(structure, curation = {}, meta = {}) { + const modules = structure && structure.modules && typeof structure.modules === 'object' ? structure.modules : {}; + const entries = Object.values(modules) + .filter((m) => m && typeof m.file === 'string' && m.file) + .sort((a, b) => (a.file < b.file ? -1 : a.file > b.file ? 1 : 0)); + const files = []; + const idOf = new Map(); + const terms = Object.create(null); + const truncated = new Set(); + const add = (tier, term, id) => { + if (!term) return; + const key = `${tier}:${term}`; + const list = terms[key] || (terms[key] = []); + if (list[list.length - 1] === id) return; + if (list.length >= LIMITS.postingsPerTerm) { + truncated.add(key); + return; + } + list.push(id); + }; + + for (const mod of entries) { + if (idOf.has(mod.file)) continue; + const id = files.length; + idOf.set(mod.file, id); + const description = typeof mod.description === 'string' ? mod.description.slice(0, LIMITS.descriptionChars) : ''; + files.push([mod.file, description]); + + const base = basenameOf(mod.file); + add(TIER.PATH, fold(mod.file), id); + add(TIER.BASENAME, fold(base), id); + add(TIER.BASENAME, fold(stripExt(base)), id); + + const symbols = new Set(); + for (const name of Object.keys(mod.functions && typeof mod.functions === 'object' ? mod.functions : {})) symbols.add(name); + for (const name of Array.isArray(mod.exports) ? mod.exports : []) if (typeof name === 'string') symbols.add(name); + const ipc = mod.ipc && typeof mod.ipc === 'object' ? mod.ipc : {}; + for (const name of [...(ipc.listens || []), ...(ipc.emits || [])]) if (typeof name === 'string') symbols.add(name); + for (const name of symbols) add(TIER.SYMBOL, fold(name), id); + + const seen = new Set(); + for (const w of splitWords(mod.file)) { + if (w.length < 2 || seen.has(w)) continue; + seen.add(w); + add(TIER.PATH_TOKEN, w, id); + } + for (const w of splitWords(description)) { + if (w.length < 3 || NOISE.has(w) || seen.has(w)) continue; + seen.add(w); + add(TIER.DESCRIPTION, w, id); + } + } + + const synonymsOf = Object.create(null); + for (const [concept, entry] of Object.entries(curation && typeof curation === 'object' ? curation : {})) { + if (concept === '_comment' || !entry || !Array.isArray(entry.synonyms)) continue; + synonymsOf[fold(concept)] = entry.synonyms.filter((s) => typeof s === 'string').map(fold); + } + const concepts = []; + const index = structure && structure.intentIndex && typeof structure.intentIndex === 'object' ? structure.intentIndex : {}; + for (const [name, members] of Object.entries(index)) { + if (!Array.isArray(members)) continue; + const ids = []; + for (const m of members) { + const id = m && idOf.get(m.file); + if (id !== undefined && !ids.includes(id)) ids.push(id); + } + if (ids.length) concepts.push([fold(name), ids, synonymsOf[fold(name)] || []]); + } + + return { + version: INDEX_VERSION, + algorithm: ALGORITHM, + revision: (structure && structure.generation && structure.generation.revision) || null, + ...meta, + files, + terms, + concepts, + truncated: [...truncated].sort() + }; +} + +/* ------------------------------- retrieval ------------------------------- */ + +/** Every match one unit has: Map(fileId → best tier) plus the concepts it named. */ +function matchUnit(index, unit, phraseConcepts) { + const found = new Map(); + const note = (id, tier) => { + const prev = found.get(id); + if (prev === undefined || tier < prev) found.set(id, tier); + }; + const postings = (tier, term) => index.terms[`${tier}:${term}`] || []; + + // 1 path: exact, or a suffix that starts at a directory boundary + for (const id of postings(TIER.PATH, unit.folded)) note(id, TIER.PATH); + if (unit.folded.includes('/')) { + const suffix = `/${unit.folded}`; + index.files.forEach(([file], id) => { + if (fold(file).endsWith(suffix)) note(id, TIER.PATH); + }); + } + // 2–3 concepts and synonyms, 6 partial concepts + const conceptTerms = new Set([unit.folded, unit.folded.replace(/[\s_]+/g, '-')]); + for (const [name, ids, synonyms] of index.concepts) { + let tier = null; + if (conceptTerms.has(name) || phraseConcepts.has(name)) tier = TIER.CONCEPT; + else if (synonyms.some((s) => conceptTerms.has(s) || phraseConcepts.has(s))) tier = TIER.SYNONYM; + else if (!unit.noise && unit.folded.length >= 4 && name.length >= 4 && /^[\p{L}\p{N}_-]+$/u.test(unit.folded) + && (unit.folded.includes(name) || name.includes(unit.folded))) tier = TIER.PARTIAL; + if (tier !== null) for (const id of ids) note(id, tier); + } + // 4 basename, 5 symbol + for (const id of postings(TIER.BASENAME, unit.folded)) note(id, TIER.BASENAME); + for (const id of postings(TIER.SYMBOL, unit.folded)) note(id, TIER.SYMBOL); + // 7–8 words + const words = unit.words.filter((w) => w.length >= 3 && !NOISE.has(w)); + if (words.length) { + // a file matches at a word tier only when it carries every word of the + // unit: in its path (7), or across its path and description (8) + let common = null; + for (const w of words) { + const ids = new Set([...postings(TIER.PATH_TOKEN, w), ...postings(TIER.DESCRIPTION, w)]); + common = common === null ? ids : new Set([...common].filter((id) => ids.has(id))); + } + for (const id of common) { + const inPath = words.every((w) => postings(TIER.PATH_TOKEN, w).includes(id)); + note(id, inPath ? TIER.PATH_TOKEN : TIER.DESCRIPTION); + } + } + return found; +} + +/** + * Rank the files that explain the query. + * + * options: { mode: 'cli' | 'hook', limit, exists(path) → boolean } + * Returns { + * status: 'resolved' | 'ambiguous' | 'no-match', + * candidates: [{ path, description, tier, evidence, missing? }], + * answer: the leading group (same coverage and tier) — what a hook shows, + * truncated + * } + * `resolved` is a single file, or one concept group; several files of equal + * standing at the file tiers are `ambiguous`. In hook mode only tiers 1–6 + * count and comment markers are ignored. + */ +function retrieve(index, query, options = {}) { + const mode = options.mode === 'hook' ? 'hook' : 'cli'; + const limit = Math.max(1, Math.min(options.limit || LIMITS.hookFiles, mode === 'hook' ? LIMITS.hookFiles : LIMITS.cliFiles)); + const { alternatives } = normalizeQuery(query); + + for (const units of alternatives) { + const result = retrieveUnits(index, units, mode, limit, options); + if (result.status !== 'no-match') return result; + } + return { status: 'no-match', candidates: [], answer: [], truncated: false }; +} + +function retrieveUnits(index, allUnits, mode, limit, options) { + const units = allUnits.filter((u) => !u.noise && !(mode === 'hook' && u.marker)); + const none = { status: 'no-match', candidates: [], answer: [], truncated: false }; + if (!units.length) return none; + + // a whole multi-word query may name a concept ("file tree" → file-tree) + const phraseConcepts = new Set(); + if (units.length > 1) { + const phrase = units.map((u) => u.folded).join('-'); + phraseConcepts.add(phrase); + } + + const perFile = new Map(); // id → { covered: Set(unitIndex), tier } + const required = []; + units.forEach((unit, i) => { + const matches = matchUnit(index, unit, phraseConcepts); + const usable = mode === 'hook' ? [...matches].filter(([, tier]) => tier <= HOOK_MAX_TIER) : [...matches]; + if (!usable.length) { + if (!unit.prose) required.push(i); // an unexplained identifier word: nothing answers the query + return; + } + required.push(i); + for (const [id, tier] of usable) { + const entry = perFile.get(id) || { covered: new Set(), tier: Infinity, sum: 0 }; + entry.covered.add(i); + entry.tier = Math.min(entry.tier, tier); + entry.sum += tier; + perFile.set(id, entry); + } + }); + + const conceptOrder = new Map(); + index.concepts.forEach(([, ids], c) => ids.forEach((id, pos) => { + if (!conceptOrder.has(id)) conceptOrder.set(id, c * 100000 + pos); + })); + + // Every required word must be explained by some match. Files carrying + // all of them come first; when none does ("GitHub paneli": the github + // group and the panel group), files rank by how many words they carry. + const explained = new Set([...perFile.values()].flatMap((e) => [...e.covered])); + if (!required.every((i) => explained.has(i))) return none; + let ranked = [...perFile.entries()] + .map(([id, e]) => ({ id, tier: e.tier, sum: e.sum, cov: e.covered.size })) + .sort((a, b) => b.cov - a.cov || a.tier - b.tier || a.sum - b.sum + || (conceptOrder.get(a.id) ?? Infinity) - (conceptOrder.get(b.id) ?? Infinity) + || (index.files[a.id][0] < index.files[b.id][0] ? -1 : 1)); + if (!ranked.length) return none; + + if (typeof options.exists === 'function') { + ranked = ranked.map((r) => ({ ...r, missing: !options.exists(index.files[r.id][0]) })); + if (mode === 'hook') ranked = ranked.filter((r) => !r.missing); + if (!ranked.length) return none; + } + + const lead = ranked.find((r) => !r.missing) || ranked[0]; + const answerIds = ranked.filter((r) => r.cov === lead.cov && r.tier === lead.tier && r.sum === lead.sum && !r.missing); + const toCandidate = (r) => { + const [file, description] = index.files[r.id]; + const c = { path: file, description, tier: r.tier, evidence: TIER_EVIDENCE[r.tier] }; + if (r.missing) c.missing = true; + return c; + }; + const shown = mode === 'hook' ? answerIds : ranked; + const candidates = shown.slice(0, limit).map(toCandidate); + const answer = answerIds.slice(0, limit).map(toCandidate); + const single = answerIds.length === 1 || lead.tier === TIER.CONCEPT || lead.tier === TIER.SYNONYM; + return { + status: single ? 'resolved' : 'ambiguous', + candidates, + answer, + truncated: shown.length > limit + }; +} + +/* --------------------------------- legacy -------------------------------- */ + +/** + * Today's behavior, kept selectable as the rollback path (STR-03 D2): + * find-module's four tiers (exact → synonym → partial → deep substring) in + * `cli` mode, the hook's curated three (exact → synonym → partial, first + * keyword that hits) in `hook` mode. Same result shape as `retrieve`, plus + * `groups` ([{ feature, modules, matchType }]) for the legacy rendering. + */ +function legacyRetrieve(structure, curation, options = {}) { + const index = (structure && structure.intentIndex) || null; + const modules = (structure && structure.modules) || {}; + const intentMap = {}; + for (const [k, v] of Object.entries(curation && typeof curation === 'object' ? curation : {})) if (k !== '_comment') intentMap[k] = v; + const tierOf = { exact: TIER.CONCEPT, synonym: TIER.SYNONYM, partial: TIER.PARTIAL, deep: TIER.DESCRIPTION }; + const shape = (groups) => { + const candidates = []; + for (const g of groups) { + for (const m of g.modules) { + if (!candidates.some((c) => c.path === m.file)) { + candidates.push({ path: m.file, description: m.description || '', tier: tierOf[g.matchType], evidence: g.matchType, feature: g.feature, module: m.module }); + } + } + } + return { status: candidates.length ? (groups.length === 1 && groups[0].matchType !== 'deep' ? 'resolved' : 'ambiguous') : 'no-match', candidates, answer: candidates, groups, truncated: false }; + }; + if (!index) return { ...shape([]), status: 'unavailable' }; + + if (options.mode === 'hook') { + for (const keyword of options.words || []) { + const hit = legacyHookLookup(index, intentMap, keyword); + if (hit) return { ...shape([{ ...hit, matchType: hit.matchType }]), keyword }; + } + return shape([]); + } + + const keyword = String(options.query || ''); + const kw = keyword.toLowerCase(); + const results = []; + for (const [feature, mods] of Object.entries(index)) if (feature === kw) results.push({ feature, modules: mods, matchType: 'exact' }); + if (!results.length) { + for (const [concept, entry] of Object.entries(intentMap)) { + const synonyms = (entry.synonyms || []).map((s) => s.toLowerCase()); + if (synonyms.includes(kw) && index[concept]) results.push({ feature: `${concept} (synonym: "${keyword}")`, modules: index[concept], matchType: 'synonym' }); + } + } + if (!results.length) { + for (const [feature, mods] of Object.entries(index)) { + if (feature.includes(kw) || kw.includes(feature)) results.push({ feature, modules: mods, matchType: 'partial' }); + } + } + if (!results.length) { + const matched = []; + for (const [key, mod] of Object.entries(modules)) { + const searchable = [key, mod.description || '', ...(mod.exports || []), ...((mod.ipc && mod.ipc.listens) || []), ...((mod.ipc && mod.ipc.emits) || [])].join(' ').toLowerCase(); + if (searchable.includes(kw)) matched.push({ module: key, file: mod.file, description: mod.description || '' }); + } + if (matched.length) results.push({ feature: `search: "${keyword}"`, modules: matched, matchType: 'deep' }); + } + return shape(results); +} + +function legacyHookLookup(index, intentMap, keyword) { + for (const [feature, modules] of Object.entries(index)) { + if (feature === keyword) return { feature, modules, matchType: 'exact' }; + } + for (const [concept, entry] of Object.entries(intentMap)) { + const synonyms = (entry.synonyms || []).map((s) => String(s).toLowerCase()); + if (synonyms.includes(keyword) && index[concept]) { + return { feature: `${concept} (synonym: "${keyword}")`, modules: index[concept], matchType: 'synonym' }; + } + } + for (const [feature, modules] of Object.entries(index)) { + if (feature.includes(keyword) || keyword.includes(feature)) return { feature, modules, matchType: 'partial' }; + } + return null; +} + +module.exports = { + compileIndex, + normalizeQuery, + retrieve, + legacyRetrieve, + fold, + splitWords, + TIER, + HOOK_MAX_TIER, + LIMITS, + INDEX_VERSION, + ALGORITHM, + NOISE +}; diff --git a/test/structureRetrieval.test.js b/test/structureRetrieval.test.js new file mode 100644 index 00000000..2cf6efab --- /dev/null +++ b/test/structureRetrieval.test.js @@ -0,0 +1,251 @@ +/** + * structure-retrieval tests (STR-03): evidence tiers, curated precedence, + * files outside every concept, ambiguity, query normalization (Turkish, + * camelCase, regex/glob syntax), limits, and parity of the legacy engine + * with today's find-module / module-hint behavior. + */ + +const { test } = require('node:test'); +const assert = require('node:assert/strict'); +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const { spawnSync } = require('child_process'); + +const R = require('../scripts/structure-retrieval'); + +const SCRIPTS = path.join(__dirname, '..', 'scripts'); + +function mod(file, description, extra = {}) { + return { file, description, exports: [], depends: [], functions: {}, ...extra }; +} + +const STRUCTURE = { + generation: { revision: 'rev-1' }, + modules: { + 'main/githubManager': mod('src/main/githubManager.js', 'GitHub Manager Module', { + functions: { checkGhAuth: { line: 1 } }, ipc: { listens: ['LOAD_GITHUB_ISSUES'], emits: [] } + }), + 'renderer/githubPanel': mod('src/renderer/githubPanel.js', 'GitHub Panel — the sidebar tab'), + 'renderer/github/rowModels': mod('src/renderer/github/rowModels.js', 'Row view-models — pure', { + functions: { issueBranchName: { line: 3 } } + }), + 'main/frameStore': mod('src/main/frameStore.js', 'frameStore — the one module that knows where meta files live'), + 'main/fsSafe': mod('src/main/fsSafe.js', 'Durable state-file primitives', { functions: { writeFileAtomic: { line: 1 } } }), + 'main/uiZoom': mod('src/main/uiZoom.js', 'UI zoom — main-process owner'), + 'shared/uiZoom': mod('src/shared/uiZoom.js', 'UI zoom ladder', { functions: { factorFor: { line: 2 } } }), + 'main/settings': mod('src/main/userSettings.js', 'User Settings'), + 'renderer/settingsOverlay': mod('src/renderer/settingsOverlay.js', 'Settings overlay plumbing'), + 'lang/rust': mod('scripts/lang/rust.js', 'Rust extractor', { functions: { extractFunctions: { line: 1 } } }), + 'lang/go': mod('scripts/lang/go.js', 'Go extractor', { functions: { extractFunctions: { line: 1 } } }), + 'renderer/statusBar': mod('src/renderer/statusBar.js', 'Status bar'), + 'renderer/sidebarResize': mod('src/renderer/sidebarResize.js', 'Sidebar resize'), + 'main/A': mod('src/main/Case.js', 'Upper'), + 'main/a': mod('src/main/case.js', 'Lower') + }, + intentIndex: { + github: [{ module: 'main/githubManager', file: 'src/main/githubManager.js' }, { module: 'renderer/githubPanel', file: 'src/renderer/githubPanel.js' }], + settings: [{ module: 'main/settings', file: 'src/main/userSettings.js' }, { module: 'renderer/settingsOverlay', file: 'src/renderer/settingsOverlay.js' }], + sidebar: [{ module: 'renderer/sidebarResize', file: 'src/renderer/sidebarResize.js' }], + zoom: [{ module: 'main/uiZoom', file: 'src/main/uiZoom.js' }, { module: 'shared/uiZoom', file: 'src/shared/uiZoom.js' }] + } +}; +const CURATION = { + _comment: 'test', + github: { synonyms: ['issues', 'pull-requests'] }, + settings: { synonyms: ['preferences', 'config'] }, + tasks: { synonyms: ['todo'] } +}; + +const index = R.compileIndex(STRUCTURE, CURATION); +const paths = (result) => result.candidates.map((c) => c.path); +const cli = (q, opts = {}) => R.retrieve(index, q, { mode: 'cli', ...opts }); +const hook = (q, opts = {}) => R.retrieve(index, q, { mode: 'hook', ...opts }); + +/* --------------------------------- tiers --------------------------------- */ + +test('a file outside every concept is found by path, file name and symbol', () => { + assert.deepEqual(paths(cli('src/renderer/github/rowModels.js')), ['src/renderer/github/rowModels.js']); + assert.equal(cli('github/rowModels.js').candidates[0].evidence, 'path', 'a path suffix at a directory boundary'); + assert.deepEqual(paths(hook('rowModels')), ['src/renderer/github/rowModels.js']); + assert.deepEqual(paths(hook('rowModels.js')), ['src/renderer/github/rowModels.js']); + const bySymbol = hook('issueBranchName'); + assert.deepEqual(paths(bySymbol), ['src/renderer/github/rowModels.js']); + assert.equal(bySymbol.candidates[0].evidence, 'symbol'); + assert.equal(bySymbol.status, 'resolved'); + assert.deepEqual(paths(hook('LOAD_GITHUB_ISSUES')), ['src/main/githubManager.js'], 'IPC channels are symbols'); +}); + +test('curated concepts and synonyms answer with the whole group, in curated order', () => { + const concept = hook('github'); + assert.deepEqual(paths(concept), ['src/main/githubManager.js', 'src/renderer/githubPanel.js']); + assert.equal(concept.status, 'resolved', 'one concept group is one answer'); + assert.equal(concept.candidates[0].evidence, 'concept'); + assert.deepEqual(paths(hook('pull-requests')), ['src/main/githubManager.js', 'src/renderer/githubPanel.js']); + assert.equal(hook('preferences').candidates[0].evidence, 'synonym'); +}); + +test('an exact file name beats a concept it happens to contain', () => { + const r = hook('githubPanel'); + assert.deepEqual(paths(r), ['src/renderer/githubPanel.js'], 'not the whole github group'); + assert.equal(r.candidates[0].evidence, 'file name'); + // the CLI still lists the rest, ranked below + assert.equal(cli('githubPanel').candidates[0].path, 'src/renderer/githubPanel.js'); +}); + +test('ambiguous names list every file of equal standing', () => { + const zoom = hook('uiZoom'); + assert.equal(zoom.status, 'ambiguous'); + assert.deepEqual(paths(zoom), ['src/main/uiZoom.js', 'src/shared/uiZoom.js']); + const fn = hook('extractFunctions'); + assert.equal(fn.status, 'ambiguous'); + assert.deepEqual(paths(fn), ['scripts/lang/go.js', 'scripts/lang/rust.js']); +}); + +test('files differing only in case stay two candidates', () => { + assert.deepEqual(paths(cli('case.js')).sort(), ['src/main/Case.js', 'src/main/case.js']); + assert.equal(cli('case.js').status, 'ambiguous'); +}); + +test('description and path words reach the CLI only', () => { + const r = cli('rust extractor'); + assert.deepEqual(paths(r).slice(0, 1), ['scripts/lang/rust.js']); + assert.deepEqual(paths(cli('extractor')), ['scripts/lang/go.js', 'scripts/lang/rust.js']); + assert.equal(hook('extractor').status, 'no-match', 'a description word never becomes a hint'); + assert.equal(hook('primitives').status, 'no-match'); +}); + +/* ------------------------------ silence rules ----------------------------- */ + +test('every identifier word must be explained; one matching word does not answer the query', () => { + assert.equal(cli('webpack config').status, 'no-match', '"config" alone is a synonym, "webpack" explains nothing'); + assert.equal(hook('elasticsearch index').status, 'no-match'); + assert.equal(hook('payment gateway').status, 'no-match'); +}); + +test('words with non-ASCII letters are prose and may stay unexplained', () => { + assert.deepEqual(paths(hook('frameStore dosyası')), ['src/main/frameStore.js']); + assert.deepEqual(paths(hook('GitHub yöneticisi')), ['src/main/githubManager.js', 'src/renderer/githubPanel.js']); + assert.equal(hook('ödeme sayfası').status, 'no-match', 'prose alone is no query'); +}); + +test('when no file carries every word, files rank by how many they carry', () => { + // githubPanel.js carries both words (concept + "the sidebar tab"), so it leads + assert.equal(cli('github sidebar').candidates[0].path, 'src/renderer/githubPanel.js'); + // "GitHub paneli": no file is in both groups, yet both words are explained + const loose = cli('github zoom'); + assert.deepEqual(paths(loose).sort(), ['src/main/githubManager.js', 'src/main/uiZoom.js', 'src/renderer/github/rowModels.js', 'src/renderer/githubPanel.js', 'src/shared/uiZoom.js'], 'rowModels.js by its github/ directory'); +}); + +test('noise, comment markers and short partial words never produce a hint', () => { + assert.equal(hook('console.log').status, 'no-match'); + assert.equal(hook('async function').status, 'no-match'); + assert.equal(hook('TODO').status, 'no-match', 'a grep for TODO looks for comments'); + assert.equal(cli('todo').status, 'no-match', 'no tasks concept in this map'); + assert.equal(hook('bar').status, 'no-match', '"bar" ⊂ "sidebar" is below the partial-concept minimum'); + assert.equal(hook('foo|bar').status, 'no-match'); +}); + +test('a partial concept is a hook-eligible tier below exact file evidence', () => { + const r = hook('settingsPanel'); + assert.equal(r.candidates[0].evidence, 'partial concept'); + assert.deepEqual(paths(r), ['src/main/userSettings.js', 'src/renderer/settingsOverlay.js']); +}); + +/* ------------------------------ normalization ----------------------------- */ + +test('Turkish letters, dotted capitals and accents fold; identifiers split on case and separators', () => { + assert.equal(R.fold('GİTHUB'), 'github'); + assert.equal(R.fold('İPC KANALLARI'), 'ipc kanallari'); + assert.equal(R.fold('çıkarıcı'), 'cikarici'); + assert.equal(R.fold('Café'), 'cafe'); + assert.deepEqual(R.splitWords('structureBootstrap'), ['structure', 'bootstrap']); + assert.deepEqual(R.splitWords('LOAD_GITHUB_ISSUES'), ['load', 'github', 'issues']); + assert.deepEqual(R.splitWords('XMLParser-v2'), ['xml', 'parser', 'v2']); + assert.deepEqual(paths(hook('GİTHUB')), ['src/main/githubManager.js', 'src/renderer/githubPanel.js']); +}); + +test('regex and glob syntax is stripped; alternations are tried in order', () => { + assert.deepEqual(paths(hook('\\bwriteFileAtomic\\b')), ['src/main/fsSafe.js']); + assert.deepEqual(paths(hook('**/frameStore.js')), ['src/main/frameStore.js']); + assert.deepEqual(paths(hook('nothingHere|factorFor')), ['src/shared/uiZoom.js']); + const q = R.normalizeQuery('a'.repeat(2000)); + assert.equal(q.raw.length, R.LIMITS.queryChars); + assert.ok(R.normalizeQuery('a b c d e f g h i j k').alternatives[0].length <= R.LIMITS.queryUnits); +}); + +/* --------------------------------- limits -------------------------------- */ + +test('results are bounded: hooks show at most 8 files, the CLI at most 20', () => { + const modules = {}; + const group = []; + for (let i = 0; i < 30; i++) { + modules[`m${i}`] = mod(`src/widget${i}.js`, `Widget ${i}`); + group.push({ module: `m${i}`, file: `src/widget${i}.js` }); + } + const big = R.compileIndex({ modules, intentIndex: { widgets: group } }, {}); + const h = R.retrieve(big, 'widgets', { mode: 'hook' }); + assert.equal(h.candidates.length, 8); + assert.equal(h.truncated, true); + assert.equal(R.retrieve(big, 'widgets', { mode: 'cli', limit: 50 }).candidates.length, 20); + assert.equal(R.retrieve(big, 'widgets', { mode: 'cli' }).candidates.length, 8, 'default limit'); +}); + +test('postings are capped per term and the truncation is recorded', () => { + const modules = {}; + for (let i = 0; i < 100; i++) modules[`m${i}`] = mod(`pkg${i}/index.js`, 'x', { functions: { handle: { line: 1 } } }); + const idx = R.compileIndex({ modules, intentIndex: {} }, {}); + assert.equal(idx.terms['5:handle'].length, R.LIMITS.postingsPerTerm); + assert.ok(idx.truncated.includes('5:handle')); + assert.ok(idx.truncated.includes('4:index.js')); +}); + +test('the index is deterministic and carries its revision and algorithm', () => { + const again = R.compileIndex(JSON.parse(JSON.stringify(STRUCTURE)), CURATION); + assert.equal(JSON.stringify(again), JSON.stringify(index)); + assert.equal(index.revision, 'rev-1'); + assert.equal(index.algorithm, R.ALGORITHM); + assert.equal(index.version, R.INDEX_VERSION); + assert.ok(index.files.every(([p, d]) => typeof p === 'string' && d.length <= R.LIMITS.descriptionChars)); +}); + +test('missing files are flagged for the CLI and dropped from hooks', () => { + const exists = (p) => p !== 'src/renderer/githubPanel.js'; + const c = cli('github', { exists }); + assert.equal(c.candidates.find((x) => x.path === 'src/renderer/githubPanel.js').missing, true); + assert.deepEqual(paths(hook('github', { exists })), ['src/main/githubManager.js']); + assert.ok(!paths(hook('githubPanel', { exists })).includes('src/renderer/githubPanel.js'), 'falls back to other evidence, never the missing file'); +}); + +/* --------------------------------- legacy -------------------------------- */ + +function legacyProject() { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'frame-legacy-parity-')); + fs.mkdirSync(path.join(dir, '.frame'), { recursive: true }); + fs.writeFileSync(path.join(dir, '.frame', 'STRUCTURE.json'), JSON.stringify(STRUCTURE)); + for (const m of Object.values(STRUCTURE.modules)) { + fs.mkdirSync(path.dirname(path.join(dir, m.file)), { recursive: true }); + fs.writeFileSync(path.join(dir, m.file), '// x\n'); + } + return dir; +} + +test('the legacy engine reproduces find-module\'s four tiers', (t) => { + const dir = legacyProject(); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + const curation = JSON.parse(fs.readFileSync(path.join(SCRIPTS, 'intent-map.json'), 'utf8')); + for (const q of ['github', 'issues', 'git', 'extractor', 'checkGhAuth', 'nothing-here']) { + const out = spawnSync('node', [path.join(SCRIPTS, 'find-module.js'), q], { encoding: 'utf8', env: { ...process.env, FRAME_PROJECT_ROOT: dir } }).stdout; + const listed = out.split('\n').map((l) => /^ {2}(\S+\.\w+)/.exec(l)).filter(Boolean).map((m) => m[1]); + assert.deepEqual(paths(R.legacyRetrieve(STRUCTURE, curation, { mode: 'cli', query: q })), listed, q); + } +}); + +test('the legacy hook engine takes the first keyword that hits a curated tier', () => { + const r = R.legacyRetrieve(STRUCTURE, CURATION, { mode: 'hook', words: ['nothing', 'issues', 'github'] }); + assert.equal(r.keyword, 'issues'); + assert.equal(r.groups[0].matchType, 'synonym'); + assert.deepEqual(paths(r), ['src/main/githubManager.js', 'src/renderer/githubPanel.js']); + assert.equal(R.legacyRetrieve(STRUCTURE, CURATION, { mode: 'hook', words: ['checkghauth'] }).status, 'no-match', 'no deep tier in hooks'); + assert.equal(R.legacyRetrieve({ modules: {} }, {}, { mode: 'cli', query: 'x' }).status, 'unavailable'); +}); From eee3118a8297f68b523d3c1265ae876e42c7c8b0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kaan=20=C3=96zhan?= Date: Thu, 1 Oct 2026 16:17:20 +0300 Subject: [PATCH 4/9] feat(structure): publish the lookup index with the working view (STR-03 T03) --- .frame/STRUCTURE.json | 178 +++++++++++++----- .../str-03-local-file-retrieval/outcome.md | 17 ++ package.json | 1 + scripts/structure-lifecycle.js | 11 ++ scripts/structure-retrieval.js | 168 ++++++++++++++++- scripts/update-structure.js | 19 ++ src/main/structureBootstrap.js | 6 +- test/projectAgnostic.test.js | 32 ++++ test/scriptsProjectRoot.test.js | 18 ++ test/structureLifecycle.test.js | 21 +++ test/structureRetrieval.test.js | 73 +++++++ 11 files changed, 493 insertions(+), 51 deletions(-) diff --git a/.frame/STRUCTURE.json b/.frame/STRUCTURE.json index e152d376..0a811d37 100644 --- a/.frame/STRUCTURE.json +++ b/.frame/STRUCTURE.json @@ -5214,7 +5214,7 @@ "purpose": "Top-level bootstrap orchestrator. Called from frameProject.js after the" } }, - "sizeBytes": 26865, + "sizeBytes": 26943, "extraction": { "status": "parsed" }, @@ -6028,7 +6028,7 @@ "exports": [], "depends": [], "functions": {}, - "sizeBytes": 3707, + "sizeBytes": 3747, "extraction": { "status": "unsupported", "reason": "no-extractor" @@ -18754,6 +18754,7 @@ "structure-state", "structure-snapshot", "structure-read", + "structure-retrieval", "structure-commit", "activity-log" ], @@ -18826,13 +18827,13 @@ "purpose": "One full reconciliation of the working tree: discover, observe content" }, "ownerPaths": { - "line": 452, + "line": 463, "params": [ "root" ] }, "acquireOwner": { - "line": 457, + "line": 468, "params": [ "root", "options = {}" @@ -18840,21 +18841,21 @@ "purpose": "One coordinator per checkout; reclaimed only from a demonstrably dead owner." }, "releaseOwner": { - "line": 461, + "line": 472, "params": [ "root", "token" ] }, "resolveGitDirs": { - "line": 468, + "line": 479, "params": [ "root" ], "purpose": "Git directories of a checkout, from `.git` or a `gitdir:` file. No Git process." }, "classify": { - "line": 488, + "line": 499, "params": [ "rel", "ownedRootMap" @@ -18862,13 +18863,13 @@ "purpose": "Classify one notification (root-relative POSIX path, or null) into a" }, "ownsRootMap": { - "line": 502, + "line": 513, "params": [ "root" ] }, "createWatcher": { - "line": 513, + "line": 524, "params": [ "root", "onEvent", @@ -18877,14 +18878,14 @@ "purpose": "Watch a checkout. Recursive watching where the runtime supports it," }, "directoriesOf": { - "line": 614, + "line": 625, "params": [ "root" ], "purpose": "Directories to watch in per-directory mode: parents of eligible files." }, "startWorker": { - "line": 646, + "line": 657, "params": [ "root", "options = {}" @@ -18892,24 +18893,24 @@ "purpose": "Run the lifecycle for one checkout until stopped." }, "resolveProjectRoot": { - "line": 788 + "line": 799 }, "writeLine": { - "line": 797, + "line": 808, "params": [ "stream", "value" ] }, "runOnce": { - "line": 801, + "line": 812, "params": [ "root", "json" ] }, "runLongLived": { - "line": 823, + "line": 834, "params": [ "root", "supervised" @@ -18917,10 +18918,10 @@ "purpose": "--watch: foreground coordinator with its own timer until SIGINT/SIGTERM." }, "main": { - "line": 865 + "line": 876 } }, - "sizeBytes": 31157, + "sizeBytes": 31557, "extraction": { "status": "parsed" }, @@ -19066,6 +19067,11 @@ "description": "STRUCTURE retrieval (STR-03) — from a path, file name, symbol or concept", "exports": [ "compileIndex", + "publishLookup", + "loadLookup", + "indexFromMap", + "lookupPath", + "curationPath", "normalizeQuery", "retrieve", "legacyRetrieve", @@ -19078,43 +19084,47 @@ "ALGORITHM", "NOISE" ], - "depends": [], + "depends": [ + "fs", + "path", + "structure-read" + ], "functions": { "fold": { - "line": 62, + "line": 63, "params": [ "text" ], "purpose": "Case- and accent-insensitive search form; Turkish letters fold to ASCII." }, "splitWords": { - "line": 72, + "line": 73, "params": [ "text" ], "purpose": "Identifier words: camelCase, snake_case, kebab-case, dotted and path parts." }, "normalizeQuery": { - "line": 89, + "line": 90, "params": [ "raw" ], "purpose": "A raw query → its search units. Regex syntax is dropped, alternations" }, "basenameOf": { - "line": 120, + "line": 121, "params": [ "file" ] }, "stripExt": { - "line": 124, + "line": 125, "params": [ "name" ] }, "compileIndex": { - "line": 138, + "line": 139, "params": [ "structure", "curation = {}", @@ -19123,7 +19133,7 @@ "purpose": "Compile the lookup index from a map (STR-02c read view) and the curated" }, "matchUnit": { - "line": 223, + "line": 224, "params": [ "index", "unit", @@ -19132,7 +19142,7 @@ "purpose": "Every match one unit has: Map(fileId → best tier) plus the concepts it named." }, "retrieve": { - "line": 284, + "line": 285, "params": [ "index", "query", @@ -19141,7 +19151,7 @@ "purpose": "Rank the files that explain the query." }, "retrieveUnits": { - "line": 296, + "line": 297, "params": [ "index", "allUnits", @@ -19150,8 +19160,64 @@ "options" ] }, + "lookupPath": { + "line": 381, + "params": [ + "root" + ] + }, + "curationPath": { + "line": 386, + "params": [ + "dir = __dirname" + ], + "purpose": "intent-map.json beside the running scripts (scripts/ or .frame/bin/)." + }, + "readCuration": { + "line": 390, + "params": [ + "file" + ] + }, + "signatureOf": { + "line": 398, + "params": [ + "file" + ] + }, + "sameSignature": { + "line": 407, + "params": [ + "a", + "b" + ] + }, + "publishLookup": { + "line": 419, + "params": [ + "root", + "options = {}" + ], + "purpose": "Compile and atomically publish lookup.json from the map at `mapPath`" + }, + "loadLookup": { + "line": 465, + "params": [ + "root", + "options = {}" + ], + "purpose": "Read the published index for a lookup. Never compiles, never writes." + }, + "indexFromMap": { + "line": 503, + "params": [ + "root", + "options = {}" + ], + "purpose": "Compile an index in memory from the read view — the fallback when no" + }, "legacyRetrieve": { - "line": 379, + "line": 536, "params": [ "structure", "curation", @@ -19160,7 +19226,7 @@ "purpose": "Today's behavior, kept selectable as the rollback path (STR-03 D2):" }, "legacyHookLookup": { - "line": 432, + "line": 589, "params": [ "index", "intentMap", @@ -19168,7 +19234,7 @@ ] } }, - "sizeBytes": 19438, + "sizeBytes": 25653, "extraction": { "status": "parsed" }, @@ -19176,10 +19242,14 @@ "description": "aa71905c9716ab4b", "purposes": { "compileIndex": "60c20164d8c3a443", + "curationPath": "37dd7da1d8d13d69", "fold": "475b654bf901e6d9", + "indexFromMap": "f0606664e619eacc", "legacyRetrieve": "01c5d2059f3b0d6f", + "loadLookup": "824e133bd24d973a", "matchUnit": "02ae5ca770fd9391", "normalizeQuery": "31b57c6910e130c5", + "publishLookup": "be63fd7f0ef49a09", "retrieve": "acf77b261ff3fd1e", "splitWords": "858bcfb0070cb9f5" } @@ -19765,6 +19835,7 @@ "structure-generation", "structure-state", "activity-log", + "structure-retrieval", "structure-commit" ], "functions": { @@ -19833,8 +19904,15 @@ "line": 181, "purpose": "The tracked map (committed view plus hand edits) as the generation prior" }, + "publishLookup": { + "line": 190, + "params": [ + "result" + ], + "purpose": "After a working-view publish, refresh the derived lookup index (STR-03)." + }, "mirrorWorkingView": { - "line": 187, + "line": 204, "params": [ "result", "discardsAuthored" @@ -19842,7 +19920,7 @@ "purpose": "After a working-view publish, keep an untracked map file equal to it." }, "builtFrom": { - "line": 204, + "line": 221, "params": [ "structure", "report", @@ -19852,37 +19930,37 @@ "purpose": "`reference` is the artifact's own previous content (for lastUpdated and" }, "runFull": { - "line": 222, + "line": 239, "purpose": "--full (STR-02c D3): rebuild the working view. Hand-written prose comes" }, "runDelta": { - "line": 246, + "line": 264, "params": [ "candidates" ] }, "reportMutation": { - "line": 287, + "line": 306, "params": [ "result", "command" ] }, "runCheck": { - "line": 318 + "line": 337 }, "runStaged": { - "line": 364, + "line": 383, "params": [ "command = 'staged'" ], "purpose": "--staged (STR-02b): build the commit's map from the staged snapshot," }, "main": { - "line": 383 + "line": 402 } }, - "sizeBytes": 19119, + "sizeBytes": 19854, "extraction": { "status": "parsed" }, @@ -19890,6 +19968,7 @@ "purposes": { "builtFrom": "0f3b5f39b1a05377", "mirrorWorkingView": "2b24665961d385c6", + "publishLookup": "5d6adb89f9495625", "resolveProjectRoot": "bdaf2016db7ccaef", "runFull": "018e3a0230855dd1", "runStaged": "5d25180306f61337", @@ -23319,7 +23398,7 @@ ] } }, - "sizeBytes": 28510, + "sizeBytes": 30143, "extraction": { "status": "parsed" }, @@ -23433,7 +23512,7 @@ ] } }, - "sizeBytes": 43964, + "sizeBytes": 45042, "extraction": { "status": "parsed" }, @@ -24142,7 +24221,7 @@ ] } }, - "sizeBytes": 37257, + "sizeBytes": 38349, "extraction": { "status": "parsed" }, @@ -24228,7 +24307,9 @@ "test/structureRetrieval.test": { "file": "test/structureRetrieval.test.js", "description": "structure-retrieval tests (STR-03): evidence tiers, curated precedence,", - "exports": [], + "exports": [ + "runAlpha" + ], "depends": [ "node:test", "node:assert/strict", @@ -24249,9 +24330,16 @@ }, "legacyProject": { "line": 222 + }, + "lookupProject": { + "line": 255, + "params": [ + "t", + "files = { 'src/a.js'" + ] } }, - "sizeBytes": 13860, + "sizeBytes": 18299, "extraction": { "status": "parsed" }, @@ -26334,6 +26422,6 @@ "truncated": false, "samples": [] }, - "revision": "8e89c6fc955d86d991383345d2160ec4c97f950c910236d65ca47c6d6472744b" + "revision": "9208a485423116b0dce84ba1d0ee9bd81382b15d6700a0e7ee7bd636118221f4" } } diff --git a/.frame/specs/str-03-local-file-retrieval/outcome.md b/.frame/specs/str-03-local-file-retrieval/outcome.md index 7e2acf82..6af96615 100644 --- a/.frame/specs/str-03-local-file-retrieval/outcome.md +++ b/.frame/specs/str-03-local-file-retrieval/outcome.md @@ -38,3 +38,20 @@ Development split, in process on the pinned map: recall@5 95.1%, P@1 98.3%, hook _Captured: 2026-10-01 · 2 file change(s)_ --- + +## T03 — Lookup index publication + +`structure-retrieval.js` gained three functions: +- `publishLookup(root, { mapPath, curationPath })`: atomic `.frame/runtime/structure/lookup.json` (compact JSON) recording the source map path and signature (ino/size/mtime/ctime), the intent-map.json signature, revision and algorithm. It returns `unchanged` while those inputs match, and sets `oversize: true` above 2 MiB. It never throws. +- `loadLookup(root, { maxBytes })`: `fresh` / `stale` / `missing` / `oversize` / `invalid`; above the cap it doesn't read the file. +- `indexFromMap(root, { maxBytes })`: the in-memory fallback for D6, which never writes. + +The lifecycle worker (`lookup` in its result) and `update-structure.js` (`--full` and file updates, `lookup` in the envelope, with a stderr warning on failure) publish after every working-view publication. + +Deviation: `update-structure.js` does not list the new helper in its activation requirements. An existing bootstrap test records that the parser itself must not depend on `structure-read`. The call is guarded instead, so an older `.frame/bin/` just skips the index. `find-module.js` and `module-hint.js` require `structure-retrieval.js` for activation, and it ships as a helper (bootstrap list, `package.json`). + +Unplanned, separate: a commit hook hang (170 s in `git hash-object --stdin`, STR-02c's mirror) was fixed on its own branch, `fix/structure-hook-hash-in-process` (PR #168), outside this spec's footprint. Files touched: `scripts/structure-retrieval.js`, `scripts/structure-lifecycle.js`, `scripts/update-structure.js`, `src/main/structureBootstrap.js`, `package.json`, `test/structureRetrieval.test.js`, `test/structureLifecycle.test.js`, `test/projectAgnostic.test.js`, `test/scriptsProjectRoot.test.js`. + +_Captured: 2026-10-01 · 9 file change(s)_ + +--- diff --git a/package.json b/package.json index 5e3ae9c2..58429c91 100644 --- a/package.json +++ b/package.json @@ -94,6 +94,7 @@ "scripts/structure-read.js", "scripts/structure-lifecycle.js", "scripts/structure-commit.js", + "scripts/structure-retrieval.js", "scripts/structure-ignore.js", "scripts/structure-ignore.LICENSE", "scripts/find-module.js", diff --git a/scripts/structure-lifecycle.js b/scripts/structure-lifecycle.js index 24a96508..449f1ebc 100644 --- a/scripts/structure-lifecycle.js +++ b/scripts/structure-lifecycle.js @@ -398,6 +398,16 @@ function reconcile(root, job = {}) { } } + // The search index follows the working view (STR-03); never fails the map. + let lookup = null; + if (result.artifact === 'written' || result.artifact === 'unchanged') { + try { + lookup = require('./structure-retrieval').publishLookup(root, { mapPath: workingPath }).status; + } catch (e) { + lookup = null; // an older .frame/bin/ without the retrieval helper + } + } + // Keep the tracked map on the committed view without losing hand edits: // repair a pathspec commit's index, restore a generated-only difference // (STR-02c D6, D8). @@ -443,6 +453,7 @@ function reconcile(root, job = {}) { reused: observed.reused, cacheHits: cache.stats.hits, trackedMap, + lookup, ms: Date.now() - startedAt }; } diff --git a/scripts/structure-retrieval.js b/scripts/structure-retrieval.js index 1c61cd3d..6aff9d63 100644 --- a/scripts/structure-retrieval.js +++ b/scripts/structure-retrieval.js @@ -4,9 +4,10 @@ * * One engine serves both `find-module.js` (explicit lookup) and * `module-hint.js` (the automatic search hint); the hook only takes the - * stronger evidence tiers. Pure and dependency-free: no Git, no subprocess, - * no network, nothing that builds or rewrites the map. Ships to - * `.frame/bin/`. + * stronger evidence tiers. Dependency-free: no Git, no subprocess, no + * network, nothing that builds or rewrites the map. The only file it writes + * is its own derived `lookup.json` (publishLookup, called by the map's + * writers). Ships to `.frame/bin/`. * * Evidence tiers, strongest first: * 1 path exact repo-relative path, or a path suffix ("lang/python.js") @@ -367,6 +368,162 @@ function retrieveUnits(index, allUnits, mode, limit, options) { }; } +/* ------------------------- the published lookup file ------------------------ */ +// +// `.frame/runtime/structure/lookup.json` is compiled after every working-view +// publication (STR-03 A2), so a search hook reads a small file instead of the +// whole map. It records the signature of the map file and of intent-map.json +// it was built from; a mismatch makes it stale and a reader never trusts it. + +const fs = require('fs'); +const path = require('path'); + +function lookupPath(root) { + return path.join(root, '.frame', 'runtime', 'structure', 'lookup.json'); +} + +/** intent-map.json beside the running scripts (scripts/ or .frame/bin/). */ +function curationPath(dir = __dirname) { + return path.join(dir, 'intent-map.json'); +} + +function readCuration(file) { + try { + return JSON.parse(fs.readFileSync(file, 'utf8')); + } catch { + return {}; + } +} + +function signatureOf(file) { + try { + const s = fs.statSync(file); + return { ino: s.ino, size: s.size, mtimeMs: s.mtimeMs, ctimeMs: s.ctimeMs }; + } catch { + return null; + } +} + +function sameSignature(a, b) { + if (a === null || b === null || a === undefined || b === undefined) return a == b; // eslint-disable-line eqeqeq + return a.ino === b.ino && a.size === b.size && a.mtimeMs === b.mtimeMs && a.ctimeMs === b.ctimeMs; +} + +/** + * Compile and atomically publish lookup.json from the map at `mapPath` + * (default: the read view). Never throws: a failure is reported and the + * map publication it follows stands. + * + * Returns { status: 'published' | 'unchanged' | 'failed', bytes?, oversize?, reason? }. + */ +function publishLookup(root, options = {}) { + try { + const read = require('./structure-read'); + const mapFile = options.mapPath || read.resolveReadPath(root); + const curationFile = options.curationPath || curationPath(); + const relMap = path.relative(root, mapFile).split(path.sep).join('/'); + const source = { path: relMap, signature: signatureOf(mapFile) }; + const curation = { signature: signatureOf(curationFile) }; + if (!source.signature) return { status: 'failed', reason: 'no-map' }; + + const target = lookupPath(root); + try { + const current = JSON.parse(fs.readFileSync(target, 'utf8')); + if (current.version === INDEX_VERSION && current.algorithm === ALGORITHM && current.source + && current.source.path === relMap && sameSignature(current.source.signature, source.signature) + && current.curation && sameSignature(current.curation.signature, curation.signature)) { + return { status: 'unchanged', bytes: fs.statSync(target).size, oversize: Boolean(current.oversize) }; + } + } catch { /* none or unreadable: rebuild */ } + + const structure = JSON.parse(fs.readFileSync(mapFile, 'utf8')); + const index = compileIndex(structure, readCuration(curationFile), { source, curation, oversize: false }); + let text = JSON.stringify(index); + if (Buffer.byteLength(text) > LIMITS.hookIndexBytes) { + index.oversize = true; + text = JSON.stringify(index); + } + fs.mkdirSync(path.dirname(target), { recursive: true }); + const tmp = `${target}.tmp-${process.pid}`; + fs.writeFileSync(tmp, text); + fs.renameSync(tmp, target); + return { status: 'published', bytes: Buffer.byteLength(text), oversize: index.oversize }; + } catch (err) { + return { status: 'failed', reason: (err && err.code) || 'error', message: err && err.message }; + } +} + +/** + * Read the published index for a lookup. Never compiles, never writes. + * + * options: { maxBytes } — larger files are not even read. + * Returns { state: 'fresh' | 'stale' | 'missing' | 'oversize' | 'invalid', index? }: + * fresh only when it was built from the current read view and curation by + * this algorithm. A stale index is still returned for callers that may use + * it with care; hooks must not. + */ +function loadLookup(root, options = {}) { + const file = lookupPath(root); + let size; + try { + size = fs.statSync(file).size; + } catch { + return { state: 'missing' }; + } + if (options.maxBytes && size > options.maxBytes) return { state: 'oversize' }; + let index; + try { + index = JSON.parse(fs.readFileSync(file, 'utf8')); + } catch { + return { state: 'invalid' }; + } + if (!index || index.version !== INDEX_VERSION || !Array.isArray(index.files) || !index.terms || !Array.isArray(index.concepts)) { + return { state: 'invalid' }; + } + if (index.oversize && options.maxBytes) return { state: 'oversize' }; + let fresh = index.algorithm === ALGORITHM; + try { + const read = require('./structure-read'); + const mapFile = read.resolveReadPath(root); + const relMap = path.relative(root, mapFile).split(path.sep).join('/'); + fresh = fresh && index.source && index.source.path === relMap && sameSignature(index.source.signature, signatureOf(mapFile)) + && index.curation && sameSignature(index.curation.signature, signatureOf(options.curationPath || curationPath())); + } catch { + fresh = false; + } + return { state: fresh ? 'fresh' : 'stale', index }; +} + +/** + * Compile an index in memory from the read view — the fallback when no + * fresh lookup.json exists (a clone without a running worker). Bounded: + * a map larger than `maxBytes` is not read. + * Returns { state: 'compiled' | 'missing' | 'oversize' | 'invalid', index?, structure? }. + */ +function indexFromMap(root, options = {}) { + let mapFile; + try { + mapFile = require('./structure-read').resolveReadPath(root); + } catch { + return { state: 'missing' }; + } + let size; + try { + size = fs.statSync(mapFile).size; + } catch { + return { state: 'missing' }; + } + if (options.maxBytes && size > options.maxBytes) return { state: 'oversize' }; + let structure; + try { + structure = JSON.parse(fs.readFileSync(mapFile, 'utf8')); + } catch { + return { state: 'invalid' }; + } + if (!structure || typeof structure !== 'object' || !structure.modules) return { state: 'invalid' }; + return { state: 'compiled', index: compileIndex(structure, readCuration(options.curationPath || curationPath())), structure }; +} + /* --------------------------------- legacy -------------------------------- */ /** @@ -447,6 +604,11 @@ function legacyHookLookup(index, intentMap, keyword) { module.exports = { compileIndex, + publishLookup, + loadLookup, + indexFromMap, + lookupPath, + curationPath, normalizeQuery, retrieve, legacyRetrieve, diff --git a/scripts/update-structure.js b/scripts/update-structure.js index 52b40bcb..8682c571 100644 --- a/scripts/update-structure.js +++ b/scripts/update-structure.js @@ -183,6 +183,23 @@ function trackedPrior() { return baseline.status === 'valid' ? baseline.data : null; } +/** + * After a working-view publish, refresh the derived lookup index (STR-03). + * A failure is reported in the result and never fails the map. + */ +function publishLookup(result) { + if (result.artifact !== 'written' && result.artifact !== 'unchanged') return; + let retrieval; + try { + retrieval = require('./structure-retrieval'); + } catch (err) { + return; // an older .frame/bin/ without the retrieval helper + } + const lookup = retrieval.publishLookup(ROOT_DIR, { mapPath: state.workingViewPath(ROOT_DIR) }); + result.lookup = lookup.status; + if (lookup.status === 'failed') warn(`⚠ Lookup index not published (${lookup.reason}) — searches fall back to the map.`); +} + /** After a working-view publish, keep an untracked map file equal to it. */ function mirrorWorkingView(result, discardsAuthored) { if (result.artifact !== 'written' && result.artifact !== 'unchanged') return; @@ -240,6 +257,7 @@ function runFull() { } }); mirrorWorkingView(result, discarded); + publishLookup(result); return result; } @@ -280,6 +298,7 @@ function runDelta(candidates) { } }); mirrorWorkingView(result, discarded); + publishLookup(result); result.policyInputChanged = policyInputChanged; return result; } diff --git a/src/main/structureBootstrap.js b/src/main/structureBootstrap.js index de57d123..ac8e5e05 100644 --- a/src/main/structureBootstrap.js +++ b/src/main/structureBootstrap.js @@ -51,7 +51,7 @@ const SCRIPTS_SOURCE_DIR = path.join(__dirname, '..', '..', 'scripts'); const HELPER_FILES = [ 'structure-ignore.js', 'structure-ignore.LICENSE', 'structure-discovery.js', 'structure-generation.js', 'structure-state.js', 'structure-snapshot.js', - 'structure-read.js', 'structure-commit.js', 'toolVocabulary.js', 'redact.js', 'activity-log.js' + 'structure-read.js', 'structure-commit.js', 'structure-retrieval.js', 'toolVocabulary.js', 'redact.js', 'activity-log.js' ]; const ENTRY_FILES = [ 'update-structure.js', 'structure-lifecycle.js', 'find-module.js', 'check-freshness.js', @@ -75,9 +75,9 @@ const ENTRY_REQUIRES = { 'update-structure.js': PARSER_REQUIRES, 'structure-lifecycle.js': LIFECYCLE_REQUIRES, // Readers of the freshness contract (STR-02). - 'find-module.js': ['structure-read.js'], + 'find-module.js': ['structure-read.js', 'structure-retrieval.js'], 'check-freshness.js': ['structure-read.js'], - 'module-hint.js': ['structure-read.js'] + 'module-hint.js': ['structure-read.js', 'structure-retrieval.js'] }; // Historical name list, kept for readers of this module's exports. const PARSER_FILES = [...ENTRY_FILES, 'intent-map.json', ...HELPER_FILES]; diff --git a/test/projectAgnostic.test.js b/test/projectAgnostic.test.js index d7ad8e2e..c4a979b4 100644 --- a/test/projectAgnostic.test.js +++ b/test/projectAgnostic.test.js @@ -622,3 +622,35 @@ test('templates: the pre-commit template is recognized; earlier unmodified templ } assert.equal(templates.classifyStructureHook('#!/bin/sh\necho mine\n'), null); }); + +/* ---------------------- STR-03: the derived lookup index ---------------------- */ + +test('cli: --full and file updates publish lookup.json; a failed index never fails the map', () => { + const dir = tmpProject({ 'src/a.js': '// A\nfunction alphaThing() {}\nmodule.exports = { alphaThing };\n' }); + try { + const full = envelopeOf(runParser(dir, ['--full', '--json'])); + assert.equal(full.exitCode, 0); + assert.equal(full.lookup, 'published'); + const lookupFile = path.join(dir, '.frame', 'runtime', 'structure', 'lookup.json'); + assert.ok(JSON.parse(fs.readFileSync(lookupFile, 'utf8')).terms['5:alphathing']); + + fs.writeFileSync(path.join(dir, 'src', 'b.js'), '// B\nfunction betaThing() {}\n'); + const delta = envelopeOf(runParser(dir, ['src/b.js', '--json'])); + assert.equal(delta.lookup, 'published'); + assert.ok(JSON.parse(fs.readFileSync(lookupFile, 'utf8')).terms['5:betathing']); + + // a directory where the index should go: publication fails, the map does not + fs.rmSync(lookupFile); + fs.mkdirSync(lookupFile); + fs.writeFileSync(path.join(lookupFile, 'blocker'), 'x'); + fs.writeFileSync(path.join(dir, 'src', 'c.js'), '// C\n'); + const res = runParser(dir, ['--full', '--json']); + const env = envelopeOf(res); + assert.equal(env.exitCode, 0); + assert.equal(env.lookup, 'failed'); + assert.match(res.stderr, /Lookup index not published/); + assert.ok(JSON.parse(fs.readFileSync(path.join(dir, '.frame', 'runtime', 'structure', 'working.json'), 'utf8')).modules.c, 'the map was published'); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +}); diff --git a/test/scriptsProjectRoot.test.js b/test/scriptsProjectRoot.test.js index 249df00d..4081f278 100644 --- a/test/scriptsProjectRoot.test.js +++ b/test/scriptsProjectRoot.test.js @@ -814,3 +814,21 @@ test('find-module and check-freshness read the live working view before the trac fs.rmSync(r.dir, { recursive: true, force: true }); } }); + +test('a copied .frame/bin/ publishes lookup.json without Frame\'s repository', () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'frame-03-bin-')); + try { + fs.mkdirSync(path.join(dir, 'src'), { recursive: true }); + fs.writeFileSync(path.join(dir, 'src', 'gadget.js'), '// Gadget\nfunction buildGadget() {}\nmodule.exports = { buildGadget };\n'); + structureBootstrap.copyParserScripts(dir); + assert.ok(fs.existsSync(path.join(dir, '.frame', 'bin', 'structure-retrieval.js'))); + const env = { ...process.env, FRAME_PROJECT_ROOT: undefined }; + const run = spawnSync('node', [path.join(dir, '.frame', 'bin', 'structure-lifecycle.js'), '--once'], { cwd: dir, env, encoding: 'utf8' }); + assert.equal(run.status, 0, run.stderr); + const index = JSON.parse(fs.readFileSync(path.join(dir, '.frame', 'runtime', 'structure', 'lookup.json'), 'utf8')); + assert.ok(index.terms['5:buildgadget']); + assert.ok(index.curation.signature, 'built with the copied intent-map.json'); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +}); diff --git a/test/structureLifecycle.test.js b/test/structureLifecycle.test.js index 041236a6..4eb51748 100644 --- a/test/structureLifecycle.test.js +++ b/test/structureLifecycle.test.js @@ -862,3 +862,24 @@ test('update-structure --full rebuilds the working view and leaves a tracked map fs.rmSync(dir, { recursive: true, force: true }); } }); + +/* ---------------- STR-03: the lookup index follows the working view ---------------- */ + +test('the worker publishes lookup.json with the working view and refreshes it when the map changes', () => { + const dir = project({ 'src/a.js': '// A\nfunction alphaThing() {}\nmodule.exports = { alphaThing };\n' }); + try { + const first = reconcileAndRecord(dir); + assert.equal(first.lookup, 'published'); + const lookupFile = path.join(dir, '.frame', 'runtime', 'structure', 'lookup.json'); + const index = JSON.parse(fs.readFileSync(lookupFile, 'utf8')); + assert.equal(index.source.path, '.frame/runtime/structure/working.json'); + assert.ok(index.terms['5:alphathing']); + assert.equal(reconcileAndRecord(dir).lookup, 'unchanged'); + + fs.writeFileSync(path.join(dir, 'src', 'b.js'), '// B\nfunction betaThing() {}\nmodule.exports = { betaThing };\n'); + assert.equal(reconcileAndRecord(dir).lookup, 'published'); + assert.ok(JSON.parse(fs.readFileSync(lookupFile, 'utf8')).terms['5:betathing']); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +}); diff --git a/test/structureRetrieval.test.js b/test/structureRetrieval.test.js index 2cf6efab..be93494c 100644 --- a/test/structureRetrieval.test.js +++ b/test/structureRetrieval.test.js @@ -249,3 +249,76 @@ test('the legacy hook engine takes the first keyword that hits a curated tier', assert.equal(R.legacyRetrieve(STRUCTURE, CURATION, { mode: 'hook', words: ['checkghauth'] }).status, 'no-match', 'no deep tier in hooks'); assert.equal(R.legacyRetrieve({ modules: {} }, {}, { mode: 'cli', query: 'x' }).status, 'unavailable'); }); + +/* ------------------------- the published lookup file ------------------------ */ + +function lookupProject(t, files = { 'src/a.js': '// Alpha module\nfunction runAlpha() {}\nmodule.exports = { runAlpha };\n' }) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'frame-lookup-')); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(dir, rel)), { recursive: true }); + fs.writeFileSync(path.join(dir, rel), content); + } + const working = path.join(dir, '.frame', 'runtime', 'structure', 'working.json'); + fs.mkdirSync(path.dirname(working), { recursive: true }); + fs.writeFileSync(working, JSON.stringify({ generation: { revision: 'r1' }, modules: { a: mod('src/a.js', 'Alpha module', { functions: { runAlpha: {} } }) }, intentIndex: {} })); + const curation = path.join(dir, 'intent-map.json'); + fs.writeFileSync(curation, JSON.stringify({ alpha: { synonyms: ['first'] } })); + return { dir, working, curation }; +} + +test('publishLookup writes a fresh index once and reports unchanged until an input changes', (t) => { + const { dir, working, curation } = lookupProject(t); + const first = R.publishLookup(dir, { curationPath: curation }); + assert.equal(first.status, 'published'); + assert.equal(first.oversize, false); + const loaded = R.loadLookup(dir, { curationPath: curation, maxBytes: R.LIMITS.hookIndexBytes }); + assert.equal(loaded.state, 'fresh'); + assert.equal(loaded.index.source.path, '.frame/runtime/structure/working.json'); + assert.deepEqual(R.retrieve(loaded.index, 'runAlpha', { mode: 'hook' }).candidates.map((c) => c.path), ['src/a.js']); + assert.equal(R.publishLookup(dir, { curationPath: curation }).status, 'unchanged'); + + // a changed map makes it stale until it is republished + fs.writeFileSync(working, fs.readFileSync(working, 'utf8').replace('Alpha module', 'Alpha module, edited')); + assert.equal(R.loadLookup(dir, { curationPath: curation }).state, 'stale'); + assert.equal(R.publishLookup(dir, { curationPath: curation }).status, 'published'); + assert.equal(R.loadLookup(dir, { curationPath: curation }).state, 'fresh'); + + // so does a changed curation file + fs.writeFileSync(curation, JSON.stringify({ alpha: { synonyms: ['first', 'premier'] } })); + assert.equal(R.loadLookup(dir, { curationPath: curation }).state, 'stale'); + assert.equal(R.publishLookup(dir, { curationPath: curation }).status, 'published'); +}); + +test('loadLookup reports missing, invalid and oversize indexes without reading past the cap', (t) => { + const { dir, curation } = lookupProject(t); + assert.equal(R.loadLookup(dir).state, 'missing'); + fs.writeFileSync(R.lookupPath(dir), '{ not json'); + assert.equal(R.loadLookup(dir).state, 'invalid'); + fs.writeFileSync(R.lookupPath(dir), JSON.stringify({ version: 999 })); + assert.equal(R.loadLookup(dir).state, 'invalid'); + R.publishLookup(dir, { curationPath: curation, mapPath: undefined }); + assert.equal(R.loadLookup(dir, { maxBytes: 10 }).state, 'oversize'); +}); + +test('an index above the hook cap is published, flagged oversize, and refused to hooks', (t) => { + const modules = {}; + for (let i = 0; i < 9000; i++) modules[`m${i}`] = mod(`src/area${i % 50}/widgetNumber${i}.js`, `Widget number ${i} with a long description to grow the index`, { functions: { [`handleWidget${i}`]: {} } }); + const { dir, working, curation } = lookupProject(t); + fs.writeFileSync(working, JSON.stringify({ modules, intentIndex: {} })); + const r = R.publishLookup(dir, { curationPath: curation }); + assert.equal(r.status, 'published'); + assert.equal(r.oversize, true); + assert.ok(r.bytes > R.LIMITS.hookIndexBytes); + assert.equal(R.loadLookup(dir, { curationPath: curation, maxBytes: R.LIMITS.hookIndexBytes }).state, 'oversize'); + assert.equal(R.loadLookup(dir, { curationPath: curation }).state, 'fresh', 'the CLI may still use it'); +}); + +test('indexFromMap compiles the read view in memory and refuses a map above the cap', (t) => { + const { dir, curation } = lookupProject(t); + const r = R.indexFromMap(dir, { curationPath: curation }); + assert.equal(r.state, 'compiled'); + assert.deepEqual(R.retrieve(r.index, 'first', { mode: 'hook' }).candidates, [], 'no alpha concept in the intentIndex'); + assert.equal(R.indexFromMap(dir, { maxBytes: 10 }).state, 'oversize'); + assert.equal(fs.existsSync(R.lookupPath(dir)), false, 'never writes'); +}); From ffc1e3abeb71e921b0259e9b9f73cad3412d07d4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kaan=20=C3=96zhan?= Date: Thu, 1 Oct 2026 16:20:40 +0300 Subject: [PATCH 5/9] feat(structure): find-module on the shared engine with --json, --limit and engine selection (STR-03 T04) --- .frame/STRUCTURE.json | 188 +++++--- .../str-03-local-file-retrieval/outcome.md | 18 + scripts/find-module.js | 401 +++++++++--------- scripts/structure-retrieval.js | 7 + test/findModule.test.js | 167 ++++++++ test/scriptsProjectRoot.test.js | 5 +- 6 files changed, 531 insertions(+), 255 deletions(-) create mode 100644 test/findModule.test.js diff --git a/.frame/STRUCTURE.json b/.frame/STRUCTURE.json index 0a811d37..ec28066d 100644 --- a/.frame/STRUCTURE.json +++ b/.frame/STRUCTURE.json @@ -16686,71 +16686,98 @@ "depends": [ "fs", "path", - "child_process", - "structure-read" + "structure-read", + "structure-retrieval" ], "functions": { "resolveProjectRoot": { - "line": 26, + "line": 40, "purpose": "Which project this run is about. `__dirname/..` was wrong for the shipped" }, - "repairCommand": { - "line": 37, + "scriptRel": { + "line": 53, "params": [ - "root" + "name" ] }, - "loadStructure": { - "line": 47 + "repairCommand": { + "line": 58 }, - "loadIntentMap": { - "line": 60, - "purpose": "Load the curated concept map (synonyms live here). Sits next to this" + "parseArgs": { + "line": 64, + "params": [ + "argv" + ] }, - "stalenessBanner": { - "line": 77, + "resolveEngine": { + "line": 83, "params": [ - "structure" + "flag" ], - "purpose": "One-line warning when STRUCTURE.json is stale. The date comparison is a" + "purpose": "--retrieval, then project.retrieval.engine, then the shipped default." }, - "searchIntentIndex": { + "loadStructure": { + "line": 101 + }, + "existsInProject": { "line": 111, "params": [ - "structure", - "keyword" + "rel" ], - "purpose": "Search intentIndex for matching features" + "purpose": "A candidate is shown as present only if it is a regular file inside the project." + }, + "v2Index": { + "line": 124, + "purpose": "The published index when it is current, else one compiled from the map." + }, + "freshnessLines": { + "line": 137, + "params": [ + "descriptor" + ], + "purpose": "How current the map is, from the lifecycle receipt only (STR-02). No Git," }, "listFeatures": { - "line": 179, + "line": 147, "params": [ "structure" - ], - "purpose": "List all features in intentIndex" + ] }, - "printResults": { - "line": 197, + "printLegacy": { + "line": 162, "params": [ "structure", - "results", - "keyword" + "result", + "query" ], - "purpose": "Format and print results" + "purpose": "The pre-STR-03 listing: one block per matched feature, with its IPC channels." + }, + "printV2": { + "line": 185, + "params": [ + "result", + "query" + ] + }, + "main": { + "line": 204 + }, + "readCuration": { + "line": 282 } }, - "sizeBytes": 9331, + "sizeBytes": 11585, "extraction": { "status": "parsed" }, "provenance": { "purposes": { - "listFeatures": "abee8c0e66184dfd", - "loadIntentMap": "97c2e373d690a85c", - "printResults": "92ba06a56754bee9", + "existsInProject": "3efd68ebe169d576", + "freshnessLines": "1582f67449e3f170", + "printLegacy": "8df21ce15157510e", + "resolveEngine": "7af91aa82a028c2f", "resolveProjectRoot": "bdaf2016db7ccaef", - "searchIntentIndex": "331ec57f0459ec19", - "stalenessBanner": "ef7f87ea90f30ceb" + "v2Index": "cc60ff33199e094f" } } }, @@ -19077,6 +19104,8 @@ "legacyRetrieve", "fold", "splitWords", + "ENGINES", + "DEFAULT_ENGINE", "TIER", "HOOK_MAX_TIER", "LIMITS", @@ -19091,40 +19120,40 @@ ], "functions": { "fold": { - "line": 63, + "line": 68, "params": [ "text" ], "purpose": "Case- and accent-insensitive search form; Turkish letters fold to ASCII." }, "splitWords": { - "line": 73, + "line": 78, "params": [ "text" ], "purpose": "Identifier words: camelCase, snake_case, kebab-case, dotted and path parts." }, "normalizeQuery": { - "line": 90, + "line": 95, "params": [ "raw" ], "purpose": "A raw query → its search units. Regex syntax is dropped, alternations" }, "basenameOf": { - "line": 121, + "line": 126, "params": [ "file" ] }, "stripExt": { - "line": 125, + "line": 130, "params": [ "name" ] }, "compileIndex": { - "line": 139, + "line": 144, "params": [ "structure", "curation = {}", @@ -19133,7 +19162,7 @@ "purpose": "Compile the lookup index from a map (STR-02c read view) and the curated" }, "matchUnit": { - "line": 224, + "line": 229, "params": [ "index", "unit", @@ -19142,7 +19171,7 @@ "purpose": "Every match one unit has: Map(fileId → best tier) plus the concepts it named." }, "retrieve": { - "line": 285, + "line": 290, "params": [ "index", "query", @@ -19151,7 +19180,7 @@ "purpose": "Rank the files that explain the query." }, "retrieveUnits": { - "line": 297, + "line": 302, "params": [ "index", "allUnits", @@ -19161,39 +19190,39 @@ ] }, "lookupPath": { - "line": 381, + "line": 386, "params": [ "root" ] }, "curationPath": { - "line": 386, + "line": 391, "params": [ "dir = __dirname" ], "purpose": "intent-map.json beside the running scripts (scripts/ or .frame/bin/)." }, "readCuration": { - "line": 390, + "line": 395, "params": [ "file" ] }, "signatureOf": { - "line": 398, + "line": 403, "params": [ "file" ] }, "sameSignature": { - "line": 407, + "line": 412, "params": [ "a", "b" ] }, "publishLookup": { - "line": 419, + "line": 424, "params": [ "root", "options = {}" @@ -19201,7 +19230,7 @@ "purpose": "Compile and atomically publish lookup.json from the map at `mapPath`" }, "loadLookup": { - "line": 465, + "line": 470, "params": [ "root", "options = {}" @@ -19209,7 +19238,7 @@ "purpose": "Read the published index for a lookup. Never compiles, never writes." }, "indexFromMap": { - "line": 503, + "line": 508, "params": [ "root", "options = {}" @@ -19217,7 +19246,7 @@ "purpose": "Compile an index in memory from the read view — the fallback when no" }, "legacyRetrieve": { - "line": 536, + "line": 541, "params": [ "structure", "curation", @@ -19226,7 +19255,7 @@ "purpose": "Today's behavior, kept selectable as the rollback path (STR-03 D2):" }, "legacyHookLookup": { - "line": 589, + "line": 594, "params": [ "index", "intentMap", @@ -19234,7 +19263,7 @@ ] } }, - "sizeBytes": 25653, + "sizeBytes": 25912, "extraction": { "status": "parsed" }, @@ -22004,6 +22033,45 @@ } } }, + "test/findModule.test": { + "file": "test/findModule.test.js", + "description": "find-module CLI (STR-03 T04): legacy-compatible output, the bounded", + "exports": [], + "depends": [ + "node:test", + "node:assert/strict", + "fs", + "os", + "path", + "child_process", + "scripts/structure-retrieval" + ], + "functions": { + "project": { + "line": 36, + "params": [ + "t", + "{ structure = STRUCTURE", + "engine } = {}" + ] + }, + "find": { + "line": 49, + "params": [ + "dir", + "args", + "env = {}" + ] + } + }, + "sizeBytes": 9262, + "extraction": { + "status": "parsed" + }, + "provenance": { + "description": "1f35f244df9d4926" + } + }, "test/fixtures/django-app/manage": { "file": "test/fixtures/django-app/manage.py", "description": "", @@ -23500,7 +23568,7 @@ ] }, "frameRepo": { - "line": 619, + "line": 620, "params": [ "prefix" ] @@ -23512,7 +23580,7 @@ ] } }, - "sizeBytes": 45042, + "sizeBytes": 45133, "extraction": { "status": "parsed" }, @@ -26348,7 +26416,7 @@ "extraction": { "coverage": "complete", "counts": { - "parsed": 294, + "parsed": 295, "unsupported": 83, "partial": 0 } @@ -26395,9 +26463,9 @@ } }, "counts": { - "visitedEntries": 465, + "visitedEntries": 466, "traversedDirectories": 80, - "eligibleFiles": 377, + "eligibleFiles": 378, "unreadable": 0, "binaryFiles": 5, "symlinks": 0, @@ -26414,7 +26482,7 @@ "config": 0, "depth": 0 }, - "indexedFiles": 377, + "indexedFiles": 378, "legacyGroups": 0 }, "diagnostics": { @@ -26422,6 +26490,6 @@ "truncated": false, "samples": [] }, - "revision": "9208a485423116b0dce84ba1d0ee9bd81382b15d6700a0e7ee7bd636118221f4" + "revision": "2eaaedc700d2f754c9b5e1c59391d16d9c02580c9c1c2a13a7ef2e5a4bcef624" } } diff --git a/.frame/specs/str-03-local-file-retrieval/outcome.md b/.frame/specs/str-03-local-file-retrieval/outcome.md index 6af96615..a1c57301 100644 --- a/.frame/specs/str-03-local-file-retrieval/outcome.md +++ b/.frame/specs/str-03-local-file-retrieval/outcome.md @@ -55,3 +55,21 @@ Unplanned, separate: a commit hook hang (170 s in `git hash-object --stdin`, STR _Captured: 2026-10-01 · 9 file change(s)_ --- + +## T04 — Explicit lookup on the shared engine + +`scripts/find-module.js` now uses `structure-retrieval.js` for both engines. Engine selection: `--retrieval=legacy|v2`, else `project.retrieval.engine`, else `DEFAULT_ENGINE` (still `legacy` until T06 decides). An invalid value falls back with a stderr note. + +What each engine does: +- `legacy` keeps the feature listing, IPC line, synonym tier and "⚠ file missing" marker (via `legacyRetrieve`). +- `v2` prints "Files for" or "Candidates for … (several match equally)" with `[evidence]` per line. It uses a fresh `lookup.json` when there is one, otherwise compiles from the map. + +New flags: +- `--json`: one `frame.lookup/1` envelope with status, engine, freshness, reasons, candidates and truncation. `no-match` exits 0; a missing or corrupt map gives `unavailable` with exit 1. +- `--limit`: default 8, at most 20. + +Candidates are present only as regular files inside the project (a symlink escaping it counts as missing). The Git/`--check` date banner is gone (D13). An unknown map now says "⚠ Map: unverified ()", and the STR-02c reader test was updated for that. Files touched: `scripts/find-module.js`, `scripts/structure-retrieval.js` (engine constants), `test/findModule.test.js`, `test/scriptsProjectRoot.test.js`. + +_Captured: 2026-10-01 · 4 file change(s)_ + +--- diff --git a/scripts/find-module.js b/scripts/find-module.js index a832cbbf..b2f71f20 100644 --- a/scripts/find-module.js +++ b/scripts/find-module.js @@ -1,22 +1,36 @@ #!/usr/bin/env node /** - * Module Finder — Fast file lookup using STRUCTURE.json intentIndex + * Module Finder — from a file name, path, symbol or concept to the files * * Usage: - * node scripts/find-module.js # Search by feature/concept - * node scripts/find-module.js --list # List all features + * node scripts/find-module.js # file, path, symbol or concept + * node scripts/find-module.js --list # list all concepts + * node scripts/find-module.js --json # one bounded envelope on stdout + * node scripts/find-module.js --limit 12 # up to 20 files (default 8) + * node scripts/find-module.js --retrieval=legacy|v2 * * Examples: * node scripts/find-module.js github - * node scripts/find-module.js terminal - * node scripts/find-module.js tasks + * node scripts/find-module.js frameStore.js + * node scripts/find-module.js publishStaged + * + * The engine comes from --retrieval, else `project.retrieval.engine` in + * .frame/config.json, else the shipped default (structure-retrieval.js). + * `legacy` is the pre-STR-03 behavior, kept as the rollback path. + * + * Exit codes: 0 answered or no match · 1 no usable map (`unavailable`) · + * 2 usage error. */ const fs = require('fs'); const path = require('path'); -const { execSync, spawnSync } = require('child_process'); // Ownership, freshness and generation status: the shared read contract. const structureRead = require('./structure-read'); +// One retrieval engine for this CLI and the search hook (STR-03). +const retrieval = require('./structure-retrieval'); + +const LOOKUP_SCHEMA = 'frame.lookup/1'; +const DEFAULT_LIMIT = 8; /** * Which project this run is about. `__dirname/..` was wrong for the shipped @@ -34,242 +48,243 @@ function resolveProjectRoot() { return process.cwd(); } -function repairCommand(root) { - const updater = path.join(__dirname, 'update-structure.js'); - const rel = path.relative(root, updater).split(path.sep).join('/'); - return `node ${rel && !rel.startsWith('..') ? rel : '.frame/bin/update-structure.js'} --full`; -} - const ROOT_DIR = resolveProjectRoot(); -// STR-02c: the live working-tree view first, the tracked (committed) map otherwise. -const STRUCTURE_FILE = structureRead.resolveReadPath(ROOT_DIR); -function loadStructure() { - try { - return JSON.parse(fs.readFileSync(STRUCTURE_FILE, 'utf-8')); - } catch (e) { - console.error('Error: Could not read STRUCTURE.json'); - process.exit(1); - } +function scriptRel(name) { + const rel = path.relative(ROOT_DIR, path.join(__dirname, name)).split(path.sep).join('/'); + return rel && !rel.startsWith('..') ? rel : `.frame/bin/${name}`; } -/** - * Load the curated concept map (synonyms live here). Sits next to this - * script both in Frame's repo (scripts/) and user projects (.frame/bin/). - */ -function loadIntentMap() { - try { - const map = JSON.parse(fs.readFileSync(path.join(__dirname, 'intent-map.json'), 'utf-8')); - delete map._comment; - return map; - } catch (e) { - return {}; - } +function repairCommand() { + return `node ${scriptRel('update-structure.js')} --full`; } -/** - * One-line warning when STRUCTURE.json is stale. The date comparison is a - * cheap pre-filter; when it fires, update-structure.js --check confirms - * against actual content — merges/reverts dated after lastUpdated that - * changed no module content must not produce a banner (an idempotent regen - * couldn't clear it). - */ -function stalenessBanner(structure) { - try { - const lastSrcCommit = execSync('git log -1 --format=%cs -- src', { - cwd: ROOT_DIR, - encoding: 'utf-8', - stdio: ['pipe', 'pipe', 'ignore'] - }).trim(); - if (!(lastSrcCommit && structure.lastUpdated && structure.lastUpdated < lastSrcCommit)) { - return null; - } +/* --------------------------------- input -------------------------------- */ - const checker = path.join(__dirname, 'update-structure.js'); - if (fs.existsSync(checker)) { - const r = spawnSync('node', [checker, '--check'], { - cwd: ROOT_DIR, - env: { ...process.env, FRAME_PROJECT_ROOT: ROOT_DIR }, - stdio: 'ignore', - timeout: 30000 - }); - if (r.status === 0) return null; // dates disagree, content in sync - if (r.status === 1) { - return '⚠ STRUCTURE.json content is out of date vs src — run: npm run structure'; - } - } - return `⚠ STRUCTURE.json (${structure.lastUpdated}) is older than the last src commit (${lastSrcCommit}) — run: npm run structure`; - } catch (e) { - // Not a git repo or git unavailable — no banner +function parseArgs(argv) { + const out = { json: false, list: false, limit: DEFAULT_LIMIT, engine: null, words: [], error: null }; + for (let i = 0; i < argv.length; i++) { + const a = argv[i]; + if (a === '--json') out.json = true; + else if (a === '--list') out.list = true; + else if (a === '--limit' || a.startsWith('--limit=')) { + const value = a.includes('=') ? a.slice(8) : argv[++i]; + const n = Number(value); + if (!Number.isInteger(n) || n < 1) out.error = `--limit takes a positive whole number, got ${JSON.stringify(value)}`; + else out.limit = Math.min(n, retrieval.LIMITS.cliFiles); + } else if (a === '--retrieval' || a.startsWith('--retrieval=')) { + out.engine = a.includes('=') ? a.slice(12) : argv[++i]; + } else out.words.push(a); } - return null; + return out; } -/** - * Search intentIndex for matching features - */ -function searchIntentIndex(structure, keyword) { - const index = structure.intentIndex; - if (!index) { - console.error('No intentIndex found in STRUCTURE.json. Run: npm run structure'); - process.exit(1); - } - - const kw = keyword.toLowerCase(); - const results = []; - - // 1. Exact match in intentIndex keys - for (const [feature, modules] of Object.entries(index)) { - if (feature === kw) { - results.push({ feature, modules, matchType: 'exact' }); - } +/** --retrieval, then project.retrieval.engine, then the shipped default. */ +function resolveEngine(flag) { + const valid = (e) => retrieval.ENGINES.includes(e); + if (flag) { + if (valid(flag)) return { engine: flag }; + return { engine: retrieval.DEFAULT_ENGINE, note: `unknown engine ${JSON.stringify(flag)} — using ${retrieval.DEFAULT_ENGINE}` }; } + let configured; + try { + const config = JSON.parse(fs.readFileSync(path.join(ROOT_DIR, '.frame', 'config.json'), 'utf8')); + configured = config && config.project && config.project.retrieval && config.project.retrieval.engine; + } catch { /* no config: the default */ } + if (configured === undefined || configured === null) return { engine: retrieval.DEFAULT_ENGINE }; + if (valid(configured)) return { engine: configured }; + return { engine: retrieval.DEFAULT_ENGINE, note: `project.retrieval.engine ${JSON.stringify(configured)} is not an engine — using ${retrieval.DEFAULT_ENGINE}` }; +} - // 2. Synonym match from intent-map.json (e.g. "auth" → ai-tool) - if (results.length === 0) { - const intentMap = loadIntentMap(); - for (const [concept, entry] of Object.entries(intentMap)) { - const synonyms = (entry.synonyms || []).map(s => s.toLowerCase()); - if (synonyms.includes(kw) && index[concept]) { - results.push({ feature: `${concept} (synonym: "${keyword}")`, modules: index[concept], matchType: 'synonym' }); - } - } - } +/* ------------------------------ map and paths ----------------------------- */ - // 3. Partial match in intentIndex keys - if (results.length === 0) { - for (const [feature, modules] of Object.entries(index)) { - if (feature.includes(kw) || kw.includes(feature)) { - results.push({ feature, modules, matchType: 'partial' }); - } - } +function loadStructure() { + try { + return JSON.parse(fs.readFileSync(structureRead.resolveReadPath(ROOT_DIR), 'utf-8')); + } catch (e) { + return null; } +} - // 4. Search in module descriptions, exports, IPC channels - if (results.length === 0) { - const matchedModules = []; - for (const [key, mod] of Object.entries(structure.modules)) { - const searchable = [ - key, - mod.description || '', - ...(mod.exports || []), - ...(mod.ipc?.listens || []), - ...(mod.ipc?.emits || []) - ].join(' ').toLowerCase(); - - if (searchable.includes(kw)) { - matchedModules.push({ - module: key, - file: mod.file, - description: mod.description || '' - }); - } - } - if (matchedModules.length > 0) { - results.push({ feature: `search: "${keyword}"`, modules: matchedModules, matchType: 'deep' }); - } +let rootReal = null; +/** A candidate is shown as present only if it is a regular file inside the project. */ +function existsInProject(rel) { + if (typeof rel !== 'string' || !rel || path.isAbsolute(rel) || rel.split(/[\\/]/).includes('..')) return false; + try { + rootReal = rootReal || fs.realpathSync(ROOT_DIR); + const real = fs.realpathSync(path.join(ROOT_DIR, rel)); + if (real !== rootReal && !real.startsWith(rootReal + path.sep)) return false; + return fs.statSync(real).isFile(); + } catch { + return false; } +} - return results; +/** The published index when it is current, else one compiled from the map. */ +function v2Index() { + const loaded = retrieval.loadLookup(ROOT_DIR); + if (loaded.state === 'fresh') return loaded.index; + const compiled = retrieval.indexFromMap(ROOT_DIR); + return compiled.state === 'compiled' ? compiled.index : null; } +/* -------------------------------- freshness ------------------------------- */ + /** - * List all features in intentIndex + * How current the map is, from the lifecycle receipt only (STR-02). No Git, + * no rebuild check: a lookup stays a lookup. */ +function freshnessLines(descriptor) { + const why = descriptor.reasons.length ? ` (${descriptor.reasons.join(', ')})` : ''; + if (descriptor.freshness === 'fresh') return ['Map: fresh · working tree']; + if (descriptor.freshness === 'dirty') return [`⚠ Map: dirty${why} — recent changes are still being applied`]; + if (descriptor.freshness === 'stale') return [`⚠ Map: stale${why} — run: node ${scriptRel('structure-lifecycle.js')} --once`]; + return [`⚠ Map: unverified${why || ' (no lifecycle record)'} — run: node ${scriptRel('structure-lifecycle.js')} --once`]; +} + +/* --------------------------------- output -------------------------------- */ + function listFeatures(structure) { const index = structure.intentIndex; if (!index) { - console.error('No intentIndex found. Run: npm run structure'); + console.error(`No intentIndex found. Run: ${repairCommand()}`); process.exit(1); } - console.log('Available features:\n'); for (const [feature, modules] of Object.entries(index)) { - const files = modules.map(m => m.file).join(', '); + const files = modules.map((m) => m.file).join(', '); console.log(` ${feature.padEnd(20)} → ${files}`); } console.log(`\nTotal: ${Object.keys(index).length} features, ${Object.values(index).flat().length} modules`); } -/** - * Format and print results - */ -function printResults(structure, results, keyword) { - if (results.length === 0) { - console.log(`No modules found for "${keyword}"`); - console.log('Try: node scripts/find-module.js --list'); +/** The pre-STR-03 listing: one block per matched feature, with its IPC channels. */ +function printLegacy(structure, result, query) { + if (!result.groups.length) { + console.log(`No modules found for "${query}"`); + console.log(`Try: node ${scriptRel('find-module.js')} --list`); return; } - - for (const result of results) { - console.log(`Feature: ${result.feature}`); - for (const mod of result.modules) { + for (const group of result.groups) { + console.log(`Feature: ${group.feature}`); + for (const mod of group.modules) { const desc = mod.description ? ` — ${mod.description}` : ''; - const missing = fs.existsSync(path.join(ROOT_DIR, mod.file)) - ? '' - : ' ⚠ file missing on disk — run: npm run structure'; + const missing = existsInProject(mod.file) ? '' : ` ⚠ file missing on disk — run: ${repairCommand()}`; console.log(` ${mod.file.padEnd(42)}${desc}${missing}`); } - - // Show related IPC channels - const ipcChannels = []; - for (const mod of result.modules) { - const modInfo = structure.modules[mod.module]; - if (modInfo?.ipc) { - ipcChannels.push(...(modInfo.ipc.listens || []), ...(modInfo.ipc.emits || [])); - } - } - if (ipcChannels.length > 0) { - console.log(` IPC: ${[...new Set(ipcChannels)].join(', ')}`); + const ipc = []; + for (const mod of group.modules) { + const info = (structure.modules || {})[mod.module]; + if (info && info.ipc) ipc.push(...(info.ipc.listens || []), ...(info.ipc.emits || [])); } + if (ipc.length) console.log(` IPC: ${[...new Set(ipc)].join(', ')}`); console.log(''); } } -// Main -const args = process.argv.slice(2); - -if (args.length === 0) { - console.log('Usage: node scripts/find-module.js '); - console.log(' node scripts/find-module.js --list'); - process.exit(0); +function printV2(result, query) { + if (result.status === 'no-match') { + console.log(`No modules found for "${query}"`); + console.log(`Try: node ${scriptRel('find-module.js')} --list`); + return; + } + const head = result.status === 'resolved' ? 'Files for' : `Candidates for`; + console.log(`${head} "${query}"${result.status === 'ambiguous' ? ' (several match equally — pick by path)' : ''}:`); + for (const c of result.candidates) { + const desc = c.description ? ` — ${c.description}` : ''; + const missing = c.missing ? ` ⚠ missing on disk — run: ${repairCommand()}` : ''; + console.log(` ${c.path.padEnd(42)}${desc} [${c.evidence}]${missing}`); + } + if (result.truncated) console.log(` … more — raise --limit (up to ${retrieval.LIMITS.cliFiles})`); + console.log(''); } -const structure = loadStructure(); +/* ---------------------------------- main --------------------------------- */ -// How current the map is. With a lifecycle receipt the answer is known and -// replaces the date heuristic; without one (a clone, Frame not running here) -// the old banner still applies. -const RECEIPT_MISMATCH = ['artifact-changed', 'not-working-tree']; -const descriptor = structureRead.readDescriptor(ROOT_DIR); -const known = descriptor.freshness !== 'unknown' || descriptor.reasons.some((r) => RECEIPT_MISMATCH.includes(r)); -if (known) { - const why = descriptor.reasons.length ? ` (${descriptor.reasons.join(', ')})` : ''; - if (descriptor.freshness === 'fresh') { - console.log('Map: fresh · working tree\n'); - } else if (descriptor.freshness === 'dirty') { - console.log(`⚠ Map: dirty${why} — recent changes are still being applied\n`); - } else if (descriptor.freshness === 'stale') { - console.log(`⚠ Map: stale${why} — run: node ${path.relative(ROOT_DIR, path.join(__dirname, 'structure-lifecycle.js')).split(path.sep).join('/')} --once\n`); +function main() { + const args = parseArgs(process.argv.slice(2)); + const query = args.words.join(' '); + + if (args.error) { + console.error(`Error: ${args.error}`); + process.exitCode = 2; + return; + } + if (!args.list && !query) { + console.log('Usage: node scripts/find-module.js [--json] [--limit N] [--retrieval=legacy|v2]'); + console.log(' node scripts/find-module.js --list'); + return; + } + + const { engine, note } = resolveEngine(args.engine); + if (note) console.error(`⚠ ${note}`); + const descriptor = structureRead.readDescriptor(ROOT_DIR); + const structure = loadStructure(); + + if (!structure || !structure.modules) { + if (args.json) { + process.stdout.write(`${JSON.stringify({ schema: LOOKUP_SCHEMA, status: 'unavailable', engine, query, reason: structure ? 'invalid-map' : 'no-map', freshness: descriptor.freshness, candidates: [], truncated: false })}\n`); + } else { + console.error(`Error: Could not read STRUCTURE.json — run: ${repairCommand()}`); + } + process.exitCode = 1; + return; + } + + if (args.list) { + listFeatures(structure); + return; + } + + let result; + if (engine === 'legacy') { + result = retrieval.legacyRetrieve(structure, readCuration(), { mode: 'cli', query }); + if (result.status === 'unavailable') result = { ...result, status: 'no-match' }; + result.candidates = result.candidates.slice(0, args.json ? args.limit : result.candidates.length) + .map((c) => (existsInProject(c.path) ? c : { ...c, missing: true })); } else { - console.log(`⚠ Map: unverified${why} — STRUCTURE.json changed since it was last checked\n`); + const index = v2Index(); + result = index + ? retrieval.retrieve(index, query, { mode: 'cli', limit: args.limit, exists: existsInProject }) + : { status: 'no-match', candidates: [], truncated: false }; } -} else { - const banner = stalenessBanner(structure); - if (banner) { - console.log(banner + '\n'); + + if (args.json) { + const envelope = { + schema: LOOKUP_SCHEMA, + status: result.status, + engine, + query, + freshness: descriptor.freshness, + freshnessReasons: descriptor.reasons, + candidates: result.candidates.slice(0, args.limit).map((c) => { + const out = { path: c.path, evidence: c.evidence, tier: c.tier, description: c.description || '' }; + if (c.missing) out.missing = true; + return out; + }), + truncated: Boolean(result.truncated) || result.candidates.length > args.limit + }; + process.stdout.write(`${JSON.stringify(envelope)}\n`); + return; } -} -const generationWarnings = structureRead.generationNotes(ROOT_DIR, structure); -if (generationWarnings.length > 0) { - for (const note of generationWarnings) console.log(`⚠ STRUCTURE.json ${note}`); - console.log(` Rebuild: ${repairCommand(ROOT_DIR)}\n`); + + for (const line of freshnessLines(descriptor)) console.log(line); + console.log(''); + const generationWarnings = structureRead.generationNotes(ROOT_DIR, structure); + if (generationWarnings.length > 0) { + for (const n of generationWarnings) console.log(`⚠ STRUCTURE.json ${n}`); + console.log(` Rebuild: ${repairCommand()}\n`); + } + if (engine === 'legacy') printLegacy(structure, result, query); + else printV2(result, query); } -if (args[0] === '--list') { - listFeatures(structure); -} else { - const keyword = args.join(' '); - const results = searchIntentIndex(structure, keyword); - printResults(structure, results, keyword); +function readCuration() { + try { + return JSON.parse(fs.readFileSync(retrieval.curationPath(__dirname), 'utf8')); + } catch { + return {}; + } } + +main(); diff --git a/scripts/structure-retrieval.js b/scripts/structure-retrieval.js index 6aff9d63..4db01f09 100644 --- a/scripts/structure-retrieval.js +++ b/scripts/structure-retrieval.js @@ -31,6 +31,11 @@ const INDEX_VERSION = 1; const ALGORITHM = 'str03-v2.1'; +// `legacy` is the pre-STR-03 behavior, kept selectable as the rollback path; +// the default follows the benchmark gates (scripts/eval/README.md). +const ENGINES = Object.freeze(['legacy', 'v2']); +const DEFAULT_ENGINE = 'legacy'; + const LIMITS = Object.freeze({ descriptionChars: 160, postingsPerTerm: 64, @@ -614,6 +619,8 @@ module.exports = { legacyRetrieve, fold, splitWords, + ENGINES, + DEFAULT_ENGINE, TIER, HOOK_MAX_TIER, LIMITS, diff --git a/test/findModule.test.js b/test/findModule.test.js new file mode 100644 index 00000000..a53a7f0b --- /dev/null +++ b/test/findModule.test.js @@ -0,0 +1,167 @@ +/** + * find-module CLI (STR-03 T04): legacy-compatible output, the bounded + * `--json` envelope, `unavailable` vs `no-match`, missing files, engine + * selection, and lookups that never run Git. + */ + +const { test } = require('node:test'); +const assert = require('node:assert/strict'); +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const { spawnSync } = require('child_process'); + +const FIND = path.join(__dirname, '..', 'scripts', 'find-module.js'); + +const STRUCTURE = { + generation: { revision: 'r1' }, + modules: { + 'main/githubManager': { + file: 'src/main/githubManager.js', description: 'GitHub Manager Module', exports: [], depends: [], + functions: { checkGhAuth: { line: 1 } }, ipc: { listens: ['LOAD_GITHUB_ISSUES'], emits: [] } + }, + 'renderer/githubPanel': { file: 'src/renderer/githubPanel.js', description: 'GitHub Panel', exports: [], depends: [], functions: {} }, + 'renderer/github/rowModels': { file: 'src/renderer/github/rowModels.js', description: 'Row models', exports: [], depends: [], functions: { issueBranchName: {} } }, + 'main/a': { file: 'src/main/uiZoom.js', description: 'Main zoom', exports: [], depends: [], functions: {} }, + 'shared/a': { file: 'src/shared/uiZoom.js', description: 'Shared zoom', exports: [], depends: [], functions: {} } + }, + intentIndex: { + github: [ + { module: 'main/githubManager', file: 'src/main/githubManager.js', description: 'GitHub Manager Module' }, + { module: 'renderer/githubPanel', file: 'src/renderer/githubPanel.js', description: 'GitHub Panel' } + ] + } +}; + +function project(t, { structure = STRUCTURE, engine } = {}) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'frame-find-')); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + fs.mkdirSync(path.join(dir, '.frame'), { recursive: true }); + if (structure) fs.writeFileSync(path.join(dir, '.frame', 'STRUCTURE.json'), JSON.stringify(structure)); + for (const m of Object.values((structure && structure.modules) || {})) { + fs.mkdirSync(path.dirname(path.join(dir, m.file)), { recursive: true }); + fs.writeFileSync(path.join(dir, m.file), '// x\n'); + } + if (engine) fs.writeFileSync(path.join(dir, '.frame', 'config.json'), JSON.stringify({ project: { retrieval: { engine } } })); + return dir; +} + +function find(dir, args, env = {}) { + return spawnSync('node', [FIND, ...args], { encoding: 'utf8', env: { ...process.env, FRAME_PROJECT_ROOT: dir, ...env } }); +} + +const envelope = (r) => JSON.parse(r.stdout); + +test('the legacy engine keeps the feature listing, IPC line and missing-file marker', (t) => { + const dir = project(t); + fs.rmSync(path.join(dir, 'src/renderer/githubPanel.js')); + const r = find(dir, ['github', '--retrieval=legacy']); + assert.equal(r.status, 0); + assert.match(r.stdout, /Feature: github\n {2}src\/main\/githubManager\.js +— GitHub Manager Module\n/); + assert.match(r.stdout, /src\/renderer\/githubPanel\.js .*⚠ file missing on disk/); + assert.match(r.stdout, /IPC: LOAD_GITHUB_ISSUES/); + assert.match(find(dir, ['--list']).stdout, /github +→ src\/main\/githubManager\.js, src\/renderer\/githubPanel\.js/); +}); + +test('v2 finds files outside every concept and names the evidence', (t) => { + const dir = project(t); + const r = find(dir, ['issueBranchName', '--retrieval=v2']); + assert.match(r.stdout, /Files for "issueBranchName":\n {2}src\/renderer\/github\/rowModels\.js .*\[symbol\]/); + const amb = find(dir, ['uiZoom', '--retrieval=v2']); + assert.match(amb.stdout, /Candidates for "uiZoom" \(several match equally — pick by path\)/); + assert.match(find(dir, ['kubernetes', '--retrieval=v2']).stdout, /No modules found for "kubernetes"/); +}); + +test('--json is one bounded envelope; no-match exits 0, a missing or corrupt map is unavailable with exit 1', (t) => { + const dir = project(t); + const ok = find(dir, ['github', '--json', '--retrieval=v2']); + assert.equal(ok.status, 0); + const env = envelope(ok); + assert.equal(env.schema, 'frame.lookup/1'); + assert.equal(env.status, 'resolved'); + assert.equal(env.engine, 'v2'); + assert.equal(env.freshness, 'unknown'); + assert.deepEqual(env.candidates.map((c) => [c.path, c.evidence]), [['src/main/githubManager.js', 'concept'], ['src/renderer/githubPanel.js', 'concept'], ['src/renderer/github/rowModels.js', 'path word']]); + assert.equal(ok.stdout.trim().split('\n').length, 1, 'nothing else on stdout'); + + const none = find(dir, ['kubernetes', '--json', '--retrieval=v2']); + assert.equal(none.status, 0); + assert.equal(envelope(none).status, 'no-match'); + + const empty = project(t, { structure: null }); + const missing = find(empty, ['github', '--json']); + assert.equal(missing.status, 1); + assert.deepEqual([envelope(missing).status, envelope(missing).reason], ['unavailable', 'no-map']); + fs.writeFileSync(path.join(empty, '.frame', 'STRUCTURE.json'), '{ not json'); + assert.equal(find(empty, ['github']).status, 1); + assert.match(find(empty, ['github']).stderr, /Could not read STRUCTURE\.json/); +}); + +test('--limit bounds the candidates (max 20) and reports truncation', (t) => { + const modules = {}; + const group = []; + for (let i = 0; i < 30; i++) { + modules[`m${i}`] = { file: `src/widget${i}.js`, description: `Widget ${i}`, exports: [], depends: [], functions: {} }; + group.push({ module: `m${i}`, file: `src/widget${i}.js` }); + } + const dir = project(t, { structure: { modules, intentIndex: { widgets: group } } }); + const three = envelope(find(dir, ['widgets', '--json', '--limit', '3', '--retrieval=v2'])); + assert.equal(three.candidates.length, 3); + assert.equal(three.truncated, true); + assert.equal(envelope(find(dir, ['widgets', '--json', '--limit=50', '--retrieval=v2'])).candidates.length, 20); + assert.equal(envelope(find(dir, ['widgets', '--json', '--retrieval=legacy'])).candidates.length, 8, 'the default limit applies to legacy JSON too'); + assert.equal(find(dir, ['widgets', '--limit', 'x']).status, 2); +}); + +test('missing files are flagged, never presented as present; paths outside the project never count', (t) => { + const dir = project(t); + fs.rmSync(path.join(dir, 'src/renderer/github/rowModels.js')); + const env = envelope(find(dir, ['issueBranchName', '--json', '--retrieval=v2'])); + assert.deepEqual(env.candidates, [{ path: 'src/renderer/github/rowModels.js', evidence: 'symbol', tier: 5, description: 'Row models', missing: true }]); + + // a symlink that escapes the project is not "on disk" for it + const outside = fs.mkdtempSync(path.join(os.tmpdir(), 'frame-outside-')); + t.after(() => fs.rmSync(outside, { recursive: true, force: true })); + fs.writeFileSync(path.join(outside, 'rowModels.js'), '// outside\n'); + fs.symlinkSync(path.join(outside, 'rowModels.js'), path.join(dir, 'src/renderer/github/rowModels.js')); + assert.equal(envelope(find(dir, ['issueBranchName', '--json', '--retrieval=v2'])).candidates[0].missing, true); +}); + +test('the engine comes from the flag, then project.retrieval.engine, then the default; bad values fall back with a note', (t) => { + const dir = project(t, { engine: 'v2' }); + assert.equal(envelope(find(dir, ['github', '--json'])).engine, 'v2'); + assert.equal(envelope(find(dir, ['github', '--json', '--retrieval=legacy'])).engine, 'legacy'); + const bad = find(dir, ['github', '--json', '--retrieval=fancy']); + assert.equal(envelope(bad).engine, require('../scripts/structure-retrieval').DEFAULT_ENGINE); + assert.match(bad.stderr, /unknown engine "fancy"/); + const badConfig = project(t, { engine: 'nope' }); + assert.match(find(badConfig, ['github', '--json']).stderr, /project\.retrieval\.engine "nope" is not an engine/); +}); + +test('v2 uses a fresh published index and falls back to the map when it is stale', (t) => { + const dir = project(t); + const retrieval = require('../scripts/structure-retrieval'); + assert.equal(retrieval.publishLookup(dir).status, 'published'); + // poison the published index: a fresh one would be used as-is + const file = retrieval.lookupPath(dir); + const index = JSON.parse(fs.readFileSync(file, 'utf8')); + index.terms['5:onlyinindex'] = [0]; + fs.writeFileSync(file, JSON.stringify(index)); + assert.equal(envelope(find(dir, ['onlyInIndex', '--json', '--retrieval=v2'])).status, 'resolved'); + // the map changes: the index is stale and the map answers + const map = path.join(dir, '.frame', 'STRUCTURE.json'); + fs.writeFileSync(map, fs.readFileSync(map, 'utf8') + ' '); + assert.equal(envelope(find(dir, ['onlyInIndex', '--json', '--retrieval=v2'])).status, 'no-match'); +}); + +test('a lookup never runs Git, even when the map predates the last commit', (t) => { + const dir = project(t); + const bin = fs.mkdtempSync(path.join(os.tmpdir(), 'frame-no-git-')); + t.after(() => fs.rmSync(bin, { recursive: true, force: true })); + const marker = path.join(bin, 'called'); + fs.writeFileSync(path.join(bin, 'git'), `#!/bin/sh\ntouch ${JSON.stringify(marker)}\nexit 1\n`, { mode: 0o755 }); + const r = find(dir, ['github'], { PATH: `${bin}${path.delimiter}${process.env.PATH}` }); + assert.equal(r.status, 0); + assert.equal(fs.existsSync(marker), false); + assert.match(r.stdout, /^⚠ Map: unverified \(no-working-view\) — run: node .*structure-lifecycle\.js --once/); +}); diff --git a/test/scriptsProjectRoot.test.js b/test/scriptsProjectRoot.test.js index 4081f278..5f17c119 100644 --- a/test/scriptsProjectRoot.test.js +++ b/test/scriptsProjectRoot.test.js @@ -496,9 +496,10 @@ test('readers report the lifecycle freshness and skip the date heuristic once it fs.writeFileSync(mapFile, fs.readFileSync(mapFile, 'utf8').replace('Widget maker', 'Edited by hand')); assert.match(find(), /^⚠ Map: stale \(lease-expired\)/); - // Without a working view there is nothing live to vouch for: unknown, not an alarm. + // Without a working view there is nothing live to vouch for: unknown, said + // as such (STR-03 D13: no Git date heuristic in a lookup), not an alarm. fs.rmSync(path.join(dir, '.frame', 'runtime', 'structure', 'working.json')); - assert.ok(!/^⚠ Map:|^Map:/.test(find())); + assert.match(find(), /^⚠ Map: unverified \(no-working-view\)/); assert.deepEqual(findings().filter((f) => f.startsWith('structure-freshness')), []); } finally { fs.rmSync(dir, { recursive: true, force: true }); From 96abb7fb413bb5924020a403f86c01ce8f7d5ee2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kaan=20=C3=96zhan?= Date: Thu, 1 Oct 2026 16:24:59 +0300 Subject: [PATCH 6/9] feat(structure): the search hook on the shared engine with bounded, revision-aware hints (STR-03 T05) --- .frame/STRUCTURE.json | 170 +++++++++--- .../str-03-local-file-retrieval/outcome.md | 24 ++ scripts/module-hint.js | 259 ++++++++++++------ src/shared/activityEvents.js | 9 +- test/activityEvents.test.js | 5 +- test/module-hint.test.js | 180 +++++++++++- 6 files changed, 517 insertions(+), 130 deletions(-) diff --git a/.frame/STRUCTURE.json b/.frame/STRUCTURE.json index ec28066d..e626d48d 100644 --- a/.frame/STRUCTURE.json +++ b/.frame/STRUCTURE.json @@ -17171,53 +17171,54 @@ "fs", "path", "structure-read", + "structure-retrieval", "activity-log", "toolVocabulary" ], "functions": { "toPosix": { - "line": 52, + "line": 56, "params": [ "p" ] }, "readStdin": { - "line": 54 + "line": 58 }, "readJson": { - "line": 58, + "line": 62, "params": [ "p" ] }, "resolveRoot": { - "line": 64, + "line": 68, "params": [ "hookCwd" ] }, "finderCliPath": { - "line": 81, + "line": 91, "params": [ "root" ] }, "stateFile": { - "line": 89, + "line": 99, "params": [ "root", "sessionId" ] }, "loadState": { - "line": 94, + "line": 104, "params": [ "root", "sessionId" ] }, "saveState": { - "line": 99, + "line": 112, "params": [ "root", "sessionId", @@ -17225,23 +17226,23 @@ ] }, "cleanupState": { - "line": 107, + "line": 120, "params": [ "root" ] }, "setHookCli": { - "line": 153, + "line": 166, "params": [ "payload" ], "purpose": "Called once, from the entry point, with the parsed payload." }, "hookHost": { - "line": 157 + "line": 170 }, "note": { - "line": 161, + "line": 174, "params": [ "root", "ev", @@ -17249,7 +17250,7 @@ ] }, "quiet": { - "line": 177, + "line": 190, "params": [ "root", "reason" @@ -17257,74 +17258,146 @@ "purpose": "Record a quiet path and return, so call sites stay single-expression." }, "emit": { - "line": 181, + "line": 194, "params": [ "context" ] }, "searchSegments": { - "line": 203, + "line": 216, "params": [ "cmd" ] }, "extractPattern": { - "line": 222, + "line": 235, "params": [ "seg" ], "purpose": "Pull the pattern operand out of one search segment (already known to start" }, "candidates": { - "line": 238, + "line": 251, "params": [ "raw" ], "purpose": "A raw pattern (regex, glob, or plain word) → the concept words worth" }, + "patternFrom": { + "line": 266, + "params": [ + "toolName", + "input" + ], + "purpose": "The raw pattern of a search call: null when the call is not a search at" + }, "keywordsFrom": { - "line": 249, + "line": 286, "params": [ "toolName", "input" ] }, "loadIntentMap": { - "line": 286 + "line": 301 }, - "lookup": { - "line": 293, + "render": { + "line": 308, "params": [ + "root", "structure", + "hit", "keyword" ] }, - "render": { - "line": 315, + "engineFor": { + "line": 334, + "params": [ + "root" + ], + "purpose": "project.retrieval.engine, else the shipped default; anything else is the default." + }, + "existsInProject": { + "line": 342, "params": [ "root", - "structure", - "hit", - "keyword" + "rel" + ], + "purpose": "A hinted file must be a regular file inside the project right now." + }, + "legacyMode": { + "line": 354, + "params": [ + "root", + "input", + "descriptor" + ] + }, + "loadIndex": { + "line": 375, + "params": [ + "root" + ], + "purpose": "The index to answer from: a current lookup.json, else the map compiled in memory." + }, + "fingerprint": { + "line": 385, + "params": [ + "paths" + ] + }, + "shownQuery": { + "line": 392, + "params": [ + "pattern" + ], + "purpose": "One line of the query, safe to show and to paste as a shell argument." + }, + "shellQuote": { + "line": 396, + "params": [ + "text" + ] + }, + "renderV2": { + "line": 400, + "params": [ + "root", + "result", + "pattern", + "descriptor" + ] + }, + "v2Mode": { + "line": 430, + "params": [ + "root", + "input", + "descriptor" ] }, "searchMode": { - "line": 340, + "line": 462, "params": [ "input" ] } }, - "sizeBytes": 15937, + "sizeBytes": 21053, "extraction": { "status": "parsed" }, "provenance": { "purposes": { "candidates": "a9ed75e60ec11d73", + "engineFor": "dd31c17caca3abfa", + "existsInProject": "5dddbd190b145640", "extractPattern": "2c7cae15c16d94d2", + "loadIndex": "2f9196b0793b2ccf", + "patternFrom": "79c2136a033352ff", "quiet": "3e8f4a1504407442", - "setHookCli": "c07ecdee8595a6cf" + "setHookCli": "c07ecdee8595a6cf", + "shownQuery": "97f764f95a46a385" } } }, @@ -20028,32 +20101,32 @@ "depends": [], "functions": { "isRegistered": { - "line": 420, + "line": 427, "params": [ "name" ] }, "kindOf": { - "line": 424, + "line": 431, "params": [ "name" ] }, "isSuppression": { - "line": 428, + "line": 435, "params": [ "name" ] }, "fieldPasses": { - "line": 432, + "line": 439, "params": [ "spec", "value" ] }, "validateEvent": { - "line": 455, + "line": 462, "params": [ "name", "fields" @@ -20061,7 +20134,7 @@ "purpose": "Validate an (event, fields) pair against the registry." }, "formatLabel": { - "line": 472, + "line": 479, "params": [ "name", "fields" @@ -20069,7 +20142,7 @@ "purpose": "The one-line sentence the panel's default view shows, or null when the" }, "buildRecord": { - "line": 493, + "line": 500, "params": [ "name", "fields" @@ -20077,7 +20150,7 @@ "purpose": "Build the record the append contract writes: the validated fields plus the" } }, - "sizeBytes": 22204, + "sizeBytes": 22748, "extraction": { "status": "parsed" }, @@ -21566,7 +21639,7 @@ "src/shared/activityEvents" ], "functions": {}, - "sizeBytes": 11100, + "sizeBytes": 11171, "extraction": { "status": "parsed" }, @@ -23289,7 +23362,8 @@ "os", "path", "child_process", - "crypto" + "crypto", + "scripts/structure-retrieval" ], "functions": { "runHook": { @@ -23306,16 +23380,28 @@ ] }, "withReceipt": { - "line": 223, + "line": 239, "params": [ "root", "{ dirty = []", "epoch = { requested", "applied" ] + }, + "v2Project": { + "line": 307, + "params": [ + "structure = V2_STRUCTURE" + ] + }, + "runCodex": { + "line": 317, + "params": [ + "input" + ] } }, - "sizeBytes": 12523, + "sizeBytes": 21693, "extraction": { "status": "parsed" }, @@ -26490,6 +26576,6 @@ "truncated": false, "samples": [] }, - "revision": "2eaaedc700d2f754c9b5e1c59391d16d9c02580c9c1c2a13a7ef2e5a4bcef624" + "revision": "df418cd0388ef0973a219130be810456a7fc1a28710f02aa9e2c4562af463d03" } } diff --git a/.frame/specs/str-03-local-file-retrieval/outcome.md b/.frame/specs/str-03-local-file-retrieval/outcome.md index a1c57301..e9909f2a 100644 --- a/.frame/specs/str-03-local-file-retrieval/outcome.md +++ b/.frame/specs/str-03-local-file-retrieval/outcome.md @@ -73,3 +73,27 @@ Candidates are present only as regular files inside the project (a symlink escap _Captured: 2026-10-01 · 4 file change(s)_ --- + +## T05 — Search hook on the shared engine + +`scripts/module-hint.js` resolves the engine from `project.retrieval.engine` (else `DEFAULT_ENGINE`). Legacy keeps today's behavior and output through `legacyRetrieve`. + +In v2 the hook: +- Takes the raw pattern (Grep/Glob, `find -name|-path|-wholename`, the first parsable grep/rg/ag/ack segment). +- Stays quiet when the map is dirty or its inventory coverage is not complete. +- Reads a fresh `lookup.json` (≤ 2 MiB). Otherwise it compiles the read view in memory when that is ≤ 2 MiB, else quiet with `index-oversize`. +- Emits only the leading tier-1–6 group. Files must be regular and inside the project. +- Says "points to these files" for a fresh map and "has candidates … — map not verified recently ()" otherwise. +- Shell-quotes the query in the suggested command. +- Fits at most 8 files in 1,800 characters: descriptions are clipped first, the cut falls on a whole candidate, and a "… more" line is reserved. +- Dedups by revision plus answer fingerprint (at most 256 per session). Without a session id nothing is persisted. + +New quiet reasons: `map-incomplete`, `index-oversize`, `ambiguous-weak`, with texts in `src/shared/activityEvents.js`. Deviations: +- The reasons also had to join `HINT_REASONS`, the enum `hint.quiet` validates against. `test/activityEvents.test.js` (not in the plan's file list) pins that count, so it was updated from 12 to 15. +- The import test became a transitive closure walk (comments stripped), allowing exactly `activity-log`, `redact`, `structure-read`, `structure-retrieval` and `toolVocabulary`, and no `child_process`/network built-ins. + +Files touched: `scripts/module-hint.js`, `src/shared/activityEvents.js`, `test/module-hint.test.js`, `test/activityEvents.test.js`. + +_Captured: 2026-10-01 · 4 file change(s)_ + +--- diff --git a/scripts/module-hint.js b/scripts/module-hint.js index e85dd850..12ac21b0 100644 --- a/scripts/module-hint.js +++ b/scripts/module-hint.js @@ -20,19 +20,22 @@ * Hard contract (same as scripts/spec-hint.js): * - NEVER block, NEVER break: any failure → exit 0, empty output. The host * is a tool call; a hook error must never surface as a tool error. - * - Read-only: consumes STRUCTURE.json as-is, never rebuilds, never runs - * git. `find-module.js`'s staleness banner deliberately does NOT apply - * here — it shells out to git and can spawn update-structure, which - * would blow the per-call budget many times over. + * - Read-only: never rebuilds the map, never runs Git, never touches the + * network. The only writes are its own session-dedup state and the + * activity record. + * - Only read-only helpers: `structure-read` (ownership, freshness), + * `structure-retrieval` (the shared engine, also used by find-module), + * `activity-log` and `toolVocabulary`. Nothing that builds, writes the + * map or spawns a process — test/module-hint.test.js checks the import + * closure. * - Fast bail: this fires on every Bash call, the most common tool. A - * command that is not a search returns before STRUCTURE.json is opened. - * - Session dedup: one injection per matched concept per session; state in + * command that is not a search returns before any file is opened. + * - Bounded (STR-03): reads `lookup.json` (≤ 2 MiB) or, without a current + * one, compiles the map in memory only if it is ≤ 2 MiB; at most 8 files + * and 1,800 characters of context. + * - Session dedup: one delivery per answer per map revision; state in * .frame/runtime/module-hint/.json, stale files cleaned up - * after 7 days. - * - No imports from siblings: same rule spec-hint.js states — the hook must - * not pull in builder code paths that could rebuild or slow down. The - * intentIndex lookup below is a deliberate, trimmed copy of - * find-module.js's, not a require of it. + * after 7 days. Without a session id nothing is remembered. * * Dependency-free plain node; ships to user projects' .frame/bin/. */ @@ -45,7 +48,8 @@ const path = require('path'); const STATE_DIR_REL = path.join('.frame', 'runtime', 'module-hint'); const STATE_TTL_MS = 7 * 24 * 60 * 60 * 1000; const MAX_MODULES = 8; // output ceiling: a hint, not a file listing -const MAX_CANDIDATES = 3; // how many words from one search we bother to try +const MAX_CANDIDATES = 3; // legacy: how many words from one search we bother to try +const MAX_CONTEXT_CHARS = 1800; // below Claude Code's ~2,000-character inline ceiling // ─── tiny utils ─────────────────────────────────────────── @@ -78,6 +82,12 @@ try { structureRead = require('./structure-read'); } catch { /* older tooling: stay quiet */ } +// The shared retrieval engine (STR-03). Same guard: no engine, no hint. +let retrieval = null; +try { + retrieval = require('./structure-retrieval'); +} catch { /* older tooling: stay quiet */ } + function finderCliPath(root) { const local = path.join(__dirname, 'find-module.js'); if (fs.existsSync(local)) return toPosix(path.relative(root, local)) || 'find-module.js'; @@ -92,8 +102,11 @@ function stateFile(root, sessionId) { } function loadState(root, sessionId) { - const st = readJson(stateFile(root, sessionId)); - return (st && Array.isArray(st.concepts)) ? st : { concepts: [] }; + const st = readJson(stateFile(root, sessionId)) || {}; + return { + concepts: Array.isArray(st.concepts) ? st.concepts : [], + delivered: Array.isArray(st.delivered) ? st.delivered : [] + }; } function saveState(root, sessionId, state) { @@ -226,7 +239,7 @@ function extractPattern(seg) { if (m) return m[3]; const bare = seg.match(/^\s*(?:grep|rg|ag|ack)\s+((?:-{1,2}[^\s'"]+\s+)*)([^\s'"|;&]+)/); if (bare) return bare[2]; - const f = seg.match(/^\s*find\b.*?\s-i?name\s+(['"]?)([^'"\s]+)\1/); + const f = seg.match(/^\s*find\b.*?\s-i?(?:name|path|wholename)\s+(['"]?)([^'"\s]+)\1/); return f ? f[2] : ''; } @@ -246,11 +259,15 @@ function candidates(raw) { return out; } -function keywordsFrom(toolName, input) { +/** + * The raw pattern of a search call: null when the call is not a search at + * all (silent, unrecorded), '' when it is one with nothing usable in it. + */ +function patternFrom(toolName, input) { const role = vocab ? vocab.roleOf(toolName) : (toolName === 'Grep' || toolName === 'Glob' ? 'search' : (toolName === 'Bash' ? 'shell' : null)); if (role === 'search') { const pattern = vocab ? vocab.searchPattern(toolName, input) : (input.pattern || input.glob); - return candidates(pattern || input.path || ''); + return String(pattern || input.path || ''); } if (role === 'shell') { const cmd = input.command || ''; @@ -258,30 +275,28 @@ function keywordsFrom(toolName, input) { const segs = searchSegments(cmd); if (!segs.length) return null; // grep only *mentioned*, not run for (const seg of segs) { - const words = candidates(extractPattern(seg)); - if (words.length) return words; + const pattern = extractPattern(seg); + if (pattern) return pattern; } - return []; + return ''; } - return []; + return ''; } -// ─── intentIndex lookup ─────────────────────────────────── -// -// A trimmed copy of find-module.js's tiers — trimmed, not shared, per the -// no-sibling-imports rule above. +function keywordsFrom(toolName, input) { + const pattern = patternFrom(toolName, input); + return pattern === null ? null : candidates(pattern); +} + +// ─── legacy: intentIndex lookup ─────────────────────────── // -// Deliberately only the *curated* tiers: exact → synonym → partial, all of -// them keyed on intentIndex, which is hand-maintained in intent-map.json. -// find-module's fourth tier (scan every module's description, exports and -// IPC names) is left out on measured grounds: replayed over 1011 real search -// commands from this repo's transcripts, the curated tiers hit 297 times -// with usable answers (`claude-sessions`, `panel`, `implement`) while the -// deep tier hit 136 times almost entirely on noise — `kill` from a -// `pkill|killall` search, `process`, `focus`, and once on a half-typed -// `rchestrator`. A human running find-module asked for that fallback; an -// automatic hint that fires on it is worse than silence. The deep tier stays -// where it belongs, in the CLI. +// The pre-STR-03 behavior, selectable as the rollback path: only the +// curated tiers (exact → synonym → partial on intentIndex), first keyword +// that hits. find-module's deep tier stays out on measured grounds: +// replayed over 1011 real search commands from this repo's transcripts, the +// curated tiers hit 297 times with usable answers while the deep tier hit +// 136 times almost entirely on noise (`kill`, `process`, `focus`). The +// lookup itself lives in structure-retrieval.legacyRetrieve. function loadIntentMap() { const map = readJson(path.join(__dirname, 'intent-map.json')); @@ -290,28 +305,6 @@ function loadIntentMap() { return map; } -function lookup(structure, keyword) { - const index = structure.intentIndex; - if (!index) return null; - - for (const [feature, modules] of Object.entries(index)) { - if (feature === keyword) return { feature, modules }; - } - - for (const [concept, entry] of Object.entries(loadIntentMap())) { - const synonyms = (entry.synonyms || []).map((s) => String(s).toLowerCase()); - if (synonyms.includes(keyword) && index[concept]) { - return { feature: `${concept} (synonym: "${keyword}")`, modules: index[concept] }; - } - } - - for (const [feature, modules] of Object.entries(index)) { - if (feature.includes(keyword) || keyword.includes(feature)) return { feature, modules }; - } - - return null; -} - function render(root, structure, hit, keyword) { const lines = [`Feature: ${hit.feature}`]; for (const mod of hit.modules.slice(0, MAX_MODULES)) { @@ -337,40 +330,150 @@ function render(root, structure, hit, keyword) { // ─── main (never break) ─────────────────────────────────── -function searchMode(input) { - const root = resolveRoot(input.cwd); +/** project.retrieval.engine, else the shipped default; anything else is the default. */ +function engineFor(root) { + const config = readJson(path.join(root, '.frame', 'config.json')); + const configured = config && config.project && config.project.retrieval && config.project.retrieval.engine; + return retrieval.ENGINES.includes(configured) ? configured : retrieval.DEFAULT_ENGINE; +} +let rootReal = null; +/** A hinted file must be a regular file inside the project right now. */ +function existsInProject(root, rel) { + if (typeof rel !== 'string' || !rel || path.isAbsolute(rel) || rel.split(/[\\/]/).includes('..')) return false; + try { + rootReal = rootReal || fs.realpathSync(root); + const real = fs.realpathSync(path.join(root, rel)); + if (real !== rootReal && !real.startsWith(rootReal + path.sep)) return false; + return fs.statSync(real).isFile(); + } catch { + return false; + } +} + +function legacyMode(root, input, descriptor) { const words = keywordsFrom(input.tool_name, input.tool_input || {}); - if (words === null) return; // not a search: silent, unrecorded if (!words.length) return quiet(root, 'no-words'); - - if (!structureRead) return quiet(root, 'no-index'); - // Changes are being applied: an answer from the old map could point at - // files that just moved. Stay quiet until the worker catches up. - if (structureRead.readDescriptor(root).freshness === 'dirty') return quiet(root, 'map-dirty'); - // STR-02c: the live working-tree view first, the tracked map otherwise. const structure = readJson(structureRead.resolveReadPath(root)); if (!structure || !structure.intentIndex) return quiet(root, 'no-index'); + const result = retrieval.legacyRetrieve(structure, loadIntentMap(), { mode: 'hook', words }); + if (!result.groups.length) return quiet(root, 'no-match'); + const hit = result.groups[0]; + const keyword = result.keyword; - let hit = null; - let keyword = null; - for (const w of words) { - hit = lookup(structure, w); - if (hit) { keyword = w; break; } - } - if (!hit) return quiet(root, 'no-match'); - - const sessionId = input.session_id; - const state = loadState(root, sessionId); + const state = loadState(root, input.session_id); if (state.concepts.includes(hit.feature)) return quiet(root, 'session-dedup'); cleanupState(root); state.concepts.push(hit.feature); - saveState(root, sessionId, state); + saveState(root, input.session_id, state); note(root, 'hint.injected', { concept: keyword, modules: Math.min(hit.modules.length, MAX_MODULES) }); emit(render(root, structure, hit, keyword)); } +/** The index to answer from: a current lookup.json, else the map compiled in memory. */ +function loadIndex(root) { + const cap = retrieval.LIMITS.hookIndexBytes; + const loaded = retrieval.loadLookup(root, { maxBytes: cap, curationPath: path.join(__dirname, 'intent-map.json') }); + if (loaded.state === 'fresh') return { index: loaded.index }; + if (loaded.state === 'oversize') return { reason: 'index-oversize' }; + const compiled = retrieval.indexFromMap(root, { maxBytes: cap, curationPath: path.join(__dirname, 'intent-map.json') }); + if (compiled.state === 'compiled') return { index: compiled.index }; + return { reason: compiled.state === 'oversize' ? 'index-oversize' : 'no-index' }; +} + +function fingerprint(paths) { + let h = 0; + for (const ch of paths.join('\n')) h = (Math.imul(h, 31) + ch.codePointAt(0)) | 0; + return (h >>> 0).toString(16); +} + +/** One line of the query, safe to show and to paste as a shell argument. */ +function shownQuery(pattern) { + return pattern.replace(/[\r\n\t]+/g, ' ').trim().slice(0, 80); +} + +function shellQuote(text) { + return `'${text.replace(/'/g, `'\\''`)}'`; +} + +function renderV2(root, result, pattern, descriptor) { + const q = shownQuery(pattern); + const evidence = [...new Set(result.answer.map((c) => c.evidence))].join(', '); + const verified = descriptor.freshness === 'fresh'; + const head = verified + ? `Frame's module map points to these files for "${q}" (${evidence}):` + : `Frame's module map has candidates for "${q}" (${evidence}) — map not verified recently (${descriptor.freshness}):`; + const tail = `Start from these files rather than a broad scan; your search still runs. ` + + `Full query: node ${finderCliPath(root)} ${shellQuote(q)}`; + const moreLine = ` … more — ${finderCliPath(root)} lists them`; + // head \n lines… \n [more \n] tail — the "more" line is always reserved + const total = (ls) => head.length + 1 + ls.reduce((n, l) => n + l.length + 1, 0) + moreLine.length + 1 + tail.length; + const lines = []; + const answer = result.answer.slice(0, MAX_MODULES); + for (let i = 0; i < answer.length; i++) { + const c = answer[i]; + const share = Math.floor((MAX_CONTEXT_CHARS - total(lines)) / (answer.length - i)) - 1; + const base = ` ${c.path}`; + if (share < base.length) break; // stop on a whole candidate, never a cut path + const room = share - base.length - 3; + const desc = c.description && room > 12 + ? ` — ${c.description.length > room ? `${c.description.slice(0, room - 1)}…` : c.description}` + : ''; + lines.push(base + desc); + } + if (!lines.length) return null; + if (result.answer.length > lines.length || result.truncated) lines.push(moreLine); + return `${head}\n${lines.join('\n')}\n${tail}`; +} + +function v2Mode(root, input, descriptor) { + const pattern = patternFrom(input.tool_name, input.tool_input || {}); + if (!pattern || !pattern.trim()) return quiet(root, 'no-words'); + // A scan that missed files: an answer could be the wrong one. Stay quiet. + if (descriptor.coverage && descriptor.coverage !== 'complete') return quiet(root, 'map-incomplete'); + + const { index, reason } = loadIndex(root); + if (!index) return quiet(root, reason); + const exists = (rel) => existsInProject(root, rel); + const result = retrieval.retrieve(index, pattern, { mode: 'hook', exists }); + if (result.status === 'no-match') { + const weak = retrieval.retrieve(index, pattern, { mode: 'cli', limit: 1 }); + return quiet(root, weak.status === 'no-match' ? 'no-match' : 'ambiguous-weak'); + } + + const key = `${index.revision || (index.source && JSON.stringify(index.source.signature)) || 'map'}|${fingerprint(result.answer.map((c) => c.path))}`; + const sessionId = input.session_id; + if (sessionId) { + const state = loadState(root, sessionId); + if (state.delivered.includes(key)) return quiet(root, 'session-dedup'); + cleanupState(root); + state.delivered.push(key); + if (state.delivered.length > 256) state.delivered.splice(0, state.delivered.length - 256); + saveState(root, sessionId, state); + } + + const context = renderV2(root, result, pattern, descriptor); + if (!context) return quiet(root, 'no-context'); + note(root, 'hint.injected', { concept: shownQuery(pattern), modules: result.answer.length }); + emit(context); +} + +function searchMode(input) { + const root = resolveRoot(input.cwd); + + if (patternFrom(input.tool_name, input.tool_input || {}) === null) return; // not a search: silent, unrecorded + + if (!structureRead || !retrieval) return quiet(root, 'no-index'); + const descriptor = structureRead.readDescriptor(root); + // Changes are being applied: an answer from the old map could point at + // files that just moved. Stay quiet until the worker catches up. + if (descriptor.freshness === 'dirty') return quiet(root, 'map-dirty'); + + if (engineFor(root) === 'legacy') return legacyMode(root, input, descriptor); + return v2Mode(root, input, descriptor); +} + try { const input = JSON.parse(readStdin() || '{}'); setHookCli(input); diff --git a/src/shared/activityEvents.js b/src/shared/activityEvents.js index 0cd70616..ce07cf20 100644 --- a/src/shared/activityEvents.js +++ b/src/shared/activityEvents.js @@ -52,7 +52,10 @@ const HINT_REASONS = [ 'no-match', // prompt: no spec scored 'no-stale-free-match', // prompt: matches existed but none survived filtering 'no-context', // emit: nothing composed to send - 'map-dirty' // search: STRUCTURE.json changes are still being applied (STR-02) + 'map-dirty', // search: STRUCTURE.json changes are still being applied (STR-02) + 'map-incomplete', // search: the last scan missed files, an answer could be wrong (STR-03) + 'index-oversize', // search: the lookup index or map is above the hook's 2 MiB cap (STR-03) + 'ambiguous-weak' // search: only path or description words matched — CLI evidence, not a hint (STR-03) ]; const HOSTS = ['app', 'claude-hook', 'codex-hook', 'git-precommit', 'orch-bus', 'cli']; @@ -336,6 +339,10 @@ const HINT_REASON_TEXT = { // falls back to the spec wording. const SEARCH_REASON_TEXT = { 'map-dirty': 'the module map is being updated', + 'map-incomplete': 'the module map missed files in its last scan', + 'index-oversize': 'the module map is too large to read in a hook', + 'ambiguous-weak': 'only weak matches (path or description words)', + 'no-context': 'nothing fit in the hint', 'no-index': 'no module map in STRUCTURE.json', 'no-words': 'no concept word in the search', 'no-match': 'no module matched the concept', diff --git a/test/activityEvents.test.js b/test/activityEvents.test.js index ab7f7eff..5bd7fab8 100644 --- a/test/activityEvents.test.js +++ b/test/activityEvents.test.js @@ -67,8 +67,9 @@ test('the guard events are classified as suppressions, not actions', () => { test('every spec-hint quiet path has its own reason code', () => { // eleven quiet returns in spec-hint.js, plus module-hint's `map-dirty` (STR-02) - assert.equal(events.HINT_REASONS.length, 12, 'one code per quiet return in the hint hooks'); - assert.equal(new Set(events.HINT_REASONS).size, 12, 'codes are distinct'); + // and `map-incomplete`, `index-oversize`, `ambiguous-weak` (STR-03) + assert.equal(events.HINT_REASONS.length, 15, 'one code per quiet return in the hint hooks'); + assert.equal(new Set(events.HINT_REASONS).size, 15, 'codes are distinct'); for (const reason of events.HINT_REASONS) { const out = events.validateEvent('hint.quiet', { reason }); assert.equal(out.reason, reason, `${reason} must pass the enum`); diff --git a/test/module-hint.test.js b/test/module-hint.test.js index 5935027d..0625195e 100644 --- a/test/module-hint.test.js +++ b/test/module-hint.test.js @@ -207,13 +207,29 @@ test('an unowned root STRUCTURE.json is never read as Frame\'s map', () => { assert.match(runHook(bash(root, 'grep -rn github src/', 's2')).hookSpecificOutput.additionalContext, /githubManager/); }); -test('the hook never loads builder or state code', () => { - const source = fs.readFileSync(HOOK, 'utf8'); - const requires = [...source.matchAll(/require\(\s*['"]([^'"]+)['"]\s*\)/g)].map((m) => m[1]); - // the long-standing record/vocabulary helpers plus the read-only freshness - // contract (STR-02 D10); never a builder, the state writer or a process - assert.deepEqual(requires.filter((r) => r.startsWith('.')).sort(), ['./activity-log', './structure-read', './toolVocabulary']); - assert.ok(!requires.some((r) => /structure-(discovery|generation|state|snapshot|lifecycle)|update-structure|child_process/.test(r)), requires.join(', ')); +test('the hook never loads builder or state code, directly or through a helper', () => { + // Walk the whole local import closure: the record/vocabulary helpers, the + // read-only freshness contract (STR-02 D10) and the shared retrieval engine + // (STR-03) — never a builder, the state writer, a process or the network. + const seen = new Set(); + const builtins = new Set(); + const walk = (file) => { + if (seen.has(file)) return; + seen.add(file); + // comments may quote require() examples; only code counts + const source = fs.readFileSync(file, 'utf8').replace(/\/\*[\s\S]*?\*\//g, '').replace(/^\s*\/\/.*$/gm, ''); + for (const m of source.matchAll(/require\(\s*['"]([^'"]+)['"]\s*\)/g)) { + if (m[1].startsWith('.')) walk(require.resolve(path.join(path.dirname(file), m[1]))); + else builtins.add(m[1]); + } + }; + walk(HOOK); + const local = [...seen].map((f) => path.basename(f)).sort(); + assert.deepEqual(local, ['activity-log.js', 'module-hint.js', 'redact.js', 'structure-read.js', 'structure-retrieval.js', 'toolVocabulary.js']); + assert.ok(!local.some((f) => /structure-(discovery|generation|state|snapshot|lifecycle|commit)|update-structure/.test(f)), local.join(', ')); + for (const banned of ['child_process', 'net', 'http', 'https', 'dgram', 'worker_threads']) { + assert.ok(!builtins.has(banned), `${banned} in the closure`); + } }); // ─── STR-02: freshness ──────────────────────────────────── @@ -273,3 +289,153 @@ test('the working view answers before the tracked map', () => { assert.match(ctx, /githubLive\.js/); assert.ok(!/githubManager/.test(ctx)); }); + +// ─── STR-03: the v2 engine ──────────────────────────────── + +const retrieval = require('../scripts/structure-retrieval'); + +const V2_STRUCTURE = { + generation: { revision: 'rev-a' }, + intentIndex: STRUCTURE.intentIndex, + modules: { + ...STRUCTURE.modules, + 'renderer/github/rowModels': { file: 'src/renderer/github/rowModels.js', description: 'Row view-models', functions: { issueBranchName: {} } }, + 'main/fsSafe': { file: 'src/main/fsSafe.js', description: 'Durable state-file primitives', functions: { writeFileAtomic: {} } } + } +}; + +function v2Project(structure = V2_STRUCTURE) { + const root = mkProject(structure); + fs.writeFileSync(path.join(root, '.frame', 'config.json'), JSON.stringify({ project: { retrieval: { engine: 'v2' } } })); + for (const m of Object.values(structure.modules)) { + fs.mkdirSync(path.dirname(path.join(root, m.file)), { recursive: true }); + fs.writeFileSync(path.join(root, m.file), '// x\n'); + } + return root; +} + +function runCodex(input) { + const stdout = execFileSync('node', [HOOK, 'search', 'codex'], { input: JSON.stringify(input), encoding: 'utf8' }); + return stdout.trim() ? JSON.parse(stdout) : null; +} + +const ctxOf = (out) => out && out.hookSpecificOutput.additionalContext; + +test('v2: a file outside every concept is hinted by symbol and by file name, with the evidence named', () => { + const root = v2Project(); + const bySymbol = ctxOf(runHook({ session_id: 'v1', cwd: root, tool_name: 'Grep', tool_input: { pattern: 'issueBranchName' } })); + assert.match(bySymbol, /for "issueBranchName" \(symbol\)/); + assert.match(bySymbol, /^ {2}src\/renderer\/github\/rowModels\.js — Row view-models$/m); + const byName = ctxOf(runHook({ session_id: 'v2', cwd: root, tool_name: 'Glob', tool_input: { pattern: '**/fsSafe.js' } })); + assert.match(byName, /\(file name\)/); + assert.match(byName, /src\/main\/fsSafe\.js/); + assert.match(ctxOf(runHook(bash(root, 'find . -path "*/fsSafe.js"', 'v3'))), /src\/main\/fsSafe\.js/); + assert.match(byName, /Full query: node \S+ '\*\*\/fsSafe\.js'$/, 'the query is shell-quoted'); +}); + +test('v2: the Codex adapter (Bash only, `search codex`) gets the same answer in the same output shape', () => { + const root = v2Project(); + const out = runCodex({ session_id: 'cx', cwd: root, turn_id: 't', model: 'm', tool_name: 'Bash', tool_input: { command: 'rg -n "writeFileAtomic" src/' } }); + assert.equal(out.hookSpecificOutput.hookEventName, 'PreToolUse'); + assert.match(ctxOf(out), /src\/main\/fsSafe\.js/); +}); + +test('v2: unexplained words, weak matches and comment markers stay silent', () => { + const root = v2Project(); + assert.equal(runHook(bash(root, 'grep -rn "webpack config" .', 'q1')), null); + assert.equal(runHook(bash(root, 'grep -rn "primitives" src/', 'q2')), null, 'a description word is CLI evidence'); + assert.equal(runHook(bash(root, 'grep -rn "TODO" src/', 'q3')), null); + assert.equal(runHook(bash(root, 'grep -rn "console.log" src/', 'q4')), null); +}); + +test('v2: an unverified map hints as candidates; a dirty or incomplete map stays quiet', () => { + const root = v2Project(); + assert.match(ctxOf(runHook({ session_id: 'f1', cwd: root, tool_name: 'Grep', tool_input: { pattern: 'github' } })), + /has candidates for "github" \(concept\) — map not verified recently \(unknown\)/); + + withReceipt(root); + assert.match(ctxOf(runHook({ session_id: 'f2', cwd: root, tool_name: 'Grep', tool_input: { pattern: 'github' } })), + /^Frame's module map points to these files for "github" \(concept\):/); + + withReceipt(root, { dirty: ['file-event'], epoch: { requested: 2, applied: 1 } }); + assert.equal(runHook({ session_id: 'f3', cwd: root, tool_name: 'Grep', tool_input: { pattern: 'github' } }), null); + + withReceipt(root); + const lc = path.join(root, '.frame', 'runtime', 'structure', 'lifecycle.json'); + const state = JSON.parse(fs.readFileSync(lc, 'utf8')); + state.receipt.coverage = 'partial'; + fs.writeFileSync(lc, JSON.stringify(state)); + assert.equal(runHook({ session_id: 'f4', cwd: root, tool_name: 'Grep', tool_input: { pattern: 'github' } }), null); +}); + +test('v2: a file missing on disk is never hinted', () => { + const root = v2Project(); + fs.rmSync(path.join(root, 'src/renderer/github/rowModels.js')); + assert.equal(runHook({ session_id: 'm1', cwd: root, tool_name: 'Grep', tool_input: { pattern: 'issueBranchName' } }), null); +}); + +test('v2: dedup is per answer and per map revision, and nothing is remembered without a session id', () => { + const root = v2Project(); + const ask = (session, pattern = 'github') => runHook({ session_id: session, cwd: root, tool_name: 'Grep', tool_input: { pattern } }); + assert.ok(ask('d1')); + assert.equal(ask('d1'), null, 'same answer, same revision'); + assert.ok(ask('d1', 'issueBranchName'), 'a different answer'); + + const map = path.join(root, '.frame', 'STRUCTURE.json'); + const next = JSON.parse(fs.readFileSync(map, 'utf8')); + next.generation.revision = 'rev-b'; + fs.writeFileSync(map, JSON.stringify(next)); + assert.ok(ask('d1'), 'a new revision may hint again in the same session'); + + assert.ok(runHook({ cwd: root, tool_name: 'Grep', tool_input: { pattern: 'github' } })); + assert.ok(runHook({ cwd: root, tool_name: 'Grep', tool_input: { pattern: 'github' } }), 'no session: no shared bucket'); +}); + +test('v2: a published lookup.json is used while current; a stale one is never trusted', () => { + const root = v2Project(); + assert.equal(retrieval.publishLookup(root, { curationPath: path.join(__dirname, '..', 'scripts', 'intent-map.json') }).status, 'published'); + const file = retrieval.lookupPath(root); + const index = JSON.parse(fs.readFileSync(file, 'utf8')); + index.terms['5:onlyinindex'] = [index.files.findIndex(([p]) => p === 'src/main/fsSafe.js')]; + fs.writeFileSync(file, JSON.stringify(index)); + assert.match(ctxOf(runHook({ session_id: 'l1', cwd: root, tool_name: 'Grep', tool_input: { pattern: 'onlyInIndex' } })), /fsSafe/); + const map = path.join(root, '.frame', 'STRUCTURE.json'); + fs.writeFileSync(map, fs.readFileSync(map, 'utf8') + ' '); + assert.equal(runHook({ session_id: 'l2', cwd: root, tool_name: 'Grep', tool_input: { pattern: 'onlyInIndex' } }), null); +}); + +test('v2: an index or map above the 2 MiB cap keeps the hook quiet', () => { + const root = v2Project(); + fs.mkdirSync(path.join(root, '.frame', 'runtime', 'structure'), { recursive: true }); + fs.writeFileSync(retrieval.lookupPath(root), JSON.stringify({ version: 1, pad: 'x'.repeat(retrieval.LIMITS.hookIndexBytes + 10) })); + assert.equal(runHook({ session_id: 'o1', cwd: root, tool_name: 'Grep', tool_input: { pattern: 'github' } }), null); + fs.rmSync(retrieval.lookupPath(root)); + const map = path.join(root, '.frame', 'STRUCTURE.json'); + const big = JSON.parse(fs.readFileSync(map, 'utf8')); + big.pad = 'x'.repeat(retrieval.LIMITS.hookIndexBytes + 10); + fs.writeFileSync(map, JSON.stringify(big)); + assert.equal(runHook({ session_id: 'o2', cwd: root, tool_name: 'Grep', tool_input: { pattern: 'github' } }), null); +}); + +test('v2: the payload stays within 1,800 characters and 8 files, cut on whole candidates', () => { + const modules = {}; + const group = []; + for (let i = 0; i < 20; i++) { + const file = `src/components/deeply/nested/folder/widgetComponentNumber${i}.js`; + modules[`w${i}`] = { file, description: `Widget component ${i} ${'with a very long description '.repeat(8)}` }; + group.push({ module: `w${i}`, file }); + } + const root = v2Project({ generation: { revision: 'r' }, modules, intentIndex: { widgets: group } }); + const ctx = ctxOf(runHook({ session_id: 'p1', cwd: root, tool_name: 'Grep', tool_input: { pattern: 'widgets' } })); + assert.ok(ctx.length <= 1800, `${ctx.length} characters`); + const listed = ctx.split('\n').filter((l) => /^ {2}src\//.test(l)); + assert.ok(listed.length >= 1 && listed.length <= 8); + for (const line of listed) assert.match(line, /^ {2}src\/components\/deeply\/nested\/folder\/widgetComponentNumber\d+\.js( — .+)?$/); + assert.match(ctx, /… more — /); +}); + +test('the default engine (legacy) keeps today\'s output until the gates promote v2', () => { + if (retrieval.DEFAULT_ENGINE !== 'legacy') return; + const root = mkProject(); + assert.match(ctxOf(runHook(bash(root, 'grep -rn "github" src/', 'lg'))), /already answers "github" \(STRUCTURE\.json intentIndex\)/); +}); From be4f19a4904b719f40ee475b644e5c4612676e56 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kaan=20=C3=96zhan?= Date: Thu, 1 Oct 2026 16:28:40 +0300 Subject: [PATCH 7/9] docs(eval): held-out retrieval results; legacy stays the default until v2 passes its gates (STR-03 T06) --- .frame/STRUCTURE.json | 36 ++++++------- .../str-03-local-file-retrieval/outcome.md | 18 +++++++ scripts/eval/README.md | 50 +++++++++++++++++++ src/shared/frameTemplates.js | 14 ++++++ 4 files changed, 100 insertions(+), 18 deletions(-) diff --git a/.frame/STRUCTURE.json b/.frame/STRUCTURE.json index e626d48d..30043438 100644 --- a/.frame/STRUCTURE.json +++ b/.frame/STRUCTURE.json @@ -16382,7 +16382,7 @@ "exports": [], "depends": [], "functions": {}, - "sizeBytes": 7654, + "sizeBytes": 10176, "extraction": { "status": "parsed" }, @@ -20657,7 +20657,7 @@ "purpose": "REFERENCE.md template — the reference-on-demand companion to the lean" }, "getStructureTemplate": { - "line": 708, + "line": 722, "params": [ "projectName", "project" @@ -20665,21 +20665,21 @@ "purpose": "STRUCTURE.json template" }, "getNotesTemplate": { - "line": 736, + "line": 750, "params": [ "projectName" ], "purpose": "PROJECT_NOTES.md template" }, "getTasksTemplate": { - "line": 756, + "line": 770, "params": [ "projectName" ], "purpose": "tasks.json template" }, "getQuickstartTemplate": { - "line": 802, + "line": 816, "params": [ "projectName", "project" @@ -20687,24 +20687,24 @@ "purpose": "QUICKSTART.md template" }, "getFrameConfigTemplate": { - "line": 885, + "line": 899, "params": [ "projectName" ], "purpose": ".frame/config.json template" }, "getClaudeRuleTemplate": { - "line": 928, + "line": 942, "params": [ "agentsText" ], "purpose": "`.claude/rules/frame.md` — the file Claude Code loads at session start." }, "getFrameGitignoreBlock": { - "line": 943 + "line": 957 }, "getGenericWrapperTemplate": { - "line": 1096, + "line": 1110, "params": [ "toolCommand", "promptFlag = ''" @@ -20712,38 +20712,38 @@ "purpose": "Generic AI tool wrapper template" }, "getStructureHookSnippet": { - "line": 1142 + "line": 1156 }, "getStructurePreCommitHookTemplate": { - "line": 1179, + "line": 1193, "purpose": "Full pre-commit hook file content for the \"no existing hook\" case." }, "classifyStructureHook": { - "line": 1208, + "line": 1222, "params": [ "text" ], "purpose": "'current' when `text` is today's template, 'previous' when it is an" }, "getOrchBusHeader": { - "line": 1227, + "line": 1241, "purpose": "Orchestration command-channel scripts (.frame/bin/)" }, "getOrchRequestScript": { - "line": 1239, + "line": 1253, "params": [ "type" ] }, "getOrchStatusScript": { - "line": 1259 + "line": 1273 }, "getOrchBinScripts": { - "line": 1277, + "line": 1291, "purpose": "Map of filename → script body for the orchestration bin scripts. The" } }, - "sizeBytes": 54568, + "sizeBytes": 55440, "extraction": { "status": "parsed" }, @@ -26576,6 +26576,6 @@ "truncated": false, "samples": [] }, - "revision": "df418cd0388ef0973a219130be810456a7fc1a28710f02aa9e2c4562af463d03" + "revision": "121a88a87c6e220101e0edce75b3519c0b5c050fb98c9c260548db9db91f7998" } } diff --git a/.frame/specs/str-03-local-file-retrieval/outcome.md b/.frame/specs/str-03-local-file-retrieval/outcome.md index e9909f2a..55e6943e 100644 --- a/.frame/specs/str-03-local-file-retrieval/outcome.md +++ b/.frame/specs/str-03-local-file-retrieval/outcome.md @@ -97,3 +97,21 @@ Files touched: `scripts/module-hint.js`, `src/shared/activityEvents.js`, `test/m _Captured: 2026-10-01 · 4 file change(s)_ --- + +## T06 — Held-out run and promotion decision + +Both engines were run once on the held-out split (122 cases) after T05; v2 had been tuned on the development split only. + +v2 on held-out: exact recall 100%, recall@5 89.4%, P@1 92.0%, hook precision 96.5% (both adapters), false hints 3.6% (1/28), hook p95 41 ms on this repository and 55 ms at 10k files, max payload 692 characters, a 130 KB index against the 662 KB map. Legacy: 64.9 / 57.4 / 55.3 / 56.6 / 10.7%, 10k-file p95 68 ms. + +**The default stays `legacy`** (D2): v2 misses four predeclared gates (recall@5, hint precision, false hints, 10k-file latency), even though it beats legacy on every metric. `DEFAULT_ENGINE` is unchanged, and v2 is opt-in through `project.retrieval.engine` or `--retrieval=v2`. + +Failure analysis, without changing the engine afterwards: +- All 10 recall misses are Turkish: 8 purely Turkish queries that need curated synonyms, and 2 with ASCII Turkish words. +- The 3 wrong hints come from the coverage relaxation combined with partial concepts. + +Next-round candidates are recorded in `scripts/eval/README.md`, to be validated on a new held-out split. REFERENCE (STRUCTURE.json Rules) gained "Looking Files Up": what find-module answers, `--json`/`--limit`, the hook sharing the engine, and the two engines with the opt-in. Files touched: `scripts/eval/README.md`, `src/shared/frameTemplates.js`. + +_Captured: 2026-10-01 · 2 file change(s)_ + +--- diff --git a/scripts/eval/README.md b/scripts/eval/README.md index b30936b4..585fdb4d 100644 --- a/scripts/eval/README.md +++ b/scripts/eval/README.md @@ -156,3 +156,53 @@ and false hints (both adapters), and the 10k-file hook latency. The misses are the ones the spec describes: files outside every intent group, function names, and partial concept matches that hint the wrong group (for example `structureBootstrap nerede` → every file in the `structure` group). + +## Held-out result and promotion decision — 2026-10-01 + +Same machine and pin as the baseline. v2 was tuned on the development split +only. The held-out split was run once, for both engines, after T05. + +| Held-out (122) | legacy | v2 | Gate | +|---|---|---|---| +| Exact recall (77) | 64.9% | **100%** | 100% ✓ | +| recall@5 | 57.4% | 89.4% | ≥ 90% ✗ | +| precision@1 | 55.3% | **92.0%** | ≥ 90% ✓ | +| Emitted-hint precision (Claude / Codex) | 56.6% | 96.5% | ≥ 98% ✗ | +| Hint recall | 31.9% | 88.3% | — | +| False hints on negatives (28) | 10.7% | 3.6% (1) | ≤ 2% ✗ | +| Hook p50/p95, this repository | 31 / 35 ms | 34 / 41 ms | ≤ 50 ms ✓ | +| Hook p95, 10k files | 68 ms | 55 ms | ≤ 50 ms ✗ | +| CLI p95 | 30 ms | 39 ms (10k: 105 ms) | ≤ 150 ms ✓ | +| Max payload | 852 chars | 692 chars | ≤ 1,800 ✓ | +| Index read by the hook | 662 KB map | 130 KB lookup | — | + +Development split, v2: recall@5 95.1%, P@1 98.3%, hook precision 100%, +false hints 0/11. + +**Decision: the default stays `legacy`.** v2 is better than legacy on every +measured metric, but it misses four predeclared gates, and the rule fixed +before the run was that all of them must pass. v2 is available now with +`--retrieval=v2` or `"project": { "retrieval": { "engine": "v2" } }` in +`.frame/config.json`. + +Failure analysis (no engine change was made after this run): +- **recall@5:** all 10 misses are Turkish queries. 8 are purely Turkish + ("ayarlar", "komut paleti", "güncelleme kontrolü") and need Turkish + synonyms in `intent-map.json`, which is curation and out of scope. 2 contain + ASCII Turkish words that the coverage rule cannot treat as prose + ("structureBootstrap nerede", "ana sayfa widget kaydı"). +- **Hint precision and false hints:** the 3 wrong hints come from the + "no file carries every word → rank by coverage" relaxation combined with + partial concepts. "görev paneli" hinted the `panel` group, "görev çalıştırma + modalı" the `modal` group, and "docker compose" `feedbackReport.js` (`dock` ⊂ + "docker", plus its `compose` function). +- **10k-file latency:** the 1.9 MB lookup index costs about 25 ms to parse on + top of node's startup. + +Candidates for the next round, each to be validated on a **new** held-out +split (this one is now spent): +1. Hooks require every identifier word to be covered by one file; the + relaxation stays CLI-only. +2. Turkish synonyms for the curated concepts. +3. A smaller index at scale: drop description postings for hooks, or split the + index by tier. diff --git a/src/shared/frameTemplates.js b/src/shared/frameTemplates.js index df77556c..bd69a2bb 100644 --- a/src/shared/frameTemplates.js +++ b/src/shared/frameTemplates.js @@ -643,6 +643,20 @@ No problem, continue. The user can also say what they consider important themsel runs \`--changed\` and then \`git add\`s the map does the same. Frame keeps its own unedited hook up to date. +### Looking Files Up +- \`find-module.js\` (in \`.frame/bin/\`) answers a concept, a synonym, a file + name, a path or a function/IPC name. Each result says which of those + matched; several files of equal standing are listed as candidates. Add + \`--json\` for one machine-readable result and \`--limit N\` (at most 20). +- The search hint that appears next to a grep uses the same lookup with + stricter evidence, and stays quiet while the map is being updated. +- Two engines: \`legacy\` (the default — concepts and synonyms first, then a + text search) and \`v2\` (also file names, paths and symbols, with a compact + index kept in \`.frame/runtime/structure/\`). v2 becomes the default once + it passes Frame's retrieval benchmark. Opt in now with + \`"project": { "retrieval": { "engine": "v2" } }\` in \`.frame/config.json\`, + or per call with \`--retrieval=v2\`. + ### What to Edit - Enrich entries in place in the tracked STRUCTURE.json (never in the live view): \`description\`, function \`purpose\`, fields of your own, and From c7526dd2958d64aa4daf3fc8a19abca8a93ca8f5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kaan=20=C3=96zhan?= Date: Thu, 1 Oct 2026 16:32:50 +0300 Subject: [PATCH 8/9] feat(eval): matched-agent retrieval arms with full token accounting and cell validity (STR-03 T07) --- .frame/STRUCTURE.json | 122 ++++++++++++--- .../str-03-local-file-retrieval/outcome.md | 24 +++ scripts/eval/README.md | 37 +++++ scripts/eval/run-eval.js | 144 +++++++++++++++--- scripts/eval/score.js | 141 +++++++++++++++-- scripts/eval/tasks.json | 104 ++++++++++++- test/retrievalEval.test.js | 112 ++++++++++++++ 7 files changed, 634 insertions(+), 50 deletions(-) diff --git a/.frame/STRUCTURE.json b/.frame/STRUCTURE.json index 30043438..d19e7769 100644 --- a/.frame/STRUCTURE.json +++ b/.frame/STRUCTURE.json @@ -16382,7 +16382,7 @@ "exports": [], "depends": [], "functions": {}, - "sizeBytes": 10176, + "sizeBytes": 11950, "extraction": { "status": "parsed" }, @@ -16405,7 +16405,11 @@ "scripts/eval/run-eval": { "file": "scripts/eval/run-eval.js", "description": "", - "exports": [], + "exports": [ + "shuffled", + "hookActivity", + "RETRIEVAL_ARMS" + ], "depends": [ "fs", "os", @@ -16414,38 +16418,71 @@ ], "functions": { "parseArgs": { - "line": 52 + "line": 60 + }, + "shuffled": { + "line": 84, + "params": [ + "items", + "seed" + ], + "purpose": "Deterministic shuffle (mulberry32) so a run order can be reproduced." + }, + "setupRetrievalArm": { + "line": 106, + "params": [ + "wt", + "arm", + "activityHome" + ], + "purpose": "Retrieval-arm setup: the same map for every arm (this checkout's" + }, + "hookActivity": { + "line": 131, + "params": [ + "activityHome" + ], + "purpose": "Search-hook records a cell produced: { records, injected }." }, "git": { - "line": 70, + "line": 156, "params": [ "cmd", "cwd" ] }, "setupHooks": { - "line": 80, + "line": 166, "params": [ "wt" ] }, "runOne": { - "line": 105, + "line": 191, "params": [ "task", "arm", "resultsDir", "timeoutSec", - "hooks" + "hooks", + "suite = SUITE", + "repeat = 1" ] }, "main": { - "line": 193 + "line": 290 } }, - "sizeBytes": 8937, + "sizeBytes": 13923, "extraction": { "status": "parsed" + }, + "provenance": { + "purposes": { + "hookActivity": "4631feb261596931", + "setupRetrievalArm": "783c6bac4f5c86a2", + "shuffled": "a75b35feb454edac" + } } }, "scripts/eval/run-retrieval": { @@ -16634,37 +16671,68 @@ "scripts/eval/score": { "file": "scripts/eval/score.js", "description": "", - "exports": [], + "exports": [ + "scoreTranscript", + "scoreRun", + "cellValidity", + "paired", + "aggregate" + ], "depends": [ "fs", "path" ], "functions": { "scoreTranscript": { - "line": 28, + "line": 42, "params": [ - "file" + "file", + "options = {}" ] }, + "cellValidity": { + "line": 103, + "params": [ + "meta", + "stats" + ], + "purpose": "Whether a retrieval-arm cell measured what its arm intends. Returns" + }, "scoreRun": { - "line": 61, + "line": 112, "params": [ "runDir" ] }, + "paired": { + "line": 146, + "params": [ + "runs", + "a", + "b", + "metric" + ], + "purpose": "Paired per-task comparison of arm `b` against arm `a` on valid cells:" + }, "aggregate": { - "line": 82, + "line": 169, "params": [ "runs" ] }, "main": { - "line": 100 + "line": 200 } }, - "sizeBytes": 6305, + "sizeBytes": 12560, "extraction": { "status": "parsed" + }, + "provenance": { + "purposes": { + "cellValidity": "a2468a06292c5481", + "paired": "49da72b4b2b66533" + } } }, "scripts/eval/tasks.json": { @@ -16673,7 +16741,7 @@ "exports": [], "depends": [], "functions": {}, - "sizeBytes": 4609, + "sizeBytes": 12163, "extraction": { "status": "unsupported", "reason": "no-extractor" @@ -23594,10 +23662,22 @@ "child_process", "scripts/eval/run-retrieval", "fs", - "os" + "os", + "scripts/eval/score", + "scripts/eval/run-eval", + "scripts/eval/tasks.json" ], - "functions": {}, - "sizeBytes": 7907, + "functions": { + "cell": { + "line": 154, + "params": [ + "t", + "{ meta", + "events }" + ] + } + }, + "sizeBytes": 14299, "extraction": { "status": "parsed" }, @@ -26576,6 +26656,6 @@ "truncated": false, "samples": [] }, - "revision": "121a88a87c6e220101e0edce75b3519c0b5c050fb98c9c260548db9db91f7998" + "revision": "8ddbb6d181092de18f78edbe353a52d9047cdfbea53647748c4655eaa5fab61c" } } diff --git a/.frame/specs/str-03-local-file-retrieval/outcome.md b/.frame/specs/str-03-local-file-retrieval/outcome.md index 55e6943e..d5e86d5c 100644 --- a/.frame/specs/str-03-local-file-retrieval/outcome.md +++ b/.frame/specs/str-03-local-file-retrieval/outcome.md @@ -115,3 +115,27 @@ Next-round candidates are recorded in `scripts/eval/README.md`, to be validated _Captured: 2026-10-01 · 2 file change(s)_ --- + +## T07 — Matched-agent instrument + +`run-eval.js --retrieval-arms [--repeat N] [--seed S]` runs the new `retrievalSuite` (12 navigation tasks at 262f91b) across `no-hint` / `legacy` / `v2`, in a mulberry32-shuffled, reproducible order. + +Each cell: +- builds the map with this checkout's `update-structure.js --full` (which publishes `lookup.json`); +- for hooked arms, sets `project.retrieval.engine` and registers this checkout's `module-hint.js` for Grep|Glob|Bash with a per-cell `FRAME_ACTIVITY_HOME`; +- commits the setup, so it never counts as agent diff; +- records `setupOk`, `hookRecords`, `hintsInjected`, `worktree` and `repeat` in meta. + +`score.js` adds: +- search and read calls, files read (made relative to the worktree), and expected files found (read or changed); +- token accounting from the final `result` event only: input, cache creation and cache read, with output separate. No result event means `null`/unknown, kept out of averages. +- `cellValidity`: setup failed, hook never ran while the agent searched, or hook activity in the no-hint arm. +- `paired`: per-task means across repeats, valid cells only, with mean difference and lower/higher/same counts. Comparisons are printed and included in `--json`. + +The legacy frame/bare suite is unchanged. Re-scoring the old haiku pilot with cache tokens shows the frame arm used more input tokens (373k vs 173k average), another reason no saving is claimed before S8 runs. + +Deviation: task ids are neutral (`nav-01…12`), because descriptive ids leaked file names through the marker line. A test enforces that no prompt contains its file's stem. The README records the S8 protocol as **pending**: the paid run happens only on explicit command. Files touched: `scripts/eval/run-eval.js`, `scripts/eval/score.js`, `scripts/eval/tasks.json`, `scripts/eval/README.md`, `test/retrievalEval.test.js`. + +_Captured: 2026-10-01 · 5 file change(s)_ + +--- diff --git a/scripts/eval/README.md b/scripts/eval/README.md index 585fdb4d..7221d136 100644 --- a/scripts/eval/README.md +++ b/scripts/eval/README.md @@ -206,3 +206,40 @@ split (this one is now spent): 2. Turkish synonyms for the curated concepts. 3. A smaller index at scale: drop description postings for hooks, or split the index by tier. + +## Matched-agent navigation (S8) — pending + +Does the search hint actually save the agent work? The retrieval benchmark +measures the hint, not its effect. This run measures the effect, and it costs +real agent time and tokens, so it only runs when asked. + +- **Suite** — `retrievalSuite` in `tasks.json`: 12 navigation tasks at the + corpus pin (262f91b). Each prompt describes a file's behavior, never its + name. The task is done when a marker comment lands in that file and in no + other (`successCheck`). +- **Arms** — `no-hint`, `legacy`, `v2`. Every cell gets its own worktree, + with this checkout's `update-structure.js --full` map and the same prompt, + model and permissions. Only the search hook differs: none, or this + checkout's `module-hint.js` with the arm's engine. The hook writes to a + per-cell `FRAME_ACTIVITY_HOME`. +- **Validity** — a cell is invalid, and excluded from comparisons, when: + - its setup failed; + - a hooked arm's hook never ran although the agent searched; + - the no-hint arm recorded hook activity. +- **Scoring** (`score.js`): + - expected files found (read or changed); + - search calls and read calls; + - input tokens including cache creation and cache reads, output tokens separately; + - elapsed time, failures. + + Tokens come only from the final `result` event; a run without one is + `unknown`, not 0. Arms are compared per task (repeats averaged first): mean + difference, and how many tasks went down / up. + +```bash +node scripts/eval/run-eval.js --retrieval-arms --repeat 5 --seed 7 # 12 × 3 × 5 = 180 agent runs +node scripts/eval/score.js scripts/eval/results/ +``` + +**Status: not run yet.** No token saving is claimed for v2. A hint firing +does not show that work was saved; only this paired measurement can. diff --git a/scripts/eval/run-eval.js b/scripts/eval/run-eval.js index 9bd2ddfe..7eee786d 100644 --- a/scripts/eval/run-eval.js +++ b/scripts/eval/run-eval.js @@ -15,6 +15,14 @@ * node scripts/eval/run-eval.js --timeout 600 # seconds per run (default 600) * node scripts/eval/run-eval.js --out # results dir override * node scripts/eval/run-eval.js --hooks # frame arm runs with spec-knowledge hooks (injected-vs-not comparison) + * node scripts/eval/run-eval.js --retrieval-arms # STR-03: tasks.json retrievalSuite × no-hint | legacy | v2 + * node scripts/eval/run-eval.js --retrieval-arms --repeat 5 --seed 7 # repetitions, shuffled order + * + * Retrieval arms share everything — worktree, pinned commit, the map built + * by this checkout's update-structure.js, prompt, model, permissions — except + * the search hook: none, or this checkout's module-hint.js with the legacy or + * v2 engine. Each cell's hook activity goes to its own FRAME_ACTIVITY_HOME so + * score.js can tell whether the hook ran as the arm intends. * * Agent CLI is configurable so other tools can slot in: * FRAME_EVAL_AGENT binary (default: claude) @@ -63,10 +71,88 @@ function parseArgs() { // --hooks: frame-arm worktrees get the spec-knowledge hooks // (.claude/settings.json + freshly built index + hint scripts), so runs // measure injected vs non-injected agent behavior. Bare arm never hooks. - hooks: args.includes('--hooks') + hooks: args.includes('--hooks'), + retrievalArms: args.includes('--retrieval-arms'), + repeat: Math.max(1, Number(get('--repeat')) || 1), + seed: Number(get('--seed')) || Date.now() % 100000 }; } +const RETRIEVAL_ARMS = ['no-hint', 'legacy', 'v2']; + +/** Deterministic shuffle (mulberry32) so a run order can be reproduced. */ +function shuffled(items, seed) { + let t = seed >>> 0; + const rand = () => { + t = (t + 0x6D2B79F5) >>> 0; + let r = Math.imul(t ^ (t >>> 15), 1 | t); + r ^= r + Math.imul(r ^ (r >>> 7), 61 | r); + return ((r ^ (r >>> 14)) >>> 0) / 4294967296; + }; + const out = [...items]; + for (let i = out.length - 1; i > 0; i--) { + const j = Math.floor(rand() * (i + 1)); + [out[i], out[j]] = [out[j], out[i]]; + } + return out; +} + +/** + * Retrieval-arm setup: the same map for every arm (this checkout's + * update-structure.js, which also publishes lookup.json), then the arm's + * search hook and engine. Returns false when anything failed — the cell is + * then recorded as invalid rather than silently degraded. + */ +function setupRetrievalArm(wt, arm, activityHome) { + try { + const built = spawnSync('node', [path.join(ROOT_DIR, 'scripts', 'update-structure.js'), '--full'], { + cwd: wt, encoding: 'utf-8', timeout: 120000, env: { ...process.env, FRAME_PROJECT_ROOT: wt } + }); + if (built.status !== 0) return false; + if (arm === 'no-hint') return true; + const configFile = path.join(wt, '.frame', 'config.json'); + let config = {}; + try { config = JSON.parse(fs.readFileSync(configFile, 'utf-8')); } catch (e) { /* none */ } + config.project = { ...(config.project || {}), retrieval: { engine: arm } }; + fs.writeFileSync(configFile, JSON.stringify(config, null, 2) + '\n'); + const hook = `FRAME_ACTIVITY_HOME=${JSON.stringify(activityHome)} node ${JSON.stringify(path.join(ROOT_DIR, 'scripts', 'module-hint.js'))} search`; + fs.mkdirSync(path.join(wt, '.claude'), { recursive: true }); + fs.writeFileSync(path.join(wt, '.claude', 'settings.json'), JSON.stringify({ + hooks: { PreToolUse: [{ matcher: 'Grep|Glob|Bash', hooks: [{ type: 'command', command: hook }] }] } + }, null, 2) + '\n'); + return true; + } catch (e) { + console.warn(` (retrieval arm setup failed: ${e.message})`); + return false; + } +} + +/** Search-hook records a cell produced: { records, injected }. */ +function hookActivity(activityHome) { + const out = { records: 0, injected: 0 }; + const walk = (dir) => { + let entries = []; + try { entries = fs.readdirSync(dir, { withFileTypes: true }); } catch (e) { return; } + for (const e of entries) { + const p = path.join(dir, e.name); + if (e.isDirectory()) walk(p); + else if (e.name.endsWith('.jsonl')) { + for (const line of fs.readFileSync(p, 'utf-8').split('\n')) { + if (!line.trim()) continue; + try { + const r = JSON.parse(line); + if (r.mode !== 'search' || !/^hint\./.test(r.ev)) continue; + out.records++; + if (r.ev === 'hint.injected') out.injected++; + } catch (err) { /* partial line */ } + } + } + } + }; + walk(activityHome); + return out; +} + function git(cmd, cwd) { return execSync(cmd, { cwd: cwd || ROOT_DIR, encoding: 'utf-8', maxBuffer: 64 * 1024 * 1024 }); } @@ -102,15 +188,24 @@ function setupHooks(wt) { } } -function runOne(task, arm, resultsDir, timeoutSec, hooks) { - const runDir = path.join(resultsDir, `${task.id}--${arm}`); +function runOne(task, arm, resultsDir, timeoutSec, hooks, suite = SUITE, repeat = 1) { + const retrievalArm = RETRIEVAL_ARMS.includes(arm); + const runDir = path.join(resultsDir, retrievalArm ? `${task.id}--${arm}--r${repeat}` : `${task.id}--${arm}`); fs.mkdirSync(runDir, { recursive: true }); + const activityHome = path.join(runDir, 'activity'); const wt = fs.mkdtempSync(path.join(os.tmpdir(), `frame-eval-${task.id}-${arm}-`)); - console.log(`\n▶ ${task.id} [${arm}]`); + console.log(`\n▶ ${task.id} [${arm}]${retrievalArm ? ` #${repeat}` : ''}`); try { - git(`git worktree add --detach "${wt}" ${SUITE.pinnedCommit}`); + git(`git worktree add --detach "${wt}" ${suite.pinnedCommit}`); + + let setupOk = null; + if (retrievalArm) { + setupOk = setupRetrievalArm(wt, arm, activityHome); + git('git add -A', wt); + git(`git -c user.email=eval@frame -c user.name=frame-eval commit -q --no-verify --allow-empty -m "retrieval arm setup (${arm})"`, wt); + } if (arm === 'bare') { for (const file of FRAME_CONTEXT_FILES) { @@ -166,11 +261,13 @@ function runOne(task, arm, resultsDir, timeoutSec, hooks) { checkPassed = false; } + const activity = retrievalArm ? hookActivity(activityHome) : null; const meta = { task: task.id, arm, hooksActive, - pinnedCommit: SUITE.pinnedCommit, + ...(retrievalArm ? { retrievalArm: true, repeat, setupOk, hookRecords: activity.records, hintsInjected: activity.injected, worktree: wt } : {}), + pinnedCommit: suite.pinnedCommit, agent: `${AGENT_CMD} ${AGENT_ARGS.join(' ')}`, exitCode: result.status, timedOut: Boolean(timedOut), @@ -192,33 +289,40 @@ function runOne(task, arm, resultsDir, timeoutSec, hooks) { function main() { const opts = parseArgs(); + const suite = opts.retrievalArms ? SUITE.retrievalSuite : SUITE; const tasks = opts.task - ? SUITE.tasks.filter(t => t.id === opts.task) - : SUITE.tasks; + ? suite.tasks.filter(t => t.id === opts.task) + : suite.tasks; if (tasks.length === 0) { - console.error(`No task matches "${opts.task}". Available: ${SUITE.tasks.map(t => t.id).join(', ')}`); + console.error(`No task matches "${opts.task}". Available: ${suite.tasks.map(t => t.id).join(', ')}`); process.exit(1); } - const arms = opts.arm ? [opts.arm] : ['frame', 'bare']; + const arms = opts.arm ? [opts.arm] : (opts.retrievalArms ? RETRIEVAL_ARMS : ['frame', 'bare']); const stamp = new Date().toISOString().replace(/[:.]/g, '-').slice(0, 19); const resultsDir = opts.out || path.join(__dirname, 'results', `run-${stamp}`); fs.mkdirSync(resultsDir, { recursive: true }); - console.log(`Suite: ${tasks.length} task(s) × ${arms.length} arm(s) @ ${SUITE.pinnedCommit.slice(0, 7)}`); + console.log(`Suite: ${tasks.length} task(s) × ${arms.length} arm(s)${opts.retrievalArms ? ` × ${opts.repeat} repeat(s), seed ${opts.seed}` : ''} @ ${suite.pinnedCommit.slice(0, 7)}`); console.log(`Agent: ${AGENT_CMD} ${AGENT_ARGS.join(' ')}`); console.log(`Results: ${path.relative(ROOT_DIR, resultsDir)}`); - const all = []; + let cells = []; for (const task of tasks) { for (const arm of arms) { - try { - all.push(runOne(task, arm, resultsDir, opts.timeoutSec, opts.hooks)); - } catch (e) { - console.error(` ✗ ${task.id} [${arm}] crashed: ${e.message}`); - all.push({ task: task.id, arm, crashed: true, error: e.message }); - } + for (let r = 1; r <= (opts.retrievalArms ? opts.repeat : 1); r++) cells.push({ task, arm, repeat: r }); + } + } + if (opts.retrievalArms) cells = shuffled(cells, opts.seed); + + const all = []; + for (const { task, arm, repeat } of cells) { + try { + all.push(runOne(task, arm, resultsDir, opts.timeoutSec, opts.hooks, suite, repeat)); + } catch (e) { + console.error(` ✗ ${task.id} [${arm}] crashed: ${e.message}`); + all.push({ task: task.id, arm, repeat, crashed: true, error: e.message }); } } @@ -226,4 +330,6 @@ function main() { console.log(`\nDone. Score with: node scripts/eval/score.js ${path.relative(ROOT_DIR, resultsDir)}`); } -main(); +if (require.main === module) main(); + +module.exports = { shuffled, hookActivity, RETRIEVAL_ARMS }; diff --git a/scripts/eval/score.js b/scripts/eval/score.js index 2d6b338b..26c53abb 100644 --- a/scripts/eval/score.js +++ b/scripts/eval/score.js @@ -12,6 +12,17 @@ * first Edit/Write * - turns, tokens, duration — from the transcript * + * Retrieval arms (STR-03, `run-eval.js --retrieval-arms`) add: + * - search and read calls in total, the files read, and which expected + * files were found (read or changed) + * - input tokens including cache creation and cache reads, output tokens + * separately — taken from the final `result` event only, never summed + * per message (that would count the context again on every turn); + * a transcript without one reports `null` (unknown), never 0 + * - cell validity: a hooked arm whose hook never ran although the agent + * searched, or a no-hint arm that recorded hook activity, is invalid + * - paired per-task differences between arms + * * Usage: * node scripts/eval/score.js # summary table * node scripts/eval/score.js --json # machine-readable @@ -25,9 +36,20 @@ const EDIT_TOOLS = new Set(['Edit', 'Write', 'MultiEdit', 'NotebookEdit']); const SEARCH_TOOLS = new Set(['Grep', 'Glob']); const SEARCHY_BASH = /\b(grep|rg|find|fd|ag)\b/; -function scoreTranscript(file) { - const stats = { searchBeforeFirstEdit: 0, toolCalls: 0, turns: 0, inputTokens: 0, outputTokens: 0 }; +const READ_TOOLS = new Set(['Read', 'NotebookRead']); +const READY_BASH = /^\s*(?:cat|head|tail|less|sed\s+-n)\b/; + +function scoreTranscript(file, options = {}) { + const stats = { + searchBeforeFirstEdit: 0, toolCalls: 0, turns: 0, searchCalls: 0, readCalls: 0, filesRead: [], + inputTokens: null, cacheCreationTokens: null, cacheReadTokens: null, totalInputTokens: null, outputTokens: null + }; if (!fs.existsSync(file)) return stats; + const rel = (p) => { + const text = String(p || ''); + if (options.worktree && text.startsWith(options.worktree + path.sep)) return text.slice(options.worktree.length + 1).split(path.sep).join('/'); + return text.replace(/^\.\//, ''); + }; let sawEdit = false; for (const line of fs.readFileSync(file, 'utf-8').split('\n')) { @@ -47,17 +69,46 @@ function scoreTranscript(file) { if (isEdit) sawEdit = true; if (isSearch && !sawEdit) stats.searchBeforeFirstEdit++; + if (isSearch) stats.searchCalls++; + const isRead = READ_TOOLS.has(block.name) || + (block.name === 'Bash' && block.input && READY_BASH.test(String(block.input.command || ''))); + if (isRead) { + stats.readCalls++; + const target = block.input && (block.input.file_path || block.input.notebook_path); + if (target && !stats.filesRead.includes(rel(target))) stats.filesRead.push(rel(target)); + } } } + // The final result carries the run's cumulative usage; per-message usage + // repeats the context every turn and is deliberately not summed. if (event.type === 'result' && event.usage) { - stats.inputTokens = event.usage.input_tokens || 0; - stats.outputTokens = event.usage.output_tokens || 0; + const u = event.usage; + const num = (v) => (typeof v === 'number' ? v : null); + stats.inputTokens = num(u.input_tokens); + stats.cacheCreationTokens = num(u.cache_creation_input_tokens); + stats.cacheReadTokens = num(u.cache_read_input_tokens); + stats.outputTokens = num(u.output_tokens); + const parts = [stats.inputTokens, stats.cacheCreationTokens, stats.cacheReadTokens].filter((v) => v !== null); + stats.totalInputTokens = parts.length ? parts.reduce((a, b) => a + b, 0) : null; } } return stats; } +/** + * Whether a retrieval-arm cell measured what its arm intends. Returns + * { valid, reason? }. Cells of other suites are always valid. + */ +function cellValidity(meta, stats) { + if (!meta.retrievalArm) return { valid: true }; + if (meta.setupOk === false) return { valid: false, reason: 'setup-failed' }; + const records = meta.hookRecords || 0; + if (meta.arm === 'no-hint') return records === 0 ? { valid: true } : { valid: false, reason: 'hook-ran-in-no-hint-arm' }; + if (stats.searchCalls > 0 && records === 0) return { valid: false, reason: 'hook-never-ran' }; + return { valid: true }; +} + function scoreRun(runDir) { const metaPath = path.join(runDir, 'meta.json'); if (!fs.existsSync(metaPath)) return null; @@ -67,15 +118,51 @@ function scoreRun(runDir) { const wrongFiles = (meta.changedFiles || []) .filter(f => !expected.has(f) && !META_FILES.has(f)); + const stats = scoreTranscript(path.join(runDir, 'transcript.jsonl'), { worktree: meta.worktree }); + const touched = new Set([...stats.filesRead, ...(meta.changedFiles || [])]); return { task: meta.task, arm: meta.arm, + repeat: meta.repeat || 1, pass: Boolean(meta.checkPassed) && !meta.timedOut, timedOut: Boolean(meta.timedOut), + crashed: Boolean(meta.crashed), wrongFiles, changedCount: (meta.changedFiles || []).length, durationMs: meta.durationMs || 0, - ...scoreTranscript(path.join(runDir, 'transcript.jsonl')) + filesFound: (meta.expectedFiles || []).filter((f) => touched.has(f)).length, + expectedCount: (meta.expectedFiles || []).length, + hintsInjected: meta.hintsInjected || 0, + ...cellValidity(meta, stats), + ...stats + }; +} + +/** + * Paired per-task comparison of arm `b` against arm `a` on valid cells: + * mean of the per-task mean differences (b − a), and how many tasks went + * down / up / unchanged. A metric unknown in either arm drops that task. + */ +function paired(runs, a, b, metric) { + const byTask = new Map(); + for (const r of runs) { + if (!r.valid || (r.arm !== a && r.arm !== b)) continue; + const t = byTask.get(r.task) || { [a]: [], [b]: [] }; + if (r[metric] !== null && r[metric] !== undefined) t[r.arm].push(r[metric]); + byTask.set(r.task, t); + } + const diffs = []; + for (const t of byTask.values()) { + if (!t[a].length || !t[b].length) continue; + const mean = (xs) => xs.reduce((x, y) => x + y, 0) / xs.length; + diffs.push(mean(t[b]) - mean(t[a])); + } + return { + tasks: diffs.length, + meanDiff: diffs.length ? diffs.reduce((x, y) => x + y, 0) / diffs.length : null, + lower: diffs.filter((d) => d < 0).length, + higher: diffs.filter((d) => d > 0).length, + same: diffs.filter((d) => d === 0).length }; } @@ -83,7 +170,20 @@ function aggregate(runs) { const n = runs.length; const sum = (fn) => runs.reduce((a, r) => a + fn(r), 0); const avg = (fn) => n ? sum(fn) / n : 0; + const known = (key) => runs.filter((r) => r[key] !== null && r[key] !== undefined); + const avgKnown = (key) => { + const k = known(key); + return k.length ? k.reduce((a, r) => a + r[key], 0) / k.length : null; + }; return { + invalid: runs.filter((r) => r.valid === false).length, + avgSearchCalls: avg((r) => r.searchCalls || 0), + avgReadCalls: avg((r) => r.readCalls || 0), + filesFound: sum((r) => r.filesFound || 0), + filesExpected: sum((r) => r.expectedCount || 0), + avgTotalInputTokens: avgKnown('totalInputTokens'), + avgOutputTokens: avgKnown('outputTokens'), + tokensUnknown: n - known('totalInputTokens').length, tasks: n, passed: runs.filter(r => r.pass).length, passRate: n ? runs.filter(r => r.pass).length / n : 0, @@ -93,7 +193,7 @@ function aggregate(runs) { avgToolCalls: avg(r => r.toolCalls), avgTurns: avg(r => r.turns), avgDurationSec: avg(r => r.durationMs / 1000), - totalOutputTokens: sum(r => r.outputTokens) + totalOutputTokens: sum(r => r.outputTokens || 0) }; } @@ -124,8 +224,17 @@ function main() { summary[arm] = aggregate(armRuns); } + const retrievalArms = runs.some((r) => ['no-hint', 'legacy', 'v2'].includes(r.arm)); + const comparisons = {}; + if (retrievalArms) { + for (const [a, b] of [['no-hint', 'legacy'], ['no-hint', 'v2'], ['legacy', 'v2']]) { + comparisons[`${b} vs ${a}`] = Object.fromEntries(['totalInputTokens', 'searchCalls', 'readCalls', 'durationMs', 'filesFound'] + .map((m) => [m, paired(runs, a, b, m)])); + } + } + if (args.includes('--json')) { - console.log(JSON.stringify({ summary, runs }, null, 2)); + console.log(JSON.stringify({ summary, comparisons, runs }, null, 2)); return; } @@ -140,7 +249,11 @@ function main() { ['avg tool calls', ...arms.map(a => summary[a].avgToolCalls.toFixed(1))], ['avg turns', ...arms.map(a => summary[a].avgTurns.toFixed(1))], ['avg duration (s)', ...arms.map(a => summary[a].avgDurationSec.toFixed(0))], - ['total output tokens', ...arms.map(a => summary[a].totalOutputTokens)] + ['total output tokens', ...arms.map(a => summary[a].totalOutputTokens)], + ['avg input tokens (incl. cache)', ...arms.map(a => summary[a].avgTotalInputTokens === null ? 'unknown' : summary[a].avgTotalInputTokens.toFixed(0))], + ['avg search / read calls', ...arms.map(a => `${summary[a].avgSearchCalls.toFixed(1)} / ${summary[a].avgReadCalls.toFixed(1)}`)], + ['expected files found', ...arms.map(a => `${summary[a].filesFound}/${summary[a].filesExpected}`)], + ['invalid cells', ...arms.map(a => summary[a].invalid)] ]; const widths = rows[0].map((_, i) => Math.max(...rows.map(r => String(r[i]).length))); @@ -149,6 +262,14 @@ function main() { if (ri === 0) console.log(widths.map(w => '-'.repeat(w + 2)).join('')); } + for (const [name, metrics] of Object.entries(comparisons)) { + console.log(`\nPaired ${name} (per task, valid cells):`); + for (const [metric, p] of Object.entries(metrics)) { + if (!p.tasks) continue; + console.log(` ${metric.padEnd(18)} mean diff ${p.meanDiff.toFixed(1)} · lower ${p.lower} / higher ${p.higher} / same ${p.same} (n=${p.tasks})`); + } + } + // Per-task detail for anything that failed or edited wrong files const problems = runs.filter(r => !r.pass || r.wrongFiles.length > 0); if (problems.length > 0) { @@ -162,4 +283,6 @@ function main() { } } -main(); +if (require.main === module) main(); + +module.exports = { scoreTranscript, scoreRun, cellValidity, paired, aggregate }; diff --git a/scripts/eval/tasks.json b/scripts/eval/tasks.json index d3b85d66..c7e40355 100644 --- a/scripts/eval/tasks.json +++ b/scripts/eval/tasks.json @@ -62,5 +62,107 @@ "expectedFiles": ["scripts/update-structure.js"], "successCheck": "grep -q 'test\\.js' scripts/update-structure.js && node -e \"const fs=require('fs');fs.mkdirSync('src/tmp-eval',{recursive:true});fs.writeFileSync('src/tmp-eval/foo.test.js','function fooBarEval(){}\\n');\" && node scripts/update-structure.js >/dev/null 2>&1 && node -e \"const s=JSON.parse(require('fs').readFileSync('STRUCTURE.json','utf-8'));process.exit(s.modules && s.modules['tmp-eval/foo.test'] ? 1 : 0)\"" } - ] + ], + "retrievalSuite": { + "_comment": "STR-03 matched-agent navigation suite (run-eval.js --retrieval-arms). Each prompt describes a file's behavior, never its name; the task is done when the marker line lands in that file and nowhere else. Arms differ only in the search hook (none / legacy / v2). Pinned to the retrieval corpus commit.", + "pinnedCommit": "262f91bbc584e21fe464fd9adceba7a686ba6160", + "tasks": [ + { + "id": "nav-01", + "prompt": "Find the file that implements the sample project's Google sign-in: the Passport strategy that finds or creates the user and add this exact line as a comment near the top of that file (after any shebang or opening block comment is fine): // eval-nav: nav-01. Change nothing else and edit no other file.", + "expectedFiles": [ + "src/templates/sample-project/src/auth/oauth.js" + ], + "successCheck": "grep -q 'eval-nav: nav-01' src/templates/sample-project/src/auth/oauth.js && [ \"$(git grep -l 'eval-nav: nav-01' | wc -l | tr -d ' ')\" = 1 ]" + }, + { + "id": "nav-02", + "prompt": "Find the file that implements the shared rule that decides whether a string is a name Git accepts for a branch and add this exact line as a comment near the top of that file (after any shebang or opening block comment is fine): // eval-nav: nav-02. Change nothing else and edit no other file.", + "expectedFiles": [ + "src/shared/gitRefNames.js" + ], + "successCheck": "grep -q 'eval-nav: nav-02' src/shared/gitRefNames.js && [ \"$(git grep -l 'eval-nav: nav-02' | wc -l | tr -d ' ')\" = 1 ]" + }, + { + "id": "nav-03", + "prompt": "Find the file that implements the main-process helper that writes state files atomically and recovers a corrupted JSON state file and add this exact line as a comment near the top of that file (after any shebang or opening block comment is fine): // eval-nav: nav-03. Change nothing else and edit no other file.", + "expectedFiles": [ + "src/main/fsSafe.js" + ], + "successCheck": "grep -q 'eval-nav: nav-03' src/main/fsSafe.js && [ \"$(git grep -l 'eval-nav: nav-03' | wc -l | tr -d ' ')\" = 1 ]" + }, + { + "id": "nav-04", + "prompt": "Find the file that implements the pure module that defines the interface zoom steps and the factor and label of each and add this exact line as a comment near the top of that file (after any shebang or opening block comment is fine): // eval-nav: nav-04. Change nothing else and edit no other file.", + "expectedFiles": [ + "src/shared/uiZoom.js" + ], + "successCheck": "grep -q 'eval-nav: nav-04' src/shared/uiZoom.js && [ \"$(git grep -l 'eval-nav: nav-04' | wc -l | tr -d ' ')\" = 1 ]" + }, + { + "id": "nav-05", + "prompt": "Find the file that implements the module that works out which PATH a child process should run with, by asking the user's login shell and add this exact line as a comment near the top of that file (after any shebang or opening block comment is fine): // eval-nav: nav-05. Change nothing else and edit no other file.", + "expectedFiles": [ + "src/main/envPath.js" + ], + "successCheck": "grep -q 'eval-nav: nav-05' src/main/envPath.js && [ \"$(git grep -l 'eval-nav: nav-05' | wc -l | tr -d ' ')\" = 1 ]" + }, + { + "id": "nav-06", + "prompt": "Find the file that implements the script that builds the module map for a commit from Git's staged snapshot and publishes it into the index and add this exact line as a comment near the top of that file (after any shebang or opening block comment is fine): // eval-nav: nav-06. Change nothing else and edit no other file.", + "expectedFiles": [ + "scripts/structure-commit.js" + ], + "successCheck": "grep -q 'eval-nav: nav-06' scripts/structure-commit.js && [ \"$(git grep -l 'eval-nav: nav-06' | wc -l | tr -d ' ')\" = 1 ]" + }, + { + "id": "nav-07", + "prompt": "Find the file that implements the shared helper that masks secrets before the app logs or records anything and add this exact line as a comment near the top of that file (after any shebang or opening block comment is fine): // eval-nav: nav-07. Change nothing else and edit no other file.", + "expectedFiles": [ + "scripts/redact.js" + ], + "successCheck": "grep -q 'eval-nav: nav-07' scripts/redact.js && [ \"$(git grep -l 'eval-nav: nav-07' | wc -l | tr -d ' ')\" = 1 ]" + }, + { + "id": "nav-08", + "prompt": "Find the file that implements the pure list model behind the status bar's branch picker (sorting, matching, worktree rows) and add this exact line as a comment near the top of that file (after any shebang or opening block comment is fine): // eval-nav: nav-08. Change nothing else and edit no other file.", + "expectedFiles": [ + "src/renderer/statusBar/branchPickerModel.js" + ], + "successCheck": "grep -q 'eval-nav: nav-08' src/renderer/statusBar/branchPickerModel.js && [ \"$(git grep -l 'eval-nav: nav-08' | wc -l | tr -d ' ')\" = 1 ]" + }, + { + "id": "nav-09", + "prompt": "Find the file that implements the main-process module that checks whether a newer version of the app has been released and add this exact line as a comment near the top of that file (after any shebang or opening block comment is fine): // eval-nav: nav-09. Change nothing else and edit no other file.", + "expectedFiles": [ + "src/main/updateChecker.js" + ], + "successCheck": "grep -q 'eval-nav: nav-09' src/main/updateChecker.js && [ \"$(git grep -l 'eval-nav: nav-09' | wc -l | tr -d ' ')\" = 1 ]" + }, + { + "id": "nav-10", + "prompt": "Find the file that implements the renderer module that supplies projects, terminals and specs as items to the command palette and add this exact line as a comment near the top of that file (after any shebang or opening block comment is fine): // eval-nav: nav-10. Change nothing else and edit no other file.", + "expectedFiles": [ + "src/renderer/paletteSources.js" + ], + "successCheck": "grep -q 'eval-nav: nav-10' src/renderer/paletteSources.js && [ \"$(git grep -l 'eval-nav: nav-10' | wc -l | tr -d ' ')\" = 1 ]" + }, + { + "id": "nav-11", + "prompt": "Find the file that implements the module that decides which widgets the Home board shows and in what order and add this exact line as a comment near the top of that file (after any shebang or opening block comment is fine): // eval-nav: nav-11. Change nothing else and edit no other file.", + "expectedFiles": [ + "src/renderer/home/registry.js" + ], + "successCheck": "grep -q 'eval-nav: nav-11' src/renderer/home/registry.js && [ \"$(git grep -l 'eval-nav: nav-11' | wc -l | tr -d ' ')\" = 1 ]" + }, + { + "id": "nav-12", + "prompt": "Find the file that implements the pure module that checks whether Frame's own documentation still describes the project correctly and add this exact line as a comment near the top of that file (after any shebang or opening block comment is fine): // eval-nav: nav-12. Change nothing else and edit no other file.", + "expectedFiles": [ + "src/shared/docsHealth.js" + ], + "successCheck": "grep -q 'eval-nav: nav-12' src/shared/docsHealth.js && [ \"$(git grep -l 'eval-nav: nav-12' | wc -l | tr -d ' ')\" = 1 ]" + } + ] + } } diff --git a/test/retrievalEval.test.js b/test/retrievalEval.test.js index 8156ca1b..80c4d16a 100644 --- a/test/retrievalEval.test.js +++ b/test/retrievalEval.test.js @@ -143,3 +143,115 @@ test('quantiles use the nearest-rank definition', () => { assert.equal(bench.quantile([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 0.95), 10); assert.equal(bench.quantile([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 0.5), 5); }); + +/* ------------------------- matched-agent instrument (S8) ------------------------- */ + +const fs = require('fs'); +const os = require('os'); +const score = require('../scripts/eval/score'); +const runEval = require('../scripts/eval/run-eval'); + +function cell(t, { meta, events }) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'frame-cell-')); + t.after(() => fs.rmSync(dir, { recursive: true, force: true })); + fs.writeFileSync(path.join(dir, 'meta.json'), JSON.stringify(meta)); + fs.writeFileSync(path.join(dir, 'transcript.jsonl'), events.map((e) => JSON.stringify(e)).join('\n')); + return dir; +} +const tool = (name, input) => ({ type: 'assistant', message: { content: [{ type: 'tool_use', name, input }], usage: { input_tokens: 999999 } } }); + +test('scoring counts searches, reads and found files; tokens come from the final result only, cache included', (t) => { + const wt = '/tmp/frame-eval-x'; + const dir = cell(t, { + meta: { task: 'nav-a', arm: 'v2', retrievalArm: true, setupOk: true, hookRecords: 2, hintsInjected: 1, worktree: wt, expectedFiles: ['src/a.js', 'src/b.js'], changedFiles: ['src/b.js'], checkPassed: true, durationMs: 5000 }, + events: [ + tool('Grep', { pattern: 'alpha' }), + tool('Bash', { command: 'rg -n beta src/' }), + tool('Read', { file_path: `${wt}/src/a.js` }), + tool('Bash', { command: 'cat src/c.js' }), + tool('Edit', { file_path: `${wt}/src/b.js` }), + tool('Glob', { pattern: '**/*.md' }), + { type: 'result', usage: { input_tokens: 120, cache_creation_input_tokens: 3000, cache_read_input_tokens: 45000, output_tokens: 800 } } + ] + }); + const r = score.scoreRun(dir); + assert.equal(r.searchCalls, 3); + assert.equal(r.searchBeforeFirstEdit, 2); + assert.equal(r.readCalls, 2); + assert.deepEqual(r.filesRead, ['src/a.js']); + assert.equal(r.filesFound, 2, 'read a.js, changed b.js'); + assert.equal(r.totalInputTokens, 48120, 'never the per-message usage'); + assert.equal(r.outputTokens, 800); + assert.equal(r.valid, true); +}); + +test('missing token telemetry is unknown, not zero, and stays out of averages', (t) => { + const known = score.scoreRun(cell(t, { meta: { task: 'x', arm: 'legacy', expectedFiles: [] }, events: [{ type: 'result', usage: { input_tokens: 100, output_tokens: 10 } }] })); + const unknown = score.scoreRun(cell(t, { meta: { task: 'x', arm: 'legacy', expectedFiles: [] }, events: [tool('Grep', { pattern: 'x' })] })); + assert.equal(unknown.totalInputTokens, null); + assert.equal(unknown.outputTokens, null); + const agg = score.aggregate([known, unknown]); + assert.equal(agg.avgTotalInputTokens, 100); + assert.equal(agg.tokensUnknown, 1); +}); + +test('a cell is invalid when its hook did not run as the arm intends', () => { + const v = (meta, stats = { searchCalls: 1 }) => score.cellValidity({ retrievalArm: true, setupOk: true, ...meta }, stats); + assert.deepEqual(v({ arm: 'v2', hookRecords: 0 }), { valid: false, reason: 'hook-never-ran' }); + assert.deepEqual(v({ arm: 'v2', hookRecords: 0 }, { searchCalls: 0 }), { valid: true }, 'no search, nothing to hint'); + assert.deepEqual(v({ arm: 'legacy', hookRecords: 3 }), { valid: true }); + assert.deepEqual(v({ arm: 'no-hint', hookRecords: 1 }), { valid: false, reason: 'hook-ran-in-no-hint-arm' }); + assert.deepEqual(v({ arm: 'no-hint', hookRecords: 0 }), { valid: true }); + assert.deepEqual(v({ arm: 'v2', setupOk: false, hookRecords: 5 }), { valid: false, reason: 'setup-failed' }); + assert.deepEqual(score.cellValidity({ arm: 'frame' }, { searchCalls: 9 }), { valid: true }, 'other suites are unaffected'); +}); + +test('paired comparison averages repeats per task, uses valid cells only, and counts direction', () => { + const run = (task, arm, totalInputTokens, valid = true) => ({ task, arm, totalInputTokens, valid }); + const runs = [ + run('a', 'legacy', 100), run('a', 'legacy', 140), run('a', 'v2', 90), run('a', 'v2', 110), + run('b', 'legacy', 200), run('b', 'v2', 260), + run('c', 'legacy', 50), run('c', 'v2', 10, false), + run('d', 'legacy', null), run('d', 'v2', 70) + ]; + const p = score.paired(runs, 'legacy', 'v2', 'totalInputTokens'); + assert.equal(p.tasks, 2, 'c has no valid v2 cell, d has no known legacy value'); + assert.equal(p.meanDiff, ((100 - 120) + (260 - 200)) / 2); + assert.deepEqual([p.lower, p.higher, p.same], [1, 1, 0]); +}); + +test('the navigation suite: at least 12 tasks, files named only by behavior, checks bound to the expected file', (t) => { + const suite = require('../scripts/eval/tasks.json').retrievalSuite; + assert.ok(suite.tasks.length >= 12); + const r = spawnSync('git', ['ls-tree', '-r', '--name-only', suite.pinnedCommit], { cwd: REPO_ROOT, encoding: 'utf8', maxBuffer: 64 * 1024 * 1024 }); + const files = r.status === 0 ? new Set(r.stdout.split('\n')) : null; + for (const task of suite.tasks) { + assert.equal(task.expectedFiles.length, 1); + const file = task.expectedFiles[0]; + const stem = path.basename(file).replace(/\.[^.]+$/, ''); + assert.ok(!task.prompt.toLowerCase().includes(stem.toLowerCase()), `${task.id}: the prompt must not name ${stem}`); + assert.ok(task.successCheck.includes(file) && task.successCheck.includes(`eval-nav: ${task.id}`), task.id); + if (files) assert.ok(files.has(file), `${task.id}: ${file} at the pinned commit`); + } + if (!files) t.diagnostic('pinned commit unavailable; file existence not checked'); +}); + +test('cell order is shuffled reproducibly from the seed', () => { + const cells = Array.from({ length: 30 }, (_, i) => i); + const a = runEval.shuffled(cells, 7); + assert.deepEqual(runEval.shuffled(cells, 7), a); + assert.notDeepEqual(a, cells); + assert.deepEqual([...a].sort((x, y) => x - y), cells); + assert.deepEqual(runEval.RETRIEVAL_ARMS, ['no-hint', 'legacy', 'v2']); +}); + +test('hook activity counts only search-hint records', (t) => { + const home = fs.mkdtempSync(path.join(os.tmpdir(), 'frame-act-')); + t.after(() => fs.rmSync(home, { recursive: true, force: true })); + fs.mkdirSync(path.join(home, 'p'), { recursive: true }); + fs.writeFileSync(path.join(home, 'p', 'a.jsonl'), [ + { ev: 'hint.injected', mode: 'search' }, { ev: 'hint.quiet', mode: 'search' }, + { ev: 'hint.injected', mode: 'pre-edit' }, { ev: 'watch.fired' } + ].map((r) => JSON.stringify(r)).join('\n') + '\n{"partial'); + assert.deepEqual(runEval.hookActivity(home), { records: 2, injected: 1 }); +}); From 8c45290a8303888533932c2ed183c1b37f0d50d4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kaan=20=C3=96zhan?= Date: Thu, 1 Oct 2026 16:33:00 +0300 Subject: [PATCH 9/9] docs(spec): close STR-03 with its digest --- .../specs/str-03-local-file-retrieval/digest.md | 17 +++++++++++++++++ .../str-03-local-file-retrieval/status.json | 6 +++--- 2 files changed, 20 insertions(+), 3 deletions(-) create mode 100644 .frame/specs/str-03-local-file-retrieval/digest.md diff --git a/.frame/specs/str-03-local-file-retrieval/digest.md b/.frame/specs/str-03-local-file-retrieval/digest.md new file mode 100644 index 00000000..52154f37 --- /dev/null +++ b/.frame/specs/str-03-local-file-retrieval/digest.md @@ -0,0 +1,17 @@ +--- +keywords: STR, retrieval, find-module, module-hint, search hint, lookup index, benchmark, eval, Turkish +related: str-02c-working-view-in-runtime, str-02-structure-lifecycle, str-04-jev-evaluation, audit-q3-core-value-efficacy, codex-parity +--- +One retrieval engine (`structure-retrieval.js`) serves find-module and the +search hook. Evidence tiers: path, concept, synonym, file name, symbol and +partial concept (hooks stop here), then path and description words (CLI only). +Every identifier word must be explained; non-ASCII words count as prose. +The worker publishes a compact `lookup.json` beside the live view (130 KB +against the 662 KB map). Hooks read it, or compile a map of at most 2 MiB. +find-module gained `--json`, `--limit` and `--retrieval`. On a frozen +194-query corpus (held-out run once), v2 beat legacy everywhere but missed 4 +gates (recall 89.4%, hint precision 96.5%, 1 false hint, 10k-file p95 55 ms), +so `legacy` stays the default and v2 is opt-in (`project.retrieval.engine`). +The S8 agent instrument ships; its paid run is pending and no saving is claimed. + +Chain: spec.md → plan.md → tasks.md → outcome.md diff --git a/.frame/specs/str-03-local-file-retrieval/status.json b/.frame/specs/str-03-local-file-retrieval/status.json index 3cb878ee..b6b50833 100644 --- a/.frame/specs/str-03-local-file-retrieval/status.json +++ b/.frame/specs/str-03-local-file-retrieval/status.json @@ -1,7 +1,7 @@ { "slug": "str-03-local-file-retrieval", "title": "STR-03 — Local File Retrieval", - "phase": "implementing", + "phase": "done", "generated_task_ids": [ "task-spec-str-03-local-file-retrieval-T01", "task-spec-str-03-local-file-retrieval-T02", @@ -13,7 +13,7 @@ ], "ai_tool": "claude-code", "created_at": "2026-09-17T23:58:30Z", - "updated_at": "2026-10-01T12:54:14Z", - "last_phase_at": "2026-10-01T12:54:14Z", + "updated_at": "2026-10-01T13:33:00Z", + "last_phase_at": "2026-10-01T13:33:00Z", "implement_mode": "custom" }