diff --git a/CHANGELOG.md b/CHANGELOG.md index 6e2cd5dce..94c68a747 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,803 @@ All notable changes to this project will be documented in this file. Format based on [Keep a Changelog](https://keepachangelog.com/). +## Stabilize timing-sensitive Swift CI regressions (2026-10-25) + +Hosted Tests run [40174238672](https://github.com/bbingz/engram/actions/runs/40174238672) +on documentation head `db601947` failed two existing Swift assertions: the +26 MiB digest regression used 10.197 seconds of wall time against a 10-second +limit, and the launcher test assumed an ordinary sleeper would remain alive +after SIGTERM. Both failure logs are retained; the previous green head does +not establish success for this head. + +Only test code changed. The digest test now measures process CPU time while +retaining its 10-second budget, full-size input and tail-marker assertions. +The stop test uses the existing SIGTERM-ignoring fixture and waits for its +readiness marker before asserting nonblocking return and retained ownership. +A first local class run also reproduced five assertions in two restart tests +whose fixture-start deadlines were only 250-500 ms. Those readiness deadlines +are now 5 seconds; exit-ordering and elapsed-shutdown assertions are unchanged. + +All 70 related tests passed (10 digest and 60 launcher tests). Negative controls +then disabled the ASCII search fast path and inserted a blocking stop wait: +the tests correctly rejected 26.799 seconds of CPU work and 2.002 seconds of +blocking, respectively. Both runtime files were restored byte for byte, and +all 70 tests passed again. Tests used an isolated CFFIXED_USER_HOME and +TEST_RUNNER_CFFIXED_USER_HOME with the Engram scheme and only the two test +classes selected. Swift conventions, invariant ledger and whitespace checks +passed. No product source, configuration or deployed binary changed. + +Durable logs, negative-control script and hashes: +`~/.engram-shadow-core-20260911/hq/state/ci-timing-20261025/`. +Local xcresult bundles remain under `/tmp/engram-ci-timing-20261025/`. +The corrective head still requires hosted CI; the final exact-head result will +be recorded on PR #496 without another documentation-only CI cycle. Active-write +collector performance, full UI acceptance and rollback drills remain unverified. + +## Verify Daily collector quiet-source resource targets (2026-10-25) + +Completed the previously missing 30-minute Daily resource window without +changing code, configuration, source data or services. The deployed collector +was `fb9538a83f5eec95fab12af5cc5da9699ee80d25`, PID 98680, SHA-256 +`38caff9c22b891b4e47034060632b51c2482e567a516bf158ca9a8c3084fcd5b`, with its +existing 10,000 ms poll interval on the arm64 Daily MacBookPro18,4. + +The 07:16-07:46 UTC window contains 61 samples over 1800.000 monotonic seconds. +Independent recomputation gives 34.57 CPU seconds, mean one-core CPU **1.921%** +(target <=2%, PASS) and maximum sampled RSS **58.5 MiB** (target <=150 MiB, +PASS). All 16 roots were converged at every sample; requested/completed revisions, +PID/start time, binary hash and log sizes stayed unchanged. No active scan was +observed. Maximum sampling gap was 30.023 seconds. The 30-second interval CPU +range was 1.133-2.733%, median 1.917%; 37/60 intervals were <=2%. The gate uses +the whole-window average, not a claim that every interval met the target. + +This is a quiet-source receipt: no root revision changes and no new publication +ACKs occurred. It closes the missing steady-state window, not active-write or +startup performance. It does not isolate the contribution of any one earlier +optimization or supersede historical busy-workload failures. Margin below the +CPU target is small; the next performance acceptance is a representative active +source workload, with profiling only if measurements justify a code change. + +Read-only inventory checks found 603 pending publications per replica, all +`privacyWithheld`; no transport-error backlog was found in that snapshot. The +275 dirty locators remain: 14 previously captured Codex files are now absent, +6 uncaptured Kimi files are empty, and 255 Copilot files are present/nonempty +but their deferred entrypoint cause remains unverified. No transcripts or +credential files were read, and no records were cleared to obtain the result. + +Evidence: `~/.engram-shadow-core-20260911/hq/state/collector-resource-20261025/` +contains `samples.jsonl`, `verified-result.json`, observer/recomputation helpers, +`inventory-details.json`, `runtime-settings.json`, and `sha256.json`. The observer +completed exit 0; it used `ps`, query-only SQLite, file sizes and binary hashes, +without process attachment or restarts. No new code tests or deployment were +needed for this acceptance-only change. Full UI and per-source retirement remain +outside this result. + +Also refreshed the previous delivery: App build 1782/PID 80289 and local Service +PID 21750 were live; local IPC and both HQ health endpoints passed. Documentation +head `148bfbb5ecd756dfdbafb56163b79fd0b3369736` reached 14 successful hosted checks, +two conditional skips and CLEAN. PR #496 remains Draft/unmerged. The new receipt +is documentation only and does not change any deployed runtime source. + +## Keep popover date-group identities unique (2026-10-25) + +Recent sessions are ordered by activity, so resumed sessions can produce date +headings A/B/A. Using the heading as the SwiftUI identity duplicated IDs in the +installed App's OSLog. Each nonempty group now uses its first session's stable ID; +activity order, contiguous grouping and date labels are preserved. The grouping +helper is static/internal so tests exercise the actual view's grouping path. + +The interleaved-date regression failed with 2 unique IDs for 3 groups before the +fix. All 46 `HomePopoverActionsTests` then passed, including adjacent grouping, +stable identity after inserting another group, and empty input. Tests ran with +an isolated `CFFIXED_USER_HOME`/`TEST_RUNNER_CFFIXED_USER_HOME`. Evidence: +`/tmp/engram-popover-id-20261025/{red,green}.log` and corresponding `.xcresult` +bundles. Invariants and whitespace checks passed. + +Source `a9e59d50f3ef8563de956836ea9fe608ac0c2776` then passed all 14 executed +hosted checks, with two conditional skips, and PR #496 was CLEAN. Receipts: +[Tests / CI Gate](https://github.com/bbingz/engram/actions/runs/40169215543) and +[CodeQL / CodeQL Gate](https://github.com/bbingz/engram/actions/runs/40169215501). +Hosted UI smoke passed all 14 tests and screenshot comparison; full UI was +conditionally skipped. An isolated real-App A/B/A fixture produced the warning +on build 1779, then none during 92 seconds on the signed build 1782. Local window +automation returned `cgWindowNotFound`; screenshot preflight found no Screen +Recording permission, so no local visual acceptance is claimed. + +Installed `/Applications/Engram.app` 1.0.5 build 1782 at 07:02:31 UTC, after the +exact-source gates completed. Release/arm64 was signed with the existing Daily +Developer ID through a one-shot GUI job, which was unloaded. Full signature, +runtime and timestamp verification passed; all eight code files matched the +build after signature normalization. All 80 installed bundle entries matched +the signed candidate. App PID 80289, installed version, local IPC status and both +HQ health endpoints were read back successfully. Local Service PID 21750 and its +`fb9538a-build1779-signed` release stayed unchanged; Collector/Web were not +upgraded in this App-only follow-up. No duplicate-date warning appeared in the +bounded post-install log check. + +Backup and evidence: `~/.engram-shadow-core-20260911/hq/state/deploy-a9e59d50-20261025/` +contains `Engram-before.app` (build 1779), `ROLLBACK.md`, and `evidence/` +(`installed.json`, `post-install.json`, `pr-before-install.json`, +`signed-content-verification.json`, `ui-runtime-comparison.json`, test/build logs). +No App rollback drill, complete local UI walkthrough or notarization was run. +This was a local install, not a public release. The following closeout changes +are documentation only; deployed App source remains `a9e59d50`. + +## Complete the local App and Service upgrade (2026-10-25) + +Installed and launched `/Applications/Engram.app` 1.0.5 build 1779, with paired +Service `~/.engram-service/releases/fb9538a-build1779-signed`. Runtime source +remains green `fb9538a83f5eec95fab12af5cc5da9699ee80d25`; documentation head +`1747748b` also had 14 successful checks and two conditional skips before this +upgrade. App PID 21827 and system Service PID 21750 were independently read back. + +The first authenticated attempt exposed an ad-hoc/Hardened Runtime framework +loading failure that static signature checks did not catch. Restored the old +Service and verified IPC before continuing. HQ has no signing identity; used +the existing Developer ID identity on Daily through a one-shot GUI launchd job +(SSH signing lacked Keychain context). No key export, Keychain policy change or +library-validation exception. The signing job exited 0 and was unloaded. +Full Developer ID, secure timestamp and deep signature checks passed. Removing +signatures from temporary verification copies proved all eight Mach-O files +unchanged; 45 resources and 18 symlinks also matched the original build. A +deliberately mismatched `--expected-home` then reached the Service's normal +argument error before DB access, proving the framework load on HQ. + +After the second authenticated switch, real local IPC status and all value-band +search probes passed. The installed helper matches the paired Service; the alias +audit table exists and FTS version remains 3. App AX/screenshot verification +confirmed its Welcome window (Step 1 of 5); onboarding and full search-page UI +interaction were not performed. HQ health remained `ok`. + +Refreshed local backup passed quick_check with 60,128 rows. Startup logged 628 +deduplications; read-only backup/live comparison confirmed every missing row was +already `skip`, had a duplicate path in the backup, and retained that path in the +live DB. No non-skip rows disappeared. Old App, old Service, rejected ad-hoc +candidate, refreshed DB backup and rollback instructions remain under the +existing deploy backup root. No database restore or App rollback drill occurred. + +Evidence: `~/.engram-shadow-core-20260911/hq/state/deploy-fb9538a-20261024/evidence/` +(`local-installed-20261025.json`, `signed-content-verification.json`, +`verify-signed-app-20261025.log`, `signed-service-launch-smoke.json`, +`local-ipc-after.jsonl`, `local-session-dedup-check.json`, `app-ui-smoke-20261025.json`). +Runtime code was not changed. Full UI, live Keychain business flows and the +separate 30-minute collector performance gate remain unverified. + +## Deploy the green PR 496 server and collector build (2026-10-24) + +Owner authorized deployment after CI passed. Rechecked exact PR head +`fb9538a83f5eec95fab12af5cc5da9699ee80d25`: 14 successful checks; full UI and +the smoke-failure reporter were conditionally skipped. PR remains Draft/open, +unmerged. Release arm64 App (1.0.5, build 1779), Service, RemoteServer and +Collector built successfully. App release verification and role package +hash/layout/signature verification passed. + +Deployed HQ index Service (PID 37240), HQ receiver/Web (PID 37548), and Daily +Collector (PID 98680). Each launch plist differs only in ProgramArguments[0]; +credentials, source roots and source authority remain intact. HQ status/search +passed with real high/medium/low/unscored value filters; the additive alias audit +table exists and FTS version remains 3. Both loopback and Tailscale HTTPS health +returned `ok`; served JavaScript exactly matches the compiled source literal +(SHA256 `48b4600994830eed6dcf9b9a324f83973a36017f0e690ac89cfd1d81bd2a5753`). +Daily dry-run after stopping the old lock owner found zero affected native +epochs. The new process stays on its first launch with no stderr growth; +HQ/M1 acknowledgments advanced. At 09:44:50 UTC, after 2h56m uptime, all +16/16 roots had converged with no recorded scan failure and no stderr growth. +HQ and M1 each acknowledged 52,623 publications. The single snapshot was +0.0% CPU / 77,104 KiB RSS; this does not prove the 30-minute CPU/RSS gate. + +Local App/legacy Service are **not upgraded**: system launchd denied an +unprivileged restart; the administrator dialog timed out both initially and +on the owner-requested continuation. No authenticated switch occurred. The old +`cabeeb24-build1569` Service/PID 30392 is still running, its `current` link is +restored, and `/Applications/Engram.app` is unchanged. New App is staged at +`/Applications/Engram-fb9538a-stage.app`; paired Service at +`~/.engram-service/releases/fb9538a-build1779`. The prepared +`finish-local-deploy.py` in the backup directory resumes the paired update +through macOS authentication. Its continuation attempt stopped at the +180-second authentication timeout before any mutation. Readback at 09:48:19 +UTC confirms old PID 30392 and App build 1. The staged App has 62 files +matching the verified Release bundle; its helper/frameworks match the staged +paired Service. Installation and App runtime acceptance remain unverified. + +Persistent evidence and rollback root: +`~/.engram-shadow-core-20260911/hq/state/deploy-fb9538a-20261024/` +(`evidence/` holds CI, builds, package checks, IPC and runtime receipts). +HQ index/local DB backups passed quick_check (39,879 / 60,119 sessions). +Daily inventory/identity backups passed quick_check under the corresponding +`daily/state/deploy-fb9538a-20261024/` directory. Old packages/plists remain; +HQ `switch-agent.py ROLE --rollback` restores the old binary configuration. +No data restore, rollback execution, business-write probe, live Keychain, +foreground App UI, full UI suite or long-window performance acceptance is claimed. +M1 replica and the separate legacy remote hub were not upgraded. + + +## Align the archive safety gate with authenticated alias deletion (2026-10-24) + +PR #496's Node and macOS gate failures shared one stale exact-body allowlist: +Web alias DELETE now validates the session actor and attributes the request. +Updated that exact allowlist and its fixture, retaining path/file/count checks. +Added rejection cases for an unverified actor, unattributed payload and extra +archive deletion. No Swift runtime or authorization behavior changed. + +Validation: repository gate regression failed before the fix; all 65 gate tests +then passed. Direct safety gate, Bash syntax, test typecheck and targeted Biome +passed (after formatting). Evidence: `/tmp/engram-pr496-gate-fix-20261024/`. +Hosted verification will run on the pushed fix; prior green Swift/UI/CodeQL +results belong to `a4d9837`, not this new commit. + +## Close the local delivery batch before expanding scope (2026-10-23) + +After owner feedback on cycle time, bound this delivery at `b17783c9` (16 local +commits since `5216ff3a`) and prepare it for review. Updated the canonical search +row to include completed value filters. Cosine insight supersession remains a +decision: synchronous save checks Web source visibility, while later embedding +maintenance has no originating caller policy. No replacement code was added. + +Validation: clean implementation worktree, commit/diff readback, and +`git diff --check 5216ff3a..HEAD` passed. Re-read the existing receipts: measurement +6 tests, value-filter App 193 / Service 66 and final render 20 all passed; these +were not rerun. Closeout changes are documentation only. Full all-target tests, +hosted branch CI, foreground UI and production acceptance remain unverified. +The existing reconciliation routes every plan and residual. PR publication, +merge and deployment have not been authorized or performed. + +## Add read-only lifecycle storage and score measurements (2026-10-23) + +Added `scripts/db/measure-database.py --database PATH [--timeout SECONDS]` for +Lifecycle 3.6/P0-6. Python standard library only; an explicit existing database is +required. One read-only/query-only transaction includes committed WAL and emits +aggregate JSON: page/free-page counts, per-object dbstat bytes, FTS page share +(including custom indexes on shadow tables), tier session/FTS-row counts, orphan +FTS rows, and non-skip score histogram/nearest-rank percentiles. NULL and invalid +scores are separate; no transcript or session ID is emitted. Missing optional +dbstat/FTS is explicit; failed/locked reads exit nonzero without partial JSON. +No migration, checkpoint, optimize, default home lookup or threshold change. +SQLite can create/use WAL coordination sidecars; directory byte invariance is +only verified for the closed non-WAL fixture, not promised for arbitrary live WAL. + +Validation: six CLI tests passed (distribution/storage/custom-index accounting, +empty/missing-FTS, open-writer committed WAL, missing DB/schema, argument bounds, +exclusive-lock timeout). The non-WAL database bytes and directory entries stayed +identical. A copied repository fixture produced 20 sessions, 18 FTS rows and +106 pages; this is synthetic evidence, not a production measurement. Biome on the +new TS test passed after formatting. Evidence: +`/tmp/engram-leader-measurement-20261023/` (tests-final.log, fixture-report.json). +No Swift runtime changed; no Xcode tests, production read or hosted CI was run. + +## Filter human search by the existing session value bands (2026-10-23) + +Search's advanced filters now offer any/high/medium/low/not-scored values, with +clear/reset and captured request state for Service and offline search. The shared +SQL predicates reuse the badge thresholds (<=35, >=60), intersect existing filters +before LIMIT and apply before both short-token candidate layouts' caps. Unknown +filter values return no session hits; omitted/blank fields preserve prior behavior. +Semantic/hybrid candidate filtering also uses the predicate; insight-library results +have no session value band. No schema, scoring formula or Web/MCP UI change. + +RED: actual client/socket/dispatcher requests failed four expected assertions. +GREEN: 193 App data/model/search tests and 66 Service tests passed, including +boundary/NULL values, rank-before-limit, a 270-row short-token interference fixture +in both layouts, and semantic/hybrid interference. Invariant/module/convention gates +passed. Evidence: `/tmp/engram-leader-value-filter-20261023/` (red/app/service logs, +xcresults and exact service-command.txt). Initial ImageRenderer picker screenshots +were unsupported placeholders; final visual verification uses an offscreen AppKit +host instead; the final 20 SearchOutcome tests passed and both English/Chinese +menu screenshots were inspected (`render-final.xcresult`, `renders-final/`). +Full-page interaction/VoiceOver, production-corpus latency, hosted CI +and deployment remain unverified. Upgrade App and Service together: an old Service +ignores the additive request field. Existing localization catalog duplicate keys +were compared to HEAD; this batch adds none and leaves the pre-existing key intact. + +## Show existing value bands in session cards and list rows (2026-10-23) + +Added a shared compact High/Medium/Low value badge to SessionCard and the Sessions +table title cell. Unknown scores show no badge; hover explains the activity-based +estimate and accessibility labels include the score. English/Chinese strings use +the existing catalog. Existing <=35 / >=60 thresholds and shared SessionQualityScore +function are unchanged; user overrides remain roadmap work. SessionTableCell is +internal so the render test exercises the actual row without production test hooks. + +Validation: 31 App model/read/render tests and two score golden/backfill tests passed +with isolated fixed homes, signing off and Engram-LeaderLoop-20261023 DerivedData. +Four ImageRenderer attachments (English/Chinese, light/dark) were exported and +visually inspected for actual cards and 420-point table cells. Evidence: +`/tmp/engram-leader-value-bands-20261023/` (app/score logs and xcresults, renders/). +This additive UI batch has no claimed bug RED. No full-page interaction, live +VoiceOver/tooltip, refreshed real-data distribution, hosted CI or deployment proof. + +## Human keyword search ranks all terms and aligns offline CJK behavior (2026-10-23) + +Reconciled Lifecycle 3.4 against current source: Service already had FTS rank and +CJK trigram matching, windowed highlights were shipped, and SearchPage already +filtered by source/project/time/origin. Both Service and App local queries still +used only the first term's rank. A query beginning with a short LIKE term lost +all relevance ranking; App's separate all-CJK LIKE route also disagreed with Service. + +Under `docs/superpowers/specs/2026-10-23-keyword-ranking-design.md`, both readers +sum each term's existing best-row rank in canonical token order, preserving +session-level AND across messages. Start time descending and session ID ascending +break ties before LIMIT and in Service's final projection. App search and +searchWithSnippets now use the shared scalar-count trigram routing. Snippet order, +Service's short-term candidate cap and existing visibility/filter boundaries stay. +No schema change, index rebuild, new preference or dependency. Native MCP's separate +ranking policy is outside this human-search change. Value/lifecycle facets and +title/quality/recency weighting remain separate roadmap work. + +The initial test build caught a missing await in fixture setup, fixed before the +behavioral experiment. RED: three shared-fixture App/Service regressions produced +72 expected ordering failures. GREEN: 163 tests passed under EngramTests +(DatabaseManagerTests, SearchModeTests, SearchOutcomeTests), plus 63 under +EngramServiceCore (selected IPC search tests, SemanticSearchIntegrityTests, +ServiceSearchInsightTests, WebSearchInsightProducerTests and four Web metadata +admission tests). Zero failures. Tests cover six permutations of three terms, +cross-message matching, relevance before LIMIT, recency/identity ties, short-first +Latin/Chinese/Japanese/Korean queries, highlights, both FTS identity layouts, +filters, SQL plan shape, busy handling and semantic fallback. Lint and invariant/ +module/convention gates passed; lint retains one existing warning and 40 infos. + +Evidence: `/tmp/engram-leader-keyword-ranking-20261023/`: ranking-red (compile), +ranking-red-behavior, app-green and service-green logs/xcresults, exact Service +selectors in service-command.txt, lint.log and invariants.log. Both fixed-home +variables use that directory's home/; xcodebuild uses macOS, signing off and the +existing Engram-LeaderLoop-20261023 DerivedData. Canonical roadmap/TODO, +reconciliation, historical plan annotation, invariant and MEMO now agree. +No all-target suite, UI render, hosted CI, production corpus latency/relevance +measurement or deployment. Next: inspect existing value-score/band UI and access/ +outcome signals before defining the next bounded visibility improvement. + +## UI harness residual narrowed to foreground acceptance (2026-10-23) + +Current-source review found isolated environment/test-mode configuration before +fallible fixture setup and fail-fast policy at all 27 configure call sites. The +old mcp-hermetic-1 wording overstated a missing policy; it now retains only the +unproven foreground behavior. No production or harness code changed. + +A temporary missing-fixture test, using an isolated App bundle identifier and no +actual app.launch call, built successfully but never executed: the XCUITest runner +failed initialization with LocalAuthentication Code=-4, system authentication +running. No authentication process was touched or immediate retry attempted. +The four experimental files were verified against their HEAD backups and restored +byte-for-byte. The negative control and normal foreground smoke were not run. + +After restoration, EngramTests/MCPActivationOnboardingTests source regressions +testUITestLaunchesUseHermeticHomeAndSkipSettingsMutation_repro and +testUITestConfigureFailsClosedBeforeReportingSetupFailure_repro passed (2 tests, +zero failures). These prove source ordering/wiring only. Both fixed-home variables +used `/tmp/engram-leader-ui-harness-20261023/home`; xcodebuild used the existing +Engram-LeaderLoop-20261023 DerivedData, macOS destination and signing off. +Evidence: `/tmp/engram-leader-ui-harness-20261023/`, failure-stop and source-policy +logs/xcresults, original backups and temporary test source. Canonical followups +and reconciliation updated. No full suite, hosted CI or production verification. +Next independent batch adjudicates the current human keyword-search contract. + +## Web alias mutations now commit bounded actor history atomically (2026-10-23) + +Completed the alias gap in identity-auth phase 2 under +`docs/superpowers/specs/2026-10-23-web-alias-audit-design.md`. Web alias add/remove +now takes optional validated actor metadata, injected by RemoteServer from the +bound editor session after rejecting unknown browser body keys. Credential mode +and older clients use web-editor; later identity headers cannot replace the actor. +The real Service dispatcher admits the optional field through existing capability +and writer-gate boundaries. Native alias normalization is unchanged. + +Swift's additive schema creates web_alias_audit_log with insertion ID, UTC time, +action, actor and published alias/canonical identities. An actual exact-pair change +and its audit row share one writer transaction. No-op/refused requests create no +history. Audit insertion or retention failure rolls the entire business mutation +back and returns the existing generic unavailable error. Keep the latest 10,000 +transitions by insertion ID, independent of clock rollback; no raw paths, request +bodies, historical backfill or public audit endpoint. This is bounded local Web +mutation history, not permanent auditing or coverage of native alias changes. +User-data-only backup payloads do not include the diagnostic table; whole-database +backups retain it. Existing alias data and prior project migration history stay +intact. + +RED: 12 selected Service tests produced 20 failures, including four missing-table +errors and 16 invalid-actor assertions; five HTTP tests produced four missing-actor +assertions. GREEN: all 13 selected Service tests passed (12 WebAliasWriteTests plus +CollectorWebDemoTests.testAliasEditorThroughHTTPAndServiceIPC), including actual +HTTP -> Web client -> capability socket -> dispatcher -> SQLite readback, and a +separate identity-actor socket/dispatcher test. The latter also verifies an audit +failure becomes generic unavailable without deleting the alias. All 46 selected +RemoteServer tests passed (alias HTTP 5, auth routes 28, project HTTP 4, write +client 9), plus all 20 MigrationRunnerTests. SQLite trigger injection verifies +add/remove audit rollback and prune rollback; 10,000 seeded rows prove insertion- +ordered retention. Old and repeated migrations preserve business data/history. +All passes had zero failures. Lint and invariant/module/convention gates passed. + +Evidence: `/tmp/engram-leader-alias-audit-20261023/`, with `service-red`, `http-red`, +`service-green`, `http-green`, `migrations-green` logs and xcresults; `lint.log` +and `invariants-final.log`. Both fixed-home variables point to its `home/`. +Tests use EngramServiceCore, EngramRemoteServerCore and EngramCoreTests with the +shared Engram-LeaderLoop-20261023 DerivedData, macOS destination and signing off. +The new spec and invariant describe storage, retention, rollback and exact scope; +canonical TODO/followups/reconciliation and the prior auth/addendum now agree. + +No all-target suite, hosted CI, browser rendering, production migration, restart, +push or deployment. Upgrade Service before RemoteServer: old Service rejects the +new actor key, while old RemoteServer/new Service uses web-editor. Rolling back +Service alone can interrupt writes from the new RemoteServer; mixed-host behavior +has not been accepted on a real host. Credential-mode global throttling remains +its accepted provenance/design residual. Next local queue item: reproduce or +adjudicate foreground UI harness failure policy (`mcp-hermetic-1`). + +## Measured Cursor observation cost reduced without weakening source checks (2026-10-23) + +Isolated profiling of `CollectorCursorLegacyOwnership.ownershipObservationPage` +used a temporary XCTest method, synthetic workspace.json/database/WAL placeholders, +Release optimization with testability enabled, and native arm64. There was no live +source, database or running collector access. Three rounds of 50 pages per case +measured process user+system CPU, excluding fixture creation. A five-second sample +during an additional 2048-workspace/64-item loop found 699/3738 sampled stacks in +SHA-256 digest hex formatting (`String(format:)`). The cryptographic hash itself +was not the hotspot. `ArchiveV2Hash.sha256` now uses a fixed lowercase byte table +and a 64-byte buffer. Existing directory enumeration, no-follow stat/open checks, +observation cadence, ownership fingerprint input and SHA-256 algorithm remain. + +| Workspaces | Page size | Baseline CPU ms/page | Optimized CPU ms/page | Reduction | +|---|---:|---:|---:|---:| +| 64 | 1 | 0.298 | 0.248 | 16.8% | +| 64 | 64 | 2.982 | 1.536 | 48.5% | +| 512 | 1 | 1.017 | 0.970 | 4.6% | +| 512 | 64 | 5.232 | 2.323 | 55.6% | +| 2048 | 1 | 3.867 | 3.746 | 3.1% | +| 2048 | 64 | 6.824 | 5.245 | 23.1% | + +Values are medians, not a whole-collector CPU claim. The 512/64 baseline varied +3.49-6.37 ms across its rounds; its percentage is especially timing-sensitive. +The stable 2048/64 rounds were 6.686-7.044 before and 5.242-5.255 after. The final +sample contains zero digest-formatting stacks among 3764 samples. One-item pages +still pay the full membership scan; this patch does not close collector CPU, +modern-peer discovery or startup cost. Daily's comparable 30-minute CPU <=2% / +RSS <=150 MiB acceptance remains open and separately authorized. + +Verification: 121 Collector tests passed (ownership 35, capture 6, privacy 80), +plus 169 Core archive tests (model 48, catalog 46, legacy session 9, exact source +capture 37, immutable CAS 29), all with zero failures. Fixed SHA-256 vectors now +include leading zero bytes, binary 0...255 and a 65-byte input; expected values +were independently checked with Python hashlib. Both profile test runs passed. +The initial Release attempt failed before tests because testability was disabled; +rerunning with `ENABLE_TESTABILITY=YES ONLY_ACTIVE_ARCH=YES` resolved that setup +issue without changing project configuration. Lint and invariant/module/convention +gates passed; the temporary benchmark method was removed byte-for-byte before +final correctness tests. The persistent code diff is limited to the hash helper +and its existing fixed-vector test. + +Evidence: `/tmp/engram-leader-cursor-perf-20261023/` contains `command.txt`, +`benchmark-method.swift.txt`, `measurement-context.json`, `baseline-measured` and +`optimized` logs/xcresults, both `.sample.txt` profiles, `comparison.json`, the +baseline binary receipt, `collector-green` and `archive-green` logs/xcresults, +`lint.log`, and `invariants.log`. The first failed build is `baseline.log`. +Correctness uses schemes EngramCollectorCore and EngramCoreTests, the shared +`Engram-LeaderLoop-20261023` DerivedData, signing disabled, and both fixed-home +variables pointing to this evidence directory's `home/`. No hosted CI, all-target +suite, actual collector CPU window, production rollout, push or restart. +Next local batch: specify alias audit storage, transaction and retention semantics +before implementing the remaining Web editor audit gap. + +## Legacy body error mapping verified through production routes (2026-10-23) + +Closed the local verification residual `r11-dropped-2` without a runtime patch: +`EngramRemoteServerTests` now dispatches controlled asynchronous body streams +through `EngramRemoteServerApp.buildRouter()` instead of inspecting source strings. +Four tests cover partial failure (503), cumulative overflow (413), exact-limit +success, and authentication/Origin rejection before any stream read. Failed bundle +uploads leave both new keys absent and existing encrypted file bytes unchanged. +MCP checks JSON-RPC error code/message/null ID, complete malformed JSON (400), +and initialize at the exact 1 MiB boundary. Stores and tokens are temporary test +fixtures; an EmbeddedChannel supplies request context without binding a listener. +The existing live-socket initialization and other server tests remain in the suite. + +Negative control temporarily reinstated the historical catch-all 413 mapping in +both body handlers: the two error tests failed five expected assertions. The file +was restored byte-for-byte (SHA-256 +`7b2560e39c4bb41313e9a3984d2c9d5637639231a96797796c5bef499a2f5bdc`), then all 28 +EngramRemoteServerTests passed with zero failures. Lint and invariant/module/ +convention gates passed. Evidence: `/tmp/engram-leader-body-20261023/`, including +`negative-control.{log,xcresult}`, `green-final.{log,xcresult}`, `lint.log`, and +`invariants.log`. Both fixed-home variables point to its `home/`. Test command: +`xcodebuild test -project macos/Engram.xcodeproj -scheme EngramRemoteServerCore +-destination 'platform=macOS' -derivedDataPath +/Users/bing/Library/Developer/Xcode/DerivedData/Engram-LeaderLoop-20261023 +CODE_SIGNING_ALLOWED=NO +-only-testing:EngramRemoteServerCoreTests/EngramRemoteServerTests`. + +No actual network disconnect, hosted CI, all-target suite or production acceptance +was exercised. Next local batch measures Cursor legacy observation cost using +isolated synthetic fixtures; no current-host CPU claim or service restart. + +## Transcript paging tasks retain ownership through session switches (2026-10-23) + +Mounted SessionDetailView verification narrowed `session-detail-id-3` to two +concrete paging races. A cancelled old page/all/copy task still ran an unconditional +defer and cleared the next session's busy flag; another click then cancelled the +valid new page and started a duplicate. Independently, two actions before the +scheduled task began both passed the not-busy guard. Page/all/copy now set busy +synchronously and own a fresh load token; only the owning task can clear busy or +the stored task handle. Switch/disappear invalidates the token. Returning A-B-A +cannot restore old ownership. Existing cancellation checks already prevented the +tested stale initial success/error and indexed-message applications; those paths +received verification rather than redundant production rewrites. + +Added instance-scoped DEBUG-only parse/build/copy hooks and a mounted NSHostingView +regression using controlled non-cooperative continuations. The real view's State, +task cancellation, indexing and page/copy actions run; fake transcripts, a mock +service client and an explicit absent temporary DB path avoid live data. Clipboard +output is captured by the test hook. Final tests read mounted State locations on +demand: the initial snapshot observer lagged coalesced offscreen render passes and +produced timeouts, including a same-frame onChange warning. Those failed runs are +retained; they are not passed acceptance. No test hook is compiled into Release. +The new test file was registered using the CI-pinned XcodeGen 2.45.4; initial PATH +2.46.0 generation was replaced before final acceptance, leaving four project entries. + +Validation: baseline RED demonstrated both races; after stabilizing observation, +all five mounted tests passed five iterations (25 executions). A negative control +temporarily removed both fixes with the final harness and failed 17 expected +assertions, including page/all/copy success/error and A-B-A. Restoring the fix then +passed the final 72 tests: mounted switch 5, find 40, labels/copy 19, paging/timing 6, +accessibility 2. All final passes had zero failures and exit 0. Lint passed with the +existing 1 warning/40 infos; invariant/module-boundary/convention checks, Xcode +project drift and diff checks passed. Exact commands, logs and xcresults are under +`/tmp/engram-leader-transcript-20261023/`: `red-matrix`, `green-live-state`, +`negative-control`, `green-final`, `lint-final.log`, `invariants-final.log`, and +`xcodegen-drift.log`. Both fixed-home environment variables point to its `home/`. +Release `xcodebuild build` (scheme Engram, CODE_SIGNING_ALLOWED=NO) also passed +for arm64 and x86_64. The Debug dylib contains the test-hook symbols and the +Release executable contains none; `build-receipt.json` records their paths, +architectures and the Release executable SHA-256. Build log: `release-build.log`. +All 77 local links in the touched canonical routing documents resolve. + +This closes the reproduced local transcript residual and the documented bounded +interleavings, not every possible UI sequence or end-to-end parser/clipboard case. +No production data, service restart, push, deploy or hosted CI. The next independent +residual is `r11-dropped-2` transport-error mapping verification; performance and +cutover/real-host receipts remain separate queue items. + +## Credential-mode throttle reconciled against its accepted scope (2026-10-23) + +Current source still shares five login attempts per minute across credential +clients, including successful attempts. Fresh WebAuthSessionTests (19) and +WebAuthRouteTests (28) passed, including concurrent limits, retry-window expiry, +logout behavior and forged forwarding/identity headers. Evidence: +`/tmp/engram-leader-web-endpoint-20261023/auth-adjudication.{log,xcresult}`. +This confirms the limitation; it does not fix global-lockout availability. + +The accepted 2026-10-08 identity-auth design explicitly parks credential-mode +per-client throttling. Credential routes currently have no trusted client-address +contract and intentionally ignore identity/forwarded headers. Keep this as a +deployment/auth design item: define trusted proxy/client provenance and a global +guessing budget before partitioning it. No weakening of credential checks, new +forwarded-header trust, auth-code change or host configuration change. The next +independent verification item is delayed transcript completion during session +switching; current cancellation checks and unguarded paging-task cleanup need a +mounted-view reproducer before promoting a new defect. + +## Web AI endpoint changes cannot redirect saved credentials (2026-10-23) + +The Web AI editor could change a provider base URL while retaining the server's +API key. The service writer now checks the proposed effective destinations +under the existing settings-file lock and rejects the entire patch when any +relevant credential exists or its absence cannot be established. The policy +covers summary-to-embedding fallback, legacy title URLs, title-provider switches, +disabled providers, settings markers/plaintext, environment keys, custom/default +runtime sidecars and markerless Keychain items. Environment URL overrides do not +permit staging a later credential redirect. Full URL/path changes are protected. +Keychain inspection requests attributes with interaction disabled, never secret +data; unsafe sidecars and indeterminate Keychain status fail closed. No credential +migration, clearing, schema change or new configuration flag is introduced. + +Credential-free endpoint editing and unrelated model/prompt edits remain usable. +A typed service rejection survives the real Web client/socket/dispatcher boundary, +maps to HTTP 409, and leaves a correctable draft with an explicit not-saved message. +Credential-bound endpoint changes require a local server settings operation. + +Validation: the behavioral RED reproduced an accepted redirect and changed file +bytes (2 failing assertions); the UI RED reproduced the misleading uncertain-save +message. GREEN: 26 Service tests (16 Web AI plus 10 secure writer), 24 RemoteServer +tests (6 AI HTTP, 9 Web write client, 9 Web UI routes), and all 181 Web JS behavior +tests passed. The service suite includes real capability-protected IPC and exact +settings-byte readback. Security queries are injected; fixtures contain fake keys. +Both fixed-home environment variables point to an isolated temporary directory. +Lint passed after formatting the new test (existing 1 warning / 40 infos); +invariant/module-boundary/convention gates and diff checks passed. Logs and exact +xcodebuild invocations: `/tmp/engram-leader-web-endpoint-20261023/` +(`service-red`, `service-green`, `http-green` logs/xcresults; `ui-red.log`, +`ui-green.log`, `lint-final.log`, `invariants.log`). JS verifier: +`npx vitest run tests/scripts/collector-web-ui.test.ts`. + +Scope: this protects Web configuration writes, not DNS/HTTP redirects or a trusted +local administrator changing credentials concurrently. Locked/inaccessible secret +stores can conservatively block legitimate endpoint edits. No live Keychain, +outbound AI request, browser render, all-target suite, hosted CI or production +acceptance. No push/deploy/restart. Credential login throttle and alias audit +storage remain separate work; the continuous local-development goal remains active. + +## Web project audit actor accepted by the real service dispatcher (2026-10-23) + +A continuation check found a gap in the preceding actor batch: the typed Web +client encoded actor and direct business-handler calls stored it, but the real +Service dispatcher still rejected actor as an unknown payload key. The earlier +123 passing tests verified the components separately and did not prove their +composition. Added actor to the four project commands' optional IPC field +allowlists; the HTTP body allowlists still reject browser-supplied actor. + +Upgraded the existing audit/replay regression to use the production Web write +client, a real temporary UnixSocketServiceServer and EngramServiceCommandHandler +before reading back the fixture migration_log. It exercises move, archive, +undo, batch, cross-actor replay conflict and missing-actor compatibility. +RED at `6d30437a`: the first real request failed with invalid before mutation. +GREEN: all 6 WebProjectMigrationWriteTests plus 12 ProjectMoveLongOpIPCTests +passed (18 total, 0 failures, exit 0). Lint and invariant gates passed. Evidence: +`/tmp/engram-leader-web-dispatch-20261023/` (red/green logs and xcresults, +lint/invariants logs). No new schema, production write, hosted check, push, +deploy or restart. The original method-only green was insufficient evidence +for the service-dispatch claim; this receipt supplies the missing boundary. + +## Web project operations retain the authenticated audit actor (2026-10-23) + +RemoteServer previously dropped the editor identity; Service forced project +move/archive/undo to mcp, while batch items wrote batch. Move, archive, undo +and move-batch now derive attribution from the live editor session (bound +Tailscale login or web-editor for credential mode), reject browser-supplied +actor fields, and carry a bounded optional actor through capability-protected +IPC. Older Web clients omitting actor receive web-editor. Native CLI/MCP actor +parsing and native batch defaults remain unchanged. + +The native audit actor is now a bounded RawRepresentable value instead of a +four-case enum. An explicit internal Web override reaches the existing +migration-log TEXT field and each batch item; the same value participates in +operation fingerprints so another actor cannot replay a cached operation ID. +No schema migration, broader IPC command allowlist or historical-row rewrite. +Updated the invariant, canonical ledgers and the phase-2 design addendum. + +**Validation.** HTTP RED: 8 missing-attribution assertions across the four +routes in both auth modes. Service RED: 8 assertions covering ignored invalid +actors, wrong stored actors and cross-actor replay. Final GREEN: 60 HTTP/auth/ +IPC tests (including forged later headers, browser actor rejection and local +capability-token transport); 29 service/project registry/long-operation tests +(including real temporary project move/archive/undo/batch and SQL audit +readback); 34 Core batch/migration-store tests. All passed with 0 failures, +exit 0. Service tests used an isolated fixed home. Lint passed with the +existing 1 warning/40 infos; invariant/module-boundary/convention gates and +final diff/link checks passed. Exact commands are in the xcodebuild logs; +logs and xcresults are local-only under +`/tmp/engram-leader-web-actor-20261023/` (http-red, service-red, service-green, +http-green, core-green; plus lint/invariants logs). + +**Limits and remaining scope.** The older phase-2 design incorrectly assumed +aliases already had an audit writer/actor field. Their history storage and +retention remain a separate design item; the four project operations do not +close all Web hardening. Credential throttle and endpoint/key reuse still need +triage. Audit attribution is metadata, not isolation from another authorized +editor or a trusted local capability holder. Deploy Service and RemoteServer +together before claiming identity attribution on a host; an old Service may +ignore the additive field. No full all-target suite, UI/render, hosted CI or +production acceptance; no push, deployment, restart, Docker or live data write. + +## Historical secrets and session-identity residuals re-adjudicated (2026-10-23) + +Current source at local `414b7c18` already contains later fixes missing from +the August Round-12 follow-up table: dirfd-pinned secret operations, custom +socket sidecar isolation, explicit bridge/marker failure results, empty-key +preservation, injected Settings socket/default test home, nil-navigation +completion and favorite generation invalidation. Rechecked the call sites and +ran the existing regressions before closing that summarized scope locally. +Updated TODO/followups and the dated reconciliation; no production code changed. + +Validation: `EngramTests` selected App run passed 210 tests, 0 failures, exit 0 +(36 AI settings, 19 launcher, 32 Unix socket, 1 default DB home, 43 home/popover, +39 scope and 40 transcript-find). Both `CFFIXED_USER_HOME` and +`TEST_RUNNER_CFFIXED_USER_HOME` point to a temporary home under +`/tmp/engram-leader-residuals-20261023/`; that directory holds the exact command, +`app-residuals.log` and `app-residuals.xcresult`. These are fresh tests of +existing fixes, not a new RED/GREEN implementation claim. No live Keychain, +mounted-view/render, full-suite or hosted acceptance was performed. + +`session-detail-id-3` remains verification pending: transcript apply paths +check cancellation and their tasks cancel on switch/disappear, but the +existing predicate/source tests do not prove every delayed mounted-view +interleaving. Do not claim it fixed or re-count the closed navigation/favorite +subitems. Web audit actor propagation is next; its actor enum limitation must +be handled along with the DTO boundary. No push, deployment or restart. + +## Collector blocked roots retry with bounded backoff (2026-10-23) + +`CollectorRuntime` previously restarted a blocked bootstrap root on subsequent +capture turns, repeatedly walking it and recording continuity loss. It now +uses per-root monotonic delays of 1, 2, 4, 8, 16, 32 and at most 60 seconds. +Cooling roots are excluded from listener restarts, inventory steps, new +capture and scheduler samples; healthy roots and independent upload loops +continue. Partial re-walk progress retains the delay, a completed scan resets +it, and reopening the runtime retries immediately. No schema or settings +format changed. Updated the invariant and closed +`collector-blocked-root-backoff-1` locally in the canonical ledgers. + +**Validation.** Deterministic-clock regression first failed with 24 behavioral +assertions (exit 65; an earlier scaffold compile error was corrected before +this RED). Focused GREEN and the 87-test runtime suite passed. A negative +control removing the cooling-root scheduler filter made the resident test +fail with an extra capture turn. The filter was restored, then the final +`EngramServiceCoreTests/CollectorRuntimeTests` run passed all 87 tests with +0 failures (exit 0). Tests cover cap, wall-clock jumps, healthy-root two-replica +ACKs, permission recovery, reset/restart and resident wake behavior. +`npm run lint` passed with the existing 1 warning/40 infos; +`bash scripts/check-invariants-ledger.sh` passed. Evidence is local-only in +`/tmp/engram-leader-backoff-20261023/`: `red-behavior`, `negative-scheduler`, +`green-final` logs/xcresults, plus lint/invariant logs. Xcode uses +`Engram-LeaderLoop-20261023` under DerivedData, not `macos/build`. + +**Limits.** Newly readable blocked roots may wait the remaining cooldown +(up to 60 seconds) plus scheduler/work latency. This does not establish the +Daily CPU <=2% target. No all-target suite, UI/render or hosted checks were +run for this batch; no push, deployment, restart, production write or Docker. +Next local batch is current-source adjudication of historical secrets/UI +residuals, followed by Web audit actor propagation. + +## Development queue reconciled; MCP role discovery and collector fixtures fixed (2026-10-23) + +Owner requested continuous development leadership. Work is isolated in +`.worktrees/leader-loop-20261023` on `codex/leader-loop-20261023`, starting at +local/live remote main `5216ff3a61202b4c113b1e20799a03ad7e7df706`. + +**Inventory.** `docs/reviews/2026-10-23-development-reconciliation.md` routes +all 21 implementation plans, 24 specs, 27 root design documents and every +nonterminal follow-up family. The 397 unchecked plan markers are historical +procedural steps, not an outstanding-task count. Updated TODO/followups/roadmap +and the review index; closed stale no-real-binary-gate/no-install-tooling text, +reclassified Grok/Pi and SessionStart plugin work as partial, recognized the +existing Swift AI audit writer/redaction, and restored the omitted lifecycle +3.2 decision. Product ledger: 13 families, 12 open/conditional and 1 implemented. +GitHub readback found 0 issues and 6 dependency PRs (#490–#495); baseline Tests +39366050861 and CodeQL 39366050927 passed. Historical real-binary run +39098816343 was rechecked successful, not rerun for this branch. + +**Batch 1.** `MCPToolRegistry.tools` now returns an empty catalog for collector, +replica and invalid-settings roles before a database probe. Direct calls still +provide role-unavailable guidance. Local/index/default roles keep all 27 tools +and embedding-dependent search modes. Updated the persisted-role invariant +and executable tests in both legacy and modern response formats. + +**Batch 2 (test infrastructure only).** `RuntimeFixture.writeSettings` now +stages a complete sibling document and sets 0600 before atomic rename, avoiding +both truncated JSON and temporary permission failures during runtime reload. +The publication reservation test now defers fixture cleanup. No production +collector behavior or existing temp directories were changed. The historical +hosted flake was not independently reproduced this session; hosted recurrence +is not proven eliminated by local success. + +**Validation (Xcode 26.4, Node 24.22.1).** + +- RED: `xcodebuild test -project macos/Engram.xcodeproj -scheme EngramMCPTests + -destination 'platform=macOS' -derivedDataPath + /Users/bing/Library/Developer/Xcode/DerivedData/Engram-LeaderLoop-20261023 + CODE_SIGNING_ALLOWED=NO + -only-testing:EngramMCPTests/EngramMCPExecutableTests/testRuntimeRoleDiscoveryOmitsUnavailableTools_repro`: + 1 test failed with exactly 8 expected catalog assertions, exit 65. +- GREEN: same command, selecting `EngramMCPTests/EngramMCPExecutableTests`: + 215 tests passed, 0 failed, exit 0. +- GREEN: same project/destination/DerivedData/signing, scheme + `EngramServiceCore`, selecting `CollectorRuntimeTests`, + `CollectorPublicationWorkerTests/testFreshOnlyReservationDoesNotReuseAnExistingCapture` + and `ServiceAIAuditRecordTests` under `EngramServiceCoreTests`: + 94 tests passed (85 + 1 + 8), 0 failed, exit 0. AI audit tests provide fresh + evidence for the roadmap correction, including redaction and opt-in bodies. +- `npm run lint`: exit 0, 1 existing warning and 40 infos; no autofix or + unrelated formatting changes. `bash scripts/check-invariants-ledger.sh`: + PASS, including registered Swift module-boundary and convention checks. +- Reconciliation's 72 local Markdown links resolve; `git diff --check` passed. + Logs and xcresults are local-only under + `/tmp/engram-leader-loop-20261023/` (`mcp-role-red`, `mcp-role-green`, + `collector-fixtures-green`, plus `lint.log` and `invariants.log`). + +**Not run / boundary.** Full all-target tests, UI/render checks, hosted checks +for this branch, current production readback and real-host CPU/cutover gates +were not run. No push, merge, deployment, service restart, live database write, +release, Docker or global-memory change. The main checkout's pre-existing +untracked Grok workflow is preserved. No unattended scheduler was installed; +the active conversation goal carries continuation. Next bounded work is +blocked-root retry backoff, with current security/UI residual adjudication and +Web actor propagation queued; measure collector CPU before further tuning. ## Web viewer polish deployed to the HQ receiver (2026-10-20) diff --git a/MEMO.md b/MEMO.md index d91b20b46..353b48816 100644 --- a/MEMO.md +++ b/MEMO.md @@ -2,6 +2,62 @@ ## Changelog Memo +### 2026-10-25 + +- [修复] 修正 CI 的时间敏感测试：摘要保留 10 秒 CPU 门槛，服务停止先等待夹具就绪；两项负对照均被拒绝，恢复后 70 项测试全过。仅测试与记录变更，新提交 hosted CI 待验证，详见 CHANGELOG。 +- [验证] Daily 采集器完成 30 分钟只读验收：平均 CPU 1.921%、最高 RSS 58.5 MiB，均达标；61 个样本中 16 个目录始终收敛。关闭空闲窗口证据缺口，持续写入与启动性能仍待验收。详见 CHANGELOG。 +- [排查] 603 个待上传条目均由隐私策略保留；275 个重试项分别为 14 个已捕获后缺失文件、6 个空 Kimi 文件和 255 个待核实 Copilot 条目。本轮未改代码、配置或清理数据；上一提交 14 项 CI 全过。 +- [部署] 菜单栏修复 `a9e59d50` 在 14 项 CI 全过后已升级 App 至 1.0.5 build 1782；安装字节、进程、IPC 与 HQ 健康读回通过，Service 保持原 PID，旧版 1779 可回滚。详见 CHANGELOG。 +- [修复] 菜单栏近期会话因旧会话恢复出现重复日期 ID；改用每组首个会话的稳定标识，保留排序与日期显示。回归先失败、修复后 46 项测试通过；真实隔离夹具旧版复现警告、新版观察 92 秒未出现。 +- [验证] Hosted UI smoke 14 项及截图对比通过；本机截图受屏幕录制权限限制，完整 UI 与回滚演练未做。详见 CHANGELOG。 +- [部署] 本机 App 已升级并打开为 1.0.5（build 1779），配套 Service 同步升级；正式签名、真实搜索／价值筛选、进程和安装文件读回通过，旧版与数据库备份保留。详见 CHANGELOG。 +- [修复] 首次启动暴露临时签名的框架加载问题，先恢复旧服务，再用 Daily 上现有 Developer ID 统一签名；已证明程序代码未变，并通过实际加载检查。当前 App 显示欢迎向导，完整 UI 和 30 分钟性能验收仍未执行。详见 CHANGELOG。 + +### 2026-10-24 + +- [部署] PR #496 的 `fb9538a` 经 14 项 CI 成功后，已升级 HQ 索引／Web 和 Daily 采集器；真实价值筛选、双入口健康、Web 字节匹配及采集上传进展通过。旧包与数据库备份保留，详见 CHANGELOG。 +- [未完成] 本机 App／配套 Service 因系统管理员认证超时仍保持旧版，新版 build 1779 已备妥；Daily 已运行近 3 小时，16/16 个根目录收敛且无新错误；30 分钟 CPU/RSS 仍待验收。详见 CHANGELOG。 + +- [修复] PR #496 的两项 CI 失败同源于安全门禁仍匹配旧别名删除结构；更新精确白名单并补上身份绕过、未归属载荷及额外删除的拒绝测试。先复现失败，再通过 65 项测试及脚本/typecheck/lint；新提交的 hosted CI 待验证。详见 CHANGELOG。 + +### 2026-10-23 + +- [收口] 按反馈结束本批范围扩张：16 个本地提交已完成开发盘点、可靠性/Web 修复、搜索与价值可见性及测量工具；准备送审，尚未推送部署。语义替代涉及异步写入与 Web 权限，保留待决策，不为清空台账直接启用。详见 CHANGELOG。 + +- [新增] 补齐只读数据库测量命令，显式传入路径即可输出存储／FTS 占比、tier 数量与评分分位数；6 项 CLI 测试和仓库夹具读回通过。未读取生产库、未调整评分阈值。详见 CHANGELOG。 + +- [新增] 搜索可按高／中／低／未评分筛选，Service 与离线查询均在结果截断前过滤；短词候选上限也先过滤，避免旧高价值会话漏掉。先红 4 条 IPC 断言，再通过 193 项 App＋66 项 Service 测试。需配套升级 App/Service，未部署。详见 CHANGELOG。 + +- [新增] 会话卡片和列表标题显示现有高／中／低价值标签，未评分不显示；复用既有评分与阈值，中英文及明暗主题组件截图检查完成，33 项测试通过。用户覆盖仍未实现，未部署。详见 CHANGELOG。 + +- [修复] 搜索现综合全部关键词相关度，短词放首位不再吞掉后续长词排序；App 离线中文/日文/韩文检索与 Service 使用一致路由，平分按时间/ID 稳定排序。72 条行为断言先红，修复后 163 项 App＋63 项 Service 测试通过。详见 CHANGELOG。 +- [盘点] Lifecycle 3.4 的高亮、基础筛选、Service CJK 排序已有实现，台账改为部分完成；价值/生命周期筛选及加权仍单列。未推送部署、未测生产语料质量/时延，下一批核对价值分和使用信号。详见 CHANGELOG。 + +- [排查] UI 测试隔离环境与 27 处 fail-fast 策略已存在，2 项源码约束测试通过；前台故障注入被系统认证状态阻止，测试体未执行，不能宣称行为通过。临时 bundle ID/测试已逐字节恢复，残项收窄为前台验收，继续搜索开发。详见 CHANGELOG。 + +- [新增] Web 别名增删现与操作者审计同事务提交，审计/裁剪失败回滚；仅存公开项目标识，保留最近 10,000 条实际变更。身份由登录会话绑定，浏览器不能伪造，旧客户端回退 web-editor。79 项定向测试通过；须先升级 Service 再升级 RemoteServer，未部署。详见 CHANGELOG。 + +- [优化] 隔离 Release 测量定位 Cursor 观察路径的 SHA-256 十六进制格式化开销，改为字节查表；2048 工作区/64 项分页中位 CPU 6.82→5.24 ms（约 -23%），文件安全检查保持不变。290 项采集/归档测试通过；这是合成样本收益，Daily CPU 目标仍未验收。详见 CHANGELOG。 + +- [验证] `r11-dropped-2` 已用正式 Router 的可控断流/超限请求替代源码字符串检查；覆盖 503/413、MCP 错误体、失败上传不落盘/覆盖、上限成功和鉴权先于读体。旧映射负对照出现 5 条预期失败，恢复后 28 项测试全绿；未做真实网络断流或生产验收。详见 CHANGELOG，下一批测量 Cursor 观察路径成本。 + +- [修复] 挂载真实 SessionDetailView 复现旧分页清掉新会话忙碌状态，以及任务启动前重复点击发起双请求；page/all/copy 现在同步标忙并按独立任务 token 清理，A→B→A 也不会误解锁。旧初始结果/错误与索引应用的取消守卫经验证有效。详见 `CHANGELOG.md`。 +- [验证] 观察器改为读取挂载 State 后，5 项用例连续 5 轮通过；撤掉修复的负对照出现 17 个预期失败，恢复后 72 项转录测试全绿，lint/invariants/XcodeGen drift 通过。假转录、隔离 home、复制输出注入；未推送部署。该残项本地关闭，下一项核验传输错误映射。 +- [验证] Release 的 arm64/x86_64 构建通过；Debug 动态库可见测试钩子，Release 可执行文件无对应符号。构建日志、产物哈希和 77 个本地文档链接检查见 `CHANGELOG.md`。 +- [盘点] 凭据登录全局五次/分钟限制由当前源码及 47 项认证测试确认；现行规格明确暂缓按客户端限流，且凭据模式尚无可信客户端地址契约，保留设计项，不改信任转发头。下一项为转录快速切换的挂载视图延迟回归。详见 `CHANGELOG.md`。 +- [修复] Web AI 修改地址不能继续复用服务端旧密钥：覆盖共享 Embedding 回退、旧标题字段、禁用凭据、环境变量、sidecar 与 Keychain；无法确认无凭据时拒绝整笔保存。无凭据地址编辑和普通模型/提示词编辑保留，页面明确说明未保存并允许修正草稿。详见 `CHANGELOG.md`。 +- [验证] 先复现地址重定向与页面误报，再通过 26 项 Service、24 项 HTTP/IPC/UI 路由、181 项 Web 行为测试及 lint/invariants；真实 socket 请求拒绝后文件逐字节不变。均为本地假密钥夹具，未读真实 Keychain、未推送部署。下一项核实凭据登录全局限流。 +- [修复] 持续复核发现上一批 Web actor 漏了服务总入口字段白名单：此前传输与业务方法分开通过，组合仍会被拒绝。补齐四条 IPC 白名单，把审计回归改为真实 socket→总入口→数据库链路；先复现 invalid，再通过 18 项相关测试。HTTP 仍拒绝浏览器自带 actor。详见 `CHANGELOG.md`。 +- [修复] Web 项目 move/archive/undo/batch 的审计身份从登录会话贯穿到 IPC 与 migration log；共享口令模式记为 web-editor，拒绝浏览器伪造 actor，不同 actor 不能复用同一操作 ID 的缓存结果。无 schema 迁移，原生 CLI/MCP 默认行为保留。详见 `CHANGELOG.md`。 +- [验证] HTTP 8 个、服务 8 个预期断言先失败；修复后 Web/认证/IPC 60 项、服务/注册表 29 项、底层 batch/store 34 项通过，lint/invariants 通过。尚未推送或部署；别名没有现成审计存储，单独保留设计项，Web 限流与 endpoint/key 风险仍待核实。 +- [盘点] 发现 Round-12 台账漏记后续修复：密钥文件/保存结果/测试路径隔离、导航与收藏 generation 已有实现；核对当前调用链并通过隔离 home 下 210 项 App 测试后，本地关闭这些已描述范围，未重复改生产代码。转录 apply 的快速切换场景仍未验证，单独保留。下一项为 Web 审计 actor 透传。详见 `CHANGELOG.md`。 +- [修复] blocked-root 重试改为逐根单调时钟退避（1 秒起、最高 60 秒）；冷却期间不触发捕获空转，健康根与独立上传继续，扫描完成后重置、重启后立即尝试。该 followup 本地关闭。详见 `CHANGELOG.md`。 +- [验证] 先复现 24 个行为断言失败，再通过最终 87 项运行时测试；移除调度过滤的负对照确实失败，恢复后全绿。lint/invariants 通过。尚未推送或部署；权限恢复可能等待剩余退避时间，不能据此宣称 Daily CPU 达标。下一批核实现有安全/UI 残项。 +- [盘点] 建立持续开发目标与隔离工作树 `codex/leader-loop-20261023`；盘点 21 份实施计划、24 份规格、27 份设计文档，校正 ROADMAP/TODO/followups。397 个未勾选标记不是 397 个待开发功能；产品台账现为 12 个开放/条件性方向、1 个已实现方向。完整路由见 `docs/reviews/2026-10-23-development-reconciliation.md`。 +- [修复] MCP 在 collector/replica/无效配置下不再广告不可调用的工具；正常角色保留 27 个工具。另修复 collector 测试设置文件的非原子写入与一处夹具清理遗漏。 +- [验证] MCP 先复现 8 个预期断言失败，再通过 215 项可执行测试；collector/AI 审计相关 94 项通过。lint、invariants 与 72 个文档链接检查通过；lint 保留原有 1 warning/40 infos。详见 `CHANGELOG.md`。 +- [未验证] 尚未推送、部署或跑本分支 hosted CI/UI；真实主机 CPU、HQ/M1 cutover 与历史安全/UI 残项仍分别跟进。持续目标通过当前会话推进，没有安装无人值守后台调度器。 + ### 2026-10-20 - [部署] Web 视觉精修已上 HQ：Release/arm64 从 main `881d3ffa` 打包为 `remote-server-main-881d3ffa-20261020`（verify-only 通过），receiver 只改 `ProgramArguments[0]`，plist 已备份。第一次 bootstrap 报 `5: Input/output error` 且任务被卸下，03:36:24Z 重试成功，新 PID 79714。`/v1/health` 200，新 CSS 已在线上，调色板未变，本地渲染确认徽标、卡片和转录气泡生效。回滚：用备份 plist bootstrap，旧包保留。详见 `CHANGELOG.md`。 diff --git a/docs/TODO.md b/docs/TODO.md index 474c22912..fd0f3918f 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -3,7 +3,51 @@ Confirmed engineering tasks. Product-level work belongs in `docs/roadmap.md`; verification and low-priority follow-ups belong in `docs/followups.md`. -## Open +Deployment update (2026-10-25): PR #496 runtime source `fb9538a` passed CI and +is running on HQ index/Web, Daily Collector and the local paired Service (build +1779). App-only follow-up `a9e59d50` passed all 14 executed CI checks and is now +installed as Developer ID build 1782, fixing duplicate popover date-group IDs. +Local IPC and HQ health passed; hosted UI smoke passed 14 tests and screenshot +comparison. Full App UI remains unverified. +The dated local-only receipts below are historical; real-host acceptance is +limited to the checks recorded in CHANGELOG. Collector quiet-source acceptance +passed on 2026-10-25: 61 samples/30 minutes, CPU 1.921%, RSS 58.5 MiB, 16/16 roots +converged throughout. Active-write/startup performance and full App UI remain open. + +## Current execution status (2026-10-23) + +Fresh routing: [development reconciliation](reviews/2026-10-23-development-reconciliation.md). +The detailed dated entries below preserve history; their presence does not +make a closed item open again. Local baseline and live remote main were both +`5216ff3a`. Current development is in `codex/leader-loop-20261023`. + +| Item | Current status | Remaining verifier | +|---|---|---| +| Collector CPU | Quiet-source 30-minute window passed on deployed `fb9538a`; active-write/startup acceptance remains open | 61 samples over 1800 s: mean CPU 1.921%, max RSS 58.5 MiB, all 16 roots converged and no new publications. Small CPU margin; next measure a representative active-source workload. Details in CHANGELOG and `collector-idle-turn-cost-1`. | +| W4.1 alias reconciliation | Closed, superseded by D7(c)/D9 | Reopen only as a separately scoped old-DB user-state import. | +| W4.5 old-receipt bootstrap | Closed, owner won't-do | Preserve retained old DB/archive; no automatic reimplementation. | +| W4.7 epoch operator command | Implemented | Authorized HQ binary/CLI readback and real recovery; authority-file runbook remains. | +| W6.2 real-binary gate | Closed, recorded gate exists | Hosted [39098816343](https://github.com/bbingz/engram/actions/runs/39098816343) rechecked success; not a real-host acceptance receipt. | +| App Open Web Reader | Implemented | Set/verify each intended host's URL under host-change authorization. | +| P2 duplicate quarantine | Implemented, deployment recorded | Real cross-machine duplicate remains unexercised. | +| P3 install tooling | Implemented | First real-host installation/upgrade/rollback receipt. | +| D7 legacy local rows | Closed, purge recorded | Do not rerun destructive action. | +| `cutover-role-surface-1` | Fixed locally | RED: 8 expected catalog assertions failed. GREEN: 215 MCP executable tests passed; not pushed/deployed. | +| Collector fixture race/leak | Fixed locally | 85 CollectorRuntime tests plus the affected publication test passed; hosted flake recurrence not measured. | +| `collector-blocked-root-backoff-1` | Fixed locally | Final 87 runtime tests passed after behavioral RED and scheduler negative control. Per-root 1–60 second monotonic retry preserves healthy-root progress and recovery/reset/restart; no real-host CPU claim. | +| Round-12 secrets / live navigation / favorites | Existing fixes verified locally | 210 selected App tests passed with an isolated home; source and exact coverage in the reconciliation. Transcript apply concurrency remains a narrower verification item. | +| Web project audit actor | Fixed locally for four project commands | HTTP/auth/IPC 60, service/registry 29 and native batch/store 34 tests passed after RED. Follow-up real socket/dispatcher regression caught a missing IPC allowlist; fixed and 18 service tests passed. The separate alias audit batch below closes storage; credential throttling remains an accepted design residual. Not deployed. | +| Web alias audit | Implemented locally | 13 Service, 46 HTTP/client/auth and 20 migration tests passed after RED. Audit and exact-pair mutation share one transaction; bounded last-10,000 history stores published identities. Upgrade Service before RemoteServer; host acceptance remains. | +| Human keyword ranking | Fixed locally; broader Lifecycle 3.4 remains partial | All terms contribute before LIMIT; App/Service CJK routing aligned. 72 behavioral RED assertions, then 163 App and 63 Service tests passed. Source/project/time/origin/value-band facets now exist; lifecycle facets and weighting remain roadmap decisions. See the ranking contract and reconciliation. | +| Web AI endpoint/key reuse | Fixed locally | Atomic rejection protects existing/dormant credentials across settings, env, sidecar and Keychain sources, including shared embedding fallback. 26 service, 24 HTTP/IPC/UI-route and 181 JS behavior tests passed after RED. No live Keychain or deployment acceptance. | +| `session-detail-id-3` | Fixed/verified locally for reproduced scope | Mounted view exposed stale paging cleanup and duplicate scheduling; per-load ownership and synchronous busy state fix both. Five mounted tests passed five iterations; final-harness negative control failed 17 expected assertions; restored final 72 transcript tests passed. Initial success/error and delayed indexing cancellation also verified. No full UI/production acceptance. | + +Remaining historical residuals require current repro before promotion; +remaining Web hardening and measured performance follow the bounded queue +in the reconciliation. Product choices stay in `roadmap.md`. Production changes, +push/merge and releases require separate authorization. + +## Dated engineering ledger ### Collector / central index cutover gaps (added 2026-10-02) @@ -62,6 +106,11 @@ authorizes host changes. The comparable quiet-host 30-minute window (CPU <= 2%, RSS <= 150 MiB) still needs to be taken once that workload ends; record it here and in `CHANGELOG.md`. + 2026-10-25: the missing quiet-source window now passed on deployed `fb9538a`, + PID 98680, unchanged 10 s poll: 07:16-07:46Z, 61 samples/1800 s, CPU 1.921%, + max RSS 58.5 MiB, all 16 roots converged throughout, no root revision changes + or new ACKs. This closes the quiet-source verifier only; active-write/startup + acceptance remains open. Evidence in CHANGELOG and the retirement checklist. - **Alias reconciliation (plan W4 item 1, design section 4).** Not started. The committer refuses any occupied proposed ID, so pre-cutover local rows, their insights and parent links stay in the local-ID namespace and are not visible diff --git a/docs/engram-lifecycle-upgrade-plan.md b/docs/engram-lifecycle-upgrade-plan.md index 70656048b..0f804188d 100644 --- a/docs/engram-lifecycle-upgrade-plan.md +++ b/docs/engram-lifecycle-upgrade-plan.md @@ -211,6 +211,13 @@ execution queue; `parked-see-roadmap` items are listed in ### 3.4 人工检索升级（enabler 1） +> Current-source reconciliation (2026-10-23): the proposals below are historical. +> FTS rank, highlighted snippets, source/project/time/origin filters and Service +> CJK trigram ranking already exist. The local development loop implements the +> [all-term ranking and offline CJK consistency contract](superpowers/specs/2026-10-23-keyword-ranking-design.md). +> Value/lifecycle facets and score weighting remain open in the roadmap; native +> sqlite-vec remains excluded by current repository instructions. + **目标**：把"已为 AI 接好"的检索能力补给人，CJK 用户拿到真排序。 **方案设计**： @@ -241,6 +248,17 @@ execution queue; `parked-see-roadmap` items are listed in ### 3.6 数据模型与规模支撑（enabler 3） +> Measurement tooling implemented locally (2026-10-23): +> `python3 scripts/db/measure-database.py --database /explicit/path/index.sqlite` +> emits aggregate JSON to stdout (redirect to the desired report file). +> It reads one transaction with committed WAL, reports dbstat object bytes/FTS +> share, tier counts and non-skip score histogram/p10/p33/p50/p67/p90. No default +> user database, checkpoint, optimize or migration. `--timeout` defaults to 30s; +> failed reads exit nonzero without partial JSON. Missing dbstat is explicit. +> SQLite may create/use WAL coordination sidecars; for an unchanged directory, +> use a pre-existing consistent snapshot. Fixture validation is not a current +> production distribution or authorization to retune thresholds. + **目标**：用实测数据驱动决策，加必要索引，支撑价值/归档/老化查询。 **方案设计**： diff --git a/docs/followups.md b/docs/followups.md index ca09399ad..437a8370c 100644 --- a/docs/followups.md +++ b/docs/followups.md @@ -3,6 +3,25 @@ Follow-ups are verification gaps, low-priority refactors, or items that need real data, UI exercise, or product confirmation before becoming TODOs. +Deployment update (2026-10-25): green PR #496 runtime source `fb9538a` is deployed +to HQ index/Web, Daily Collector and the local paired Service (build 1779). +App-only follow-up `a9e59d50` passed all 14 executed CI checks and is installed +as build 1782, fixing duplicate popover date-group IDs. The signing blocker is +resolved; installed bytes, local IPC, HQ health and hosted UI smoke passed. +Local screenshot access was unavailable. This supersedes earlier +"not pushed/deployed" notes for those roles. Daily quiet-source CPU/RSS passed a +30-minute window on 2026-10-25 (1.921%, 58.5 MiB); full UI, business-flow and +active-write/startup performance acceptance remain open. See CHANGELOG for receipts. + +## Current routing (2026-10-23) + +[Full development reconciliation](reviews/2026-10-23-development-reconciliation.md) +routes every nonterminal family, historical plan and current dependency PR. +Rows mix source work, recorded deployments and unverified host acceptance; +never count all rows as open defects. Three local items are closed: +`cutover-role-surface-1` (215 MCP executable tests), collector fixture +race/cleanup (85 runtime tests plus the affected publication test). + ## Collector / central index cutover follow-ups (2026-10-02) Operational observations from a read-only HQ audit on 2026-10-02, independently @@ -14,20 +33,20 @@ decisions are in `docs/roadmap.md`, engineering gaps in `docs/TODO.md`. | `cutover-feed-stalled-1` | high/ops | HQ index-role Service capture ingest; collector host | **Resolved on the collector host 2026-10-02 13:10 UTC.** Three fixed builds were deployed that day; with the third (`81beb879`) all 16 roots converged, the process stays up and idle, and the backlog reached HQ. Kept for the record and because a reboot of that host has not been exercised since. Original finding: the HQ central index has received no capture arrivals since 2026-09-22 02:21 UTC, although its ingest poller keeps running. Exactly one machine has ever fed it. Cause, confirmed read-only on the collector host: a macOS update reboot renumbered the data volume's `st_dev`, the collector's stored root identity includes `st_dev` (`CollectorPOSIXRootEnumerator.swift`), every root now fails `rootIdentityChanged`, and the runtime treats that as an unavailable source and silently re-suspends all roots instead of rebinding or exiting (`CollectorRuntime.swift` `isUnavailableSource`). The process stays alive, both replicas are reachable, and source files keep changing uncaptured. Any collector host can hit this after a reboot that changes mount order. The fix (`docs/superpowers/specs/2026-10-02-collector-volume-identity-design.md`, invariant "Collector Root Binding Survives Device Renumbering Only") merged in PR #447 and, together with the replay-loss and vanished-directory fixes of PR #448 and #449, was deployed to Daily the same day. On 2026-10-07 the collector had run 4 days 19 hours without exit, 16/16 roots converged, and HQ held capture-ingested sessions up to that morning. | | `cutover-hq-local-gap-1` | high/ops | HQ host: index-role Service and the older local-role Service | **Open.** HQ's own Claude Code and Codex sessions are not in the central index: HQ runs no collector, and those sources are disabled for the index-role local scan. They are indexed only by the older local-role Service on HQ into its own DB. That Service is not stalled: an apparent gap after 2026-09-29 was a period with no new HQ transcripts, and sessions created on 2026-10-02 were indexed by its next periodic cycle (roughly hourly, no file watcher). The long-term HQ-local path is an open decision in `docs/roadmap.md`. | | `codex-resume-window-1` | medium | `macos/Shared/EngramCore/Adapters/SessionAdapterFactory.swift` `recentActiveAdapters` | Found while checking the item above. The periodic cycle sees Codex rollouts only in the today/yesterday date directories, so a rollout resumed and appended after its start-date directory leaves that window is not re-indexed until the Service restarts and runs a full scan. On HQ 17 rollouts (about 530 MB of unindexed content) were under-indexed on 2026-10-02. Other file-backed sources use an mtime cutoff. Fixed in source on 2026-10-02 (`CodexAdapter(modifiedSince:)`, two `_repro` tests); rollouts already behind and last modified more than two days ago still need one Service restart or full sweep, and the fix is not deployed. | -| `collector-stuck-dirty-locators-1` | medium | `macos/EngramCollectorCore/CollectorInventoryOwner.swift`; capture worker | 2026-10-19: the vanished, never-captured half is fixed (invariant "Collector Vanished Uncaptured Locators Leave The Dirty Set"). A locator whose source cannot be opened and that was never captured is acknowledged without a capture, so it leaves the dirty set; `upsertDirty` re-dirties it if the path is observed again. A locator captured before it vanished keeps the one-second retry, and a present but empty or entrypoint-less source keeps its retry. Remaining: the 255 Copilot and 6 Kimi members that exist but whose group capture defers (cause inferred, not verified). | -| `collector-blocked-root-backoff-1` | medium | `macos/EngramCollectorCore/CollectorRuntime.swift`; `CollectorEventCoordinator.swift` | A root whose bootstrap is blocked is restarted every turn and each blocked walk counts as a loss, so it cycles at turn rate. It is now reported on stderr after five consecutive blocked steps, but there is no back-off. | +| `collector-stuck-dirty-locators-1` | medium | `macos/EngramCollectorCore/CollectorInventoryOwner.swift`; capture worker | 2026-10-19: the vanished, never-captured half is fixed (invariant "Collector Vanished Uncaptured Locators Leave The Dirty Set"). A locator whose source cannot be opened and that was never captured is acknowledged without a capture, so it leaves the dirty set; `upsertDirty` re-dirties it if the path is observed again. A locator captured before it vanished keeps the one-second retry, and a present but empty or entrypoint-less source keeps its retry. Remaining: the 255 Copilot and 6 Kimi members that exist but whose group capture defers (cause inferred, not verified). Read-only 2026-10-25 metadata: the six Kimi files are empty; 255 Copilot files exist and are nonempty, but the entrypoint deferral cause remains unverified. Fourteen additional, previously captured Codex locators are absent and retain the existing retry policy. No records were changed. | +| `collector-blocked-root-backoff-1` | closed locally | `macos/EngramCollectorCore/CollectorRuntime.swift`; `CollectorRuntimeTests.swift` | Fixed 2026-10-23: blocked bootstrap retries per root at 1, 2, 4, 8, 16, 32, then at most 60 seconds using a monotonic clock. Cooling roots do not force capture turns; healthy roots and independent uploads continue. Completed scans reset delay; fresh runtime retries immediately. Behavioral RED, scheduler negative control, and final 87-test GREEN recorded in CHANGELOG. Recovery may wait the remaining cooldown plus scheduler/work latency. Not pushed/deployed; real-host CPU acceptance remains open. | | `collector-start-cost-1` | medium/perf | `macos/EngramCollectorCore/CollectorEventCoordinator.swift` | Every process start forces a full walk of every root, and a new process tries each stale event checkpoint once before falling back to a walk. A busy root can also repeat walks if its event queue overflows during recovery. Likely contributors to the missed CPU target; not measured after the 2026-10-02 fixes. 2026-10-18: the 30-minute Daily window excluded startup walks (process up 10 h, no re-walk during the window); the start cost remains unmeasured. | | `collector-epoch-change-scope-1` | low | `macos/EngramCollectorCore/CollectorRuntime.swift` | A changed FSEvents epoch for one root stops the whole runtime, including uploads for other roots. Decide whether it should suspend only that root, as an identity change does. **Decided 2026-10-18 (owner):** suspend only that root, as an identity change does; the root stays fail-closed until the operator runs `EngramCollector --reconcile-native-epoch`, and other roots keep capturing and uploading. Implemented in PR #481 (`c157caeb`) (`docs/superpowers/specs/2026-10-18-collector-epoch-rotation-and-root-scope-design.md`, invariant "Native Epoch Reconciliation Is An Explicit Operator Action"); not deployed. Behaviour change for operators: an epoch change no longer ends the process or causes a launchd restart loop, so it shows only as the stderr line `source roots suspended for native epoch change`, the `--once` deferred count and `--reconcile-native-epoch --dry-run`. Deployed to Daily 2026-10-18 (`ea2bbb57`, package `collector-epoch-proof-reuse-20261018`); the pre-swap dry run reported 0 affected roots. | | `collector-epoch-producer-1` | medium | `macos/EngramCollectorCore/CollectorInventoryStore.swift` `collector_streams`; `macos/EngramCoreWrite/CaptureIngest/CaptureIngestSourceRegistry.swift` `provision` | **Open; owner decided 2026-10-18 (end of this row).** The collector never rotates an epoch under an existing source instance: a `collector_streams` row is created once per root, root revision and source with fresh UUIDs for both `source_instance_id` and `collector_epoch`, and later writes only advance `last_sequence`. A spool reset or a root re-enrollment that bumps the revision therefore reaches HQ as a new, unregistered source instance on an already registered root: its publications stay pending and are never claimed, and adding it to the source-authority file makes `CaptureIngestSourceRegistry.provision` throw `overlappingRoot` (UNIQUE machine/source/root), which aborts service-index startup. The new `collectorApproveEpoch` command keeps the source instance, so it cannot recover this case. Owner to decide: (a) the collector keeps the source instance and rotates only the epoch, making `collectorApproveEpoch` the recovery path, or (b) HQ gets a source-instance retire/rebind design. Until then the command covers only a same-instance epoch change that no current producer emits. **Decided 2026-10-18 (owner): option (a).** The only automatic path to a new stream row is a root configuration revision bump, and the collector inventory survives it, so keeping the instance and rotating only the epoch covers it. The collector has no code path that deletes its inventory; a manual spool wipe loses the instance ID and stays a documented operator limitation, with no HQ retire/rebind design for now. Implemented in PR #481 (`c157caeb`) (invariant "Collector Root Revision Bump Keeps The Source Instance"): the instance is kept only when the root's source and byte-exact path are unchanged; HQ needs no code change because the publication envelope carries no root revision and every sequence key includes the epoch. Not deployed. Deployed to Daily 2026-10-18 (`ea2bbb57`); every Daily root is still at revision 1, so the carry has not run on a real host yet. | | `collector-epoch-authority-file-1` | medium/ops, owner decision | `macos/EngramCoreWrite/CaptureIngest/CaptureIngestSourceRegistry.swift` `provision`; `macos/EngramService/Core/EngramServiceRunner.swift` startup provisioning | Found 2026-10-18 while implementing `collector-epoch-producer-1`. The service-index provisions from the source-authority file on every start, and `provision` accepts a known source instance only when the file's `initialEpoch` equals its current approved epoch. After any `collectorApproveEpoch`, an unchanged file therefore throws `sourceInstanceConflict` and aborts the next service-index start. Present since PR #477; no approval has run on HQ yet. Mitigation today is a runbook step (update `initialEpoch` to the approved epoch before the next restart, in the 2026-10-18 epoch rotation design). Code option for an owner decision: let `provision` accept a file epoch that the epoch history shows was superseded by an operator approval. **Decided 2026-10-18 (owner delegated the call):** runbook only, no HQ code change now. The change would alter an authority check and needs another service-index restart, and no approval has run yet; revisit before the first real `collectorApproveEpoch`. | | `collector-revision-bump-staged-1` | low | `macos/EngramCollectorCore/CollectorInventoryStore.swift` publication claim and reservation recovery | Found 2026-10-18 (pre-existing). Claiming, claim revalidation and reservation recovery join on the root's current `root_revision`, so publications staged under the previous revision are never uploaded after a revision bump. Runbook: drain the root (no pending publications) before bumping its revision. | | `collector-publication-cost-1` | medium/perf | `macos/EngramCollectorCore` capture and publication path | 2026-10-18 Daily window (CHANGELOG.md 2026-10-18): CPU bursts of 3-9 s per 30 s coincided with 35 Claude Code transcript publications while the quiet floor was 1.5-3.5%, so the <= 2% target fails on activity, not on polling. Hypothesis: each append re-reads and hashes the whole transcript. Unprofiled; a `sample` attach on Daily needs owner approval. 2026-10-18 source read: `ExactSourceCapturer.streamStableSource` does re-read the whole file and hash it, and every chunk is staged to a temp file with fsyncs, but `rawChunkSize` is 8 MiB and local Claude Code transcripts are mostly under 5 MB, so hashing alone cannot explain seconds of CPU per publication. The hotspot is being measured locally with synthetic transcripts before any fix; Daily stays untouched until a deploy is authorized. 2026-10-18 local measurement (Release binaries, synthetic Claude Code transcripts, loopback replicas, `sample` of a local process): the dominant per-publication cost was `CollectorPrivacyProof.assess` re-run for each replica (69% of busy samples at 20 MB; inside it the per-byte newline scan 36% and per-line JSON parsing 51%). Fixed in PR #482 (`059fe3b9`): one eligible proof per capture serves both replicas (reuse still requires a fresh policy and `isCurrent` before every request), and the default JSONL line scan uses `memchr`. Collector CPU per publication above the quiet floor: 2 MB 0.146 -> 0.117 s, 8 MB 0.240 -> 0.167 s, 20 MB 0.437 -> 0.236 s. **Still open:** Daily's excess was about 1.2-2.1 s per publication, 3-17x more than measured here, and Daily's quiet floor is 1.5-3.5% against 0.1-0.33% locally. Next lead: a fixed per-turn cost (`advanceInventory` doubled from 1 to 16 roots, `validateStorage` 31-40% of busy samples at 2 MB). Confirming it needs a `sample` of the Daily collector during a burst (owner approval) after this fix is deployed. Not done: seven other per-byte newline scans (`CollectorPrivacyProof.swift`, `CollectorCopilotSource.swift`, `CollectorGeminiSource.swift`) and the triple CAS read per replica in `transmit`. Deployed to Daily 2026-10-18 (`ea2bbb57`): publications are ACKed by both replicas and ingested at HQ with the shared proof, but the 30-minute window (5.847%, 26 publications) cannot resolve its effect because the idle floor dominates; see `collector-idle-turn-cost-1`. The remaining publication-path items above are now low priority. | -| `collector-idle-turn-cost-1` | medium/perf | `macos/EngramCollectorCore/CollectorPublicationWorker.swift` `captureOnce` (`reconcileCursorLegacyObservations`, `drainUnavailableDeferrals`, `reconcileCapturedDependencies`, `reconcileConfiguredRegistries`); `CollectorCursorSource.discoverModern`; `CollectorCursorLegacyOwnership.ObservationRead.page` | Found 2026-10-18 on Daily after deploying `ea2bbb57`. The 30-minute window (12:01-12:31Z, 16 roots converged before and after) measured CPU 5.847% with a median 30 s interval of 3.8%, so the idle floor, not publication bursts, keeps Daily above 2%. A 90 s `sample` of PID 6028 with no capture in it (2.8 s CPU, about 3.1%) attributes sampled busy stacks inside `captureOnce` to: Cursor legacy observation reconciliation 31% (`discoverModern` 801 samples, legacy observation paging 558, applying observations 171), `deferClaims` 13%, captured-dependency checks 11% (Cursor and VS Code `capturedDependenciesChanged`, catalog reads), `claimDirty` 5%, Gemini registry reconcile 5%, Copilot and Kimi groups 8%. Leaves are mostly `openat`, `fstatat` and `getdirentries64`, i.e. every turn re-opens and re-stats many paths. The 2026-09-12 hint cache and 10 s gate made the Cursor scan less frequent, not cheaper (about 9 calls of about 90 ms in 90 s). Suggested order: Cursor legacy observation paging and `discoverModern` cost, then the stuck-dirty deferrals (`collector-stuck-dirty-locators-1`), then captured-dependency checks. Raw sample in the session scratchpad. 2026-10-19: a 15 s `sample` of the Daily collector attributed the per-turn floor to `CollectorInventoryOwner.ancestorIdentities` inside `validateStorageFilesystem`, not to directory enumeration. Two fixes landed on `perf/collector-idle-turn-cost-20261019`, not yet deployed: a finished bootstrap deletes that root's frontier rows (the table had grown to about 12 M rows), and the storage-separation ancestor walk is proven once per open and re-walked only when an ancestor's identity or mode changes (invariants "Collector Frontier Rows Live Only For The Active Scan" and "Collector Storage Separation Is Proven Once Per Open"). The per-entry `validateDirectory` calls were kept; they are a pinned safety check and were not the measured hotspot. Still open and unchanged: the Cursor legacy observation reconciliation, which was the largest share of the 2026-10-18 sample. Deployed to Daily 2026-10-19 as `collector-idle-turn-20261019` (commit `81bf3f1a`): all three fixes verified live (frontier 12,130,812 -> ~1,000 rows; a 60 s `sample` shows only cached `ancestorChainHolds` and no full re-walk; stuck-dirty locators 414+ -> 266, the remaining members are the present-but-uncapturable Copilot/Kimi ones from `collector-stuck-dirty-locators-1`). The 07:59-08:25Z window read 34.6% but is NOT comparable: an active multi-agent Claude Code workflow wrote `~/.claude/projects` continuously, `daily-claude` never converged (lag 2 -> 48), and 0 of 51 intervals were below 2%. That window attributes the churn to the pre-existing per-scan walk cost under live workload, not to the idle floor. The comparable quiet-host window is still to be taken. | -| `collector-settings-write-flake-1` | low/test | `macos/EngramServiceCoreTests/CollectorRuntimeTests.swift` `RuntimeFixture.writeSettings`; `testInvalidFreshSettingsTerminateWaitAndNeverResumePublicationAutomatically` | Seen once on hosted CI (PR #481, run 39114739767): the test expected `invalidConfiguration` and got `invalidSettings`. The fixture rewrites settings with a non-atomic `Data.write`, so a runtime reload between truncate and write reads an empty file, fails JSON parsing and reports `invalidSettings`. 20/20 local passes; not seen in the 15 earlier failed CI runs. An atomic write is not a drop-in fix because the replacement file is briefly not 0600 and the runtime's permission check would then fail differently. | -| `collector-test-fixture-leak-1` | low/test | `macos/EngramServiceCoreTests/CollectorPublicationWorkerTests.swift` `testFreshOnlyReservationDoesNotReuseAnExistingCapture` | Found 2026-10-18: the test never calls `f.remove()`, so each run leaves a `.engram-publication-test-*` directory behind. | -| `cutover-install-tooling-1` | medium | `scripts/plan-headless-install.mjs`; `macos/*/Packaging/` templates | A new Mac cannot be set up as a collector from the repo alone: there is no settings-block template, identity initialization step, credentials provisioning, or apply/upgrade/rollback mode, and no repo script creates the HQ index-role job. The dry-run plan also reports every role's job as disabled with RunAtLoad/KeepAlive off, while the remote-server plist template sets RunAtLoad and KeepAlive to true. | -| `cutover-web-editor-hardening-1` | medium/security, needs triage | `macos/EngramRemoteServer/Core/WebAuthSessionStore.swift`; `WebWriteRoutes.swift` | Web login throttling is one global five-attempts window with no per-client partition, so anyone who can reach the origin can delay the editor's login. Unverified risk: an editor can change AI/embedding base URLs while stored keys stay in place. Project moves from Web are audited with actor `mcp`. `docs/invariants.md` now records Web authority; these behaviors are not guarantees there. Since 2026-10-08 (`docs/superpowers/specs/2026-10-08-web-tailscale-identity-auth-design.md`) a deployment can set `ENGRAM_REMOTE_WEB_AUTH=tailscale-serve`, where no credential or throttle exists and the session records the tailnet login; the audit actor stays `mcp` until that design's phase 2 lands. | -| `cutover-role-surface-1` | low | `macos/EngramMCP/Core/MCPToolRegistry.swift` | In `collector`/`replica`/invalid roles MCP still advertises the full tool catalog while every call returns the unavailable message. | +| `collector-idle-turn-cost-1` | medium/perf | `macos/EngramCollectorCore/CollectorPublicationWorker.swift` `captureOnce` (`reconcileCursorLegacyObservations`, `drainUnavailableDeferrals`, `reconcileCapturedDependencies`, `reconcileConfiguredRegistries`); `CollectorCursorSource.discoverModern`; `CollectorCursorLegacyOwnership.ObservationRead.page` | Found 2026-10-18 on Daily after deploying `ea2bbb57`. The 30-minute window (12:01-12:31Z, 16 roots converged before and after) measured CPU 5.847% with a median 30 s interval of 3.8%, so the idle floor, not publication bursts, keeps Daily above 2%. A 90 s `sample` of PID 6028 with no capture in it (2.8 s CPU, about 3.1%) attributes sampled busy stacks inside `captureOnce` to: Cursor legacy observation reconciliation 31% (`discoverModern` 801 samples, legacy observation paging 558, applying observations 171), `deferClaims` 13%, captured-dependency checks 11% (Cursor and VS Code `capturedDependenciesChanged`, catalog reads), `claimDirty` 5%, Gemini registry reconcile 5%, Copilot and Kimi groups 8%. Leaves are mostly `openat`, `fstatat` and `getdirentries64`, i.e. every turn re-opens and re-stats many paths. The 2026-09-12 hint cache and 10 s gate made the Cursor scan less frequent, not cheaper (about 9 calls of about 90 ms in 90 s). Suggested order: Cursor legacy observation paging and `discoverModern` cost, then the stuck-dirty deferrals (`collector-stuck-dirty-locators-1`), then captured-dependency checks. Raw sample in the session scratchpad. 2026-10-19: a 15 s `sample` of the Daily collector attributed the per-turn floor to `CollectorInventoryOwner.ancestorIdentities` inside `validateStorageFilesystem`, not to directory enumeration. Two fixes landed on `perf/collector-idle-turn-cost-20261019`, not yet deployed: a finished bootstrap deletes that root's frontier rows (the table had grown to about 12 M rows), and the storage-separation ancestor walk is proven once per open and re-walked only when an ancestor's identity or mode changes (invariants "Collector Frontier Rows Live Only For The Active Scan" and "Collector Storage Separation Is Proven Once Per Open"). The per-entry `validateDirectory` calls were kept; they are a pinned safety check and were not the measured hotspot. Still open and unchanged: the Cursor legacy observation reconciliation, which was the largest share of the 2026-10-18 sample. Deployed to Daily 2026-10-19 as `collector-idle-turn-20261019` (commit `81bf3f1a`): all three fixes verified live (frontier 12,130,812 -> ~1,000 rows; a 60 s `sample` shows only cached `ancestorChainHolds` and no full re-walk; stuck-dirty locators 414+ -> 266, the remaining members are the present-but-uncapturable Copilot/Kimi ones from `collector-stuck-dirty-locators-1`). The 07:59-08:25Z window read 34.6% but is NOT comparable: an active multi-agent Claude Code workflow wrote `~/.claude/projects` continuously, `daily-claude` never converged (lag 2 -> 48), and 0 of 51 intervals were below 2%. That window attributes the churn to the pre-existing per-scan walk cost under live workload, not to the idle floor. The comparable quiet-host window is still to be taken. 2026-10-23 local isolated Release profiling found digest hex formatting in 699/3738 sampled stacks. `ArchiveV2Hash.sha256` now encodes hex with a byte lookup table. A 2048-workspace/64-item page fell from 6.824 to 5.245 median process CPU ms (23.1%); 1-item pages fell only 3.1%, so directory enumeration/stat work remains. Synthetic fixtures only, not Daily CPU acceptance. Receipt: `/tmp/engram-leader-cursor-perf-20261023/`; full measurements and limits in CHANGELOG. 2026-10-25 quiet-source acceptance passed on deployed `fb9538a`: 61 samples/1800 s, CPU 1.921%, max RSS 58.5 MiB; all 16 roots converged, no revisions/new ACKs, PID/binary/log sizes stable. The missing quiet-window verifier is closed. Active-write/startup acceptance and the small CPU margin remain; do not infer an individual optimization effect from different workloads. Receipt: CHANGELOG and the retirement checklist. | +| `collector-settings-write-flake-1` | closed locally | `macos/EngramServiceCoreTests/CollectorRuntimeTests.swift` `RuntimeFixture.writeSettings` | Fixed 2026-10-23: encode into a unique sibling file, chmod 0600 before atomic rename, and defer staging cleanup. Readers never see the truncate/write or replacement-permissions window. All 85 runtime tests passed, including invalid fresh-settings reload. The original hosted flake was not reproduced this session; hosted recurrence is unmeasured. | +| `collector-test-fixture-leak-1` | closed locally | `macos/EngramServiceCoreTests/CollectorPublicationWorkerTests.swift` `testFreshOnlyReservationDoesNotReuseAnExistingCapture` | Fixed 2026-10-23 with `defer { f.remove() }`; the affected test passed. No historical temporary directories were removed. | +| `cutover-install-tooling-1` | medium/ops | `scripts/plan-headless-install.mjs`; `scripts/render-collector-settings.mjs`; `macos/*/Packaging/` | P3 tooling merged in #454 (`63e0a17e`) and is on current main. The old missing-template/apply/rollback claim is superseded. First real-host install, credential provisioning, upgrade and rollback remain unverified and require host-change authorization. | +| `cutover-web-editor-hardening-1` | partial; remaining security/design triage | `macos/EngramRemoteServer/Core/WebAuthSessionStore.swift`; `WebWriteRoutes.swift` | Credential-mode login throttling remains one global five-attempt window (47 current auth tests passed on 2026-10-23). The accepted identity-auth design explicitly parks per-client credential throttling; define trusted proxy/client provenance plus a global guessing budget before changing it. Credential mode has no trusted client-address contract and must not trust arbitrary forwarded headers. Endpoint/key reuse was reproduced and fixed locally on 2026-10-23: a locked atomic pre-write check protects relevant settings/env/sidecar/Keychain credentials, including dormant keys and shared embedding fallback; unknown secret state fails closed. Typed IPC refusal maps to HTTP 409 with a correctable not-saved draft. GREEN: 26 service, 24 HTTP/client/UI-route and 181 JS tests; no live Keychain or deployment claim. Project move/archive/undo/batch actor propagation is implemented locally on 2026-10-23: identity from the editor session, credential fallback web-editor, strict browser actor rejection, bounded IPC field and actor-aware replay fingerprints. Continuation fixed the omitted service-dispatch actor allowlists after a real socket regression failed; 18 service tests then passed. Full boundary evidence is in CHANGELOG and the design addendum. Alias audit implemented locally on 2026-10-23 under the new [storage design](superpowers/specs/2026-10-23-web-alias-audit-design.md): session actor, published identities, same-transaction recording and last-10,000 retention; no-op/refused requests stay silent. Audit insert/prune failure rolls back the mutation. 13 Service, 46 HTTP/client/auth and 20 migration tests passed. New RemoteServer requires the updated Service; upgrade Service first. No permanent/native-alias audit claim. No deployment claim; identity-mode auth does not resolve credential-mode throttle behavior. | +| `cutover-role-surface-1` | closed locally | `macos/EngramMCP/Core/MCPToolRegistry.swift` | Fixed 2026-10-23 on `codex/leader-loop-20261023`: collector/replica/invalid roles return an empty tools catalog before DB access; direct calls retain role guidance. Local/index/missing-role catalogs preserve 27 tools and semantic availability. RED: 8 expected assertions; GREEN: all 215 MCP executable tests passed. Not pushed or deployed. | ## HQ live ingest residuals (2026-08-24) @@ -70,19 +89,20 @@ SLA or the reboot/degraded-path checks below. ## Round-12 parked residuals (2026-08-24) The final Round-12 fix pass stopped after the thirteen merge-blocking packages -and a bounded set of adjacent high-severity fixes. The rows below remain open -and are **not claimed fixed**. They are parked for a later owner-selected pass; -this list is not a public-release readiness declaration. +and a bounded set of adjacent high-severity fixes. Later remediation changed +some of these surfaces; the current dispositions below supersede the original +blanket open status. Local tests are not a public-release readiness declaration. | IDs | Severity | Current home | Why parked | |-----|----------|--------------|------------| -| `secrets-nofollow-1/-2/-3`, `secrets-nofollow-regress-1/-2/-3`, `apikey-empty-2`, `invariant-6-prod-2` | high/medium | `macos/Engram/Core/EngramServiceLauncher.swift:82-155`; `macos/Engram/Views/Settings/AISettingsSection.swift:768-940` | Runtime-secret directory ownership, failed mailbox publication, settings result propagation, and XCTest passwd-home isolation form one security-sensitive cluster. It was not changed after the must-fix stop condition and needs its own hermetic RED/GREEN pass. | -| `session-detail-id-1`, `session-detail-id-regress-1`, `session-detail-id-3` | medium | `macos/Engram/Views/PopoverView.swift:239`; `macos/Engram/Views/Pages/SourcePulseView.swift:434`; `macos/Engram/Views/SessionDetailView.swift:26,880-952,1656-1705` | Live-navigation completion, favorite-read generation, and transcript-apply identity fencing remain a coupled UI concurrency follow-up. | -| `mcp-hermetic-1` | medium | `macos/EngramUITests/Helpers/TestLaunchConfig.swift:64-95` | The test launcher still permits execution after an early setup failure; changing the suite-wide failure policy needs a dedicated foreground UI-test pass. | +| `secrets-nofollow-1/-2/-3`, `secrets-nofollow-regress-1/-2/-3`, `apikey-empty-2`, `invariant-6-prod-2` | closed locally for summarized scope | `macos/Engram/Core/EngramServiceLauncher.swift`; `macos/Shared/Security/SecureRegularFile.swift`; `macos/Engram/Views/Settings/AISettingsSection.swift` | Re-adjudicated 2026-10-23: current source pins no-follow file operations, validates runtime directory owner/mode, preserves unrelated custom-directory siblings, propagates failed bridge refresh and uses the injected Settings socket. Empty-key policy and default test DB path are covered. Fresh isolated App run: 210 tests passed, including 19 launcher, 32 transport, 36 AI settings and the injected-home test. Original grouped IDs are not eight independent current defects. See the dated reconciliation for exact scope and evidence; no new production fix or live-Keychain test was needed. | +| `session-detail-id-1`, `session-detail-id-regress-1` | closed locally | `macos/Engram/Views/PopoverView.swift`; `macos/Engram/Views/Pages/SourcePulseView.swift`; `macos/Engram/Views/SessionDetailView.swift` | Re-adjudicated 2026-10-23: nil live resolutions complete their captured navigation token, favorites compare a generation token, successful mutation invalidates pending reads, and disappearance clears identity. Current navigation/predicate/source-wiring regressions passed in the 210-test App run. No mounted-view stress or render acceptance is claimed. | +| `session-detail-id-3` | closed locally for reproduced/verified scope | `macos/Engram/Views/SessionDetailView.swift`; `macos/EngramTests/TranscriptSessionSwitchTests.swift` | Mounted verification on 2026-10-23 found old page/all/copy cleanup clearing a new session busy flag, and duplicate clicks before task startup creating two requests. Busy state now starts synchronously and cleanup is owned by a per-load token invalidated on switch/disappear, including A-B-A. Initial success/error, delayed indexing and page/error cancellation passed the tested interleavings. Final five mounted tests passed five iterations; removing both fixes with the final harness failed 17 expected assertions; restoring them passed 72 transcript tests. Fake transcripts and captured copy output, no exhaustive UI/real-parser/production acceptance. Evidence and failed preliminary observer runs: CHANGELOG and `/tmp/engram-leader-transcript-20261023/`. | +| `mcp-hermetic-1` | medium | `macos/EngramUITests/Helpers/TestLaunchConfig.swift:64` | Source policy already establishes isolated environment/test mode before fallible setup, and all 27 configure call sites have `continueAfterFailure = false`. Two source-policy tests passed October 23. Foreground failure-injection behavior remains UNVERIFIED: XCUITest initialization failed with LocalAuthentication Code=-4 (system authentication running), before test execution. Temporary isolated-bundle experiment restored; no product change. See the [reconciliation receipt](reviews/2026-10-23-development-reconciliation.md#ui-harness-failure-policy-adjudication). | | `live-export-terminal-1`, `live-export-terminal-2` | low/medium | `macos/Engram/Views/Resume/TerminalLauncher.swift:225` | Never-launched Warp cleanup timing and cancellation propagation for Terminal/iTerm/Ghostty remain deferred terminal-integration behavior. | -| `workitem-localtime-1` | medium | `macos/Shared/EngramCore/Indexing/ImplementationDigestExtractor.swift` `dateKey` | **Fixed 2026-09-22, local `main`, not pushed.** `localDayFormatter.timeZone` is reset to `TimeZone.autoupdatingCurrent` before every `string(from:)`. Regression: `testActionDateFollowsInProcessTimeZoneChange_repro`. | -| `xcodeproj-untracked-1`, `xcodeproj-untracked-2` | medium | `scripts/check-xcodeproj-drift.sh` | **Fixed 2026-09-22, local `main`, not pushed.** Pinned XcodeGen is required. Generation runs in a temp copy of `macos/`. A match with the working tree passes even if `project.pbxproj` is dirty versus HEAD. One-sided generated files still fail. Regression: `passes when generation matches a worktree that is dirty versus HEAD_repro`. | -| `r11-dropped-2` | medium | `macos/EngramRemoteServer/Core/EngramRemoteServerApp.swift:189,773` | Generic request-body failures are still grouped with classified oversize failures; the 413/503 split and bounded GET behavior need remote-server coverage. | +| `workitem-localtime-1` | medium | `macos/Shared/EngramCore/Indexing/ImplementationDigestExtractor.swift` `dateKey` | **Fixed on current local and remote `main` (rechecked 2026-10-23; `00bb5809`).** `localDayFormatter.timeZone` is reset to `TimeZone.autoupdatingCurrent` before every `string(from:)`. Regression: `testActionDateFollowsInProcessTimeZoneChange_repro`. | +| `xcodeproj-untracked-1`, `xcodeproj-untracked-2` | medium | `scripts/check-xcodeproj-drift.sh` | **Fixed on current local and remote `main` (rechecked 2026-10-23; `00bb5809`).** Pinned XcodeGen is required. Generation runs in a temp copy of `macos/`. A match with the working tree passes even if `project.pbxproj` is dirty versus HEAD. One-sided generated files still fail. Regression: `passes when generation matches a worktree that is dirty versus HEAD_repro`. | +| `r11-dropped-2` | closed locally; behavioral verification refreshed | `macos/EngramRemoteServer/Core/EngramRemoteServerApp.swift`; `EngramRemoteServerCoreTests/EngramRemoteServerTests.swift` | 2026-10-23: production `buildRouter()` tests inject partial failing streams and cumulative oversize bodies. Bundle PUT preserves existing ciphertext and creates no new blob on either failure; complete uploads at the limit pass. MCP asserts 503/-32603 vs 413/-32600, malformed complete JSON 400/-32700, and initialize at the exact 1 MiB limit. Auth/Origin refusals consume no body. Restoring the historical catch-all mapping failed 5 expected assertions; restored source passed all 28 server tests. No runtime code change or real transport-disconnect/production claim. Receipt: `/tmp/engram-leader-body-20261023/`. | ## OPS-ALIAS-001 — Prod path-shaped project_aliases cleanup (closed, 2026-07-21) diff --git a/docs/invariants.md b/docs/invariants.md index b32486bd0..85f6f6b5a 100644 --- a/docs/invariants.md +++ b/docs/invariants.md @@ -116,7 +116,7 @@ Invariants are properties that must survive every change; each entry names where ## Persisted Host Role Before Local Index Access -- **Statement** - The App and MCP resolve the same owner-only persisted role before local-index access. Collector, replica and invalid settings cannot open the local product DB or start a service through the App; they do not automatically migrate settings/credentials. In collector/replica the unavailable view may offer Open Web Reader only for an https `webReaderURL` read from the same owner-only settings file; the action opens that URL in the default browser and never opens the local DB or starts/adopts a service. Missing role preserves local behavior. This is a host-entry guard, not proof of a complete collector or source coverage. +- **Statement** - The App and MCP resolve the same owner-only persisted role before local-index access. Collector, replica and invalid settings cannot open the local product DB or start a service through the App; they do not automatically migrate settings/credentials. In collector/replica the unavailable view may offer Open Web Reader only for an https `webReaderURL` read from the same owner-only settings file; the action opens that URL in the default browser and never opens the local DB or starts/adopts a service. MCP tool discovery returns an empty catalog for collector, replica and invalid settings before any database probe; direct calls still return explicit role-unavailable guidance. Local/index and a missing role preserve the normal catalog and reads. This is a host-entry guard, not proof of a complete collector or source coverage. - **Enforced by** - `macos/Shared/EngramCore/RuntimeRoleSettings.swift`, `macos/Engram/Core/AppEnvironment.swift`, `macos/Engram/App.swift`, `macos/Engram/Core/Database.swift`, `macos/EngramMCP/Core/MCPConfig.swift`, `macos/EngramMCP/Core/MCPToolRegistry.swift`. - **Verified by** - `macos/EngramCoreTests/RuntimeRoleSettingsTests.swift`, `macos/EngramTests/RuntimeRoleAppTests.swift`, `macos/EngramTests/DatabaseManagerTests.swift`, `macos/EngramMCPTests/EngramMCPExecutableTests.swift` (runtime-role tests). - **Gate** - `none`. @@ -156,6 +156,13 @@ Invariants are properties that must survive every change; each entry names where - **Verified by** - `macos/EngramCollectorCoreTests/CollectorPrivacyProofTests.swift`, `macos/EngramCoreTests/Adapters/SourceMetadataProjectionParityTests.swift`, `macos/EngramServiceCoreTests/CollectorPublicationWorkerTests.swift` (testPolicyIsRefreshedImmediatelyBeforeEveryHTTPStage, testOnePrivacyAssessmentServesBothReplicasOfOneCapture_repro, testPolicyChangeBetweenReplicasReassessesAndCanWithholdTheSharedCapture). - **Gate** - `none`. +## Transcript Paging Cleanup Belongs To Its Load Task + +- **Statement** - Session detail marks paging busy before scheduling a page/all/copy task. Each such task owns a fresh token; only that token may clear busy state or the stored task handle. Session switches and disappearance invalidate ownership; late non-cooperative parse/build completions cannot release a subsequent load. A return to the same session is a new load, not the old task's ownership. Existing cancellation guards protect transcript success/error/index application, and cancelled copy work cannot write its result to the clipboard. +- **Enforced by** - `macos/Engram/Views/SessionDetailView.swift`. +- **Verified by** - `macos/EngramTests/TranscriptSessionSwitchTests.swift` (mounted initial success/error, delayed index, page/all/copy cancellation, A-B-A, immediate duplicate actions and normal completion), `macos/EngramTests/TranscriptPagingTimingTests.swift`, `macos/EngramTests/TranscriptLabelAndCopyTests.swift`. +- **Gate** - `none`. + ## Transcript Continuation Preserves Redacted UTF-8 Payloads - **Statement** - The pure Service pager redacts every string field before fragmenting normalized message JSON at UTF-8 boundaries. Cursors bind session, immutable generation, projection, redaction revision, roles, ordinal, offset and payload digest; stale or unavailable snapshots fail explicitly. Encoded envelope sizing includes Data/base64 escaping and reserves framing headroom. The pager does not itself establish snapshot authority or expose an IPC command. @@ -170,6 +177,27 @@ Invariants are properties that must survive every change; each entry names where - **Verified by** - `macos/EngramRemoteServerCoreTests/WebServerIntegrationTests.swift` (testWebConfigurationDefaultsOffAndExplicitValuesRoundTrip, testDefaultOffReturns404WithZeroFactoryCallsAndZeroRealSocketAccepts, testEnvironmentIdentityModeRequiresLoopbackBindAndNoCredential), `macos/EngramRemoteServerCoreTests/WebConfigTests.swift` (testViewerCredentialMustDifferFromEveryProvidedBearer, testOptionalEditorCredentialIsDistinctAndStoredOnlyAsDigest, testTailscaleServeModeForbidsSharedCredentialsAndRequiresViewers, testIdentityAllowlistEntriesMustBeExactPrintableLoginsWithoutDuplicates), `macos/EngramRemoteServerCoreTests/WebAuthSessionTests.swift` (testEditorAuthorityIsBoundToItsOwnSessionAndRevokedOnLogoutOrExpiry, testViewerOnlyConfigurationNeverGrantsWriteAuthority, testGlobalFiveAttemptWindowCountsBothFailedAndSuccessfulLogins, testIdentityLoginMintsViewerOrEditorByExactLoginAndRecordsActor, testEachStoreAnswersOnlyItsOwnModeAndCredentialSessionsHaveNoActor), `macos/EngramRemoteServerCoreTests/WebAliasHTTPTests.swift` (testViewerPostIs403AndDoesNotCallWriteSurface, testEditorAddPreservesPathShapedAliasText), `macos/EngramRemoteServerCoreTests/WebProjectMigrationHTTPTests.swift` (testViewerCsrfAndDisabledEditorNeverReachProjectWriter), `macos/EngramRemoteServerCoreTests/WebRelationshipHTTPTests.swift` (testViewerAndMissingOriginCannotReachAnyRelationshipWriter), `macos/EngramRemoteServerCoreTests/WebAuthRouteTests.swift` (testAuthorityMismatchDuplicateHostAndForwardedSpoofingFailClosed, testAPIMarkerMustBeSingleExactOneForEveryAPIRequest, testIdentityLoginMintsFromTheServeHeaderWithEmptyBodyAndExactOrigin, testIdentityLoginRejectsCredentialBodiesMissingDuplicateAndUnlistedLogins, testCredentialModeIgnoresForgedServeHeadersAndKeepsItsStatusBody, testIdentityViewerSessionCannotReachWriteRoutesEvenWithAnEditorHeader), `macos/EngramRemoteServerCoreTests/WebReadClientTests.swift` (testMessagesSendsOnlyHardcodedCommandTypedDataUUIDAndNoCapability, testAllowlistRejectsExhaustiveLiveHandlerInventoryBeforeAnyIPC), `macos/EngramRemoteServerCoreTests/WebWriteClientTests.swift` (testAllowlistRejectsUnknownCommandsWithoutIPC, testAddAliasAttachesLocalTokenAndNeverPutsItOnHTTPJSON, testMissingTokenDoesNotOpenIPC). - **Gate** - `none`. +## Web Project Audit Attribution Comes From The Editor Session + +- **Statement** - Web move, archive, undo and move-batch stamp the live editor session's login, or web-editor in credential mode. Browser JSON cannot set actor and later identity headers cannot replace the session-bound identity. The optional IPC actor is bounded to the configured login syntax; old Web clients without it use web-editor. A capability-protected service maps this value into the existing migration-log actor and operation fingerprint, including each batch item. Different actors cannot replay the same operation ID as the same request. Native CLI/MCP actor parsing and native batch defaults remain unchanged. Attribution is metadata, not new authority; a trusted local capability holder can still supply it. Alias auditing is outside this contract. +- **Enforced by** - `macos/EngramRemoteServer/Core/WebAuthSessionStore.swift`, `macos/EngramRemoteServer/Core/WebWriteRoutes.swift`, `macos/Shared/Service/EngramServiceWebProjectModels.swift`, `macos/EngramService/Core/EngramServiceCommandHandler.swift`, `macos/EngramService/Core/EngramServiceCommandHandler+WebProject.swift`, `macos/EngramService/Core/EngramServiceCommandHandler+ProjectMigration.swift`, `macos/EngramCoreWrite/ProjectMove/MigrationLogStore.swift`, `macos/EngramCoreWrite/ProjectMove/Batch.swift`. +- **Verified by** - `macos/EngramRemoteServerCoreTests/WebProjectMigrationHTTPTests.swift` (testProjectAuditActorComesFromSessionAndRejectsBrowserOverride_repro), `macos/EngramRemoteServerCoreTests/WebWriteClientTests.swift` (testProjectAuditActorCrossesCapabilityProtectedIPC_repro), `macos/EngramRemoteServerCoreTests/WebAuthSessionTests.swift` (testEditorAuthorityIsBoundToItsOwnSessionAndRevokedOnLogoutOrExpiry), `macos/EngramServiceCoreTests/WebProjectMigrationWriteTests.swift` (testWebProjectActorsReachAuditThroughRealIPCAndScopeReplay_repro, testWebProjectActorDecodingRejectsInvalidAttribution_repro). +- **Gate** - `none`. + +## Web Alias Changes And Their Bounded Audit Commit Together + +- **Statement** - Web alias add/remove records the authenticated editor session's actor, or web-editor for credential/older clients, only when an authorized exact pair changes. Browser actor fields are rejected. The capability-protected service writes published identities and actor to web_alias_audit_log in the same transaction as the alias mutation; audit insertion or pruning failure rolls back both. No-op/refused requests do not create rows. Keep the latest 10,000 transitions by insertion id, without retaining raw paths or request bodies. This local bounded history does not claim native alias, authentication or permanent compliance coverage. +- **Enforced by** - `macos/EngramRemoteServer/Core/WebWriteRoutes.swift`, `macos/Shared/Service/EngramServiceWebWriteModels.swift`, `macos/EngramService/Core/EngramServiceCommandHandler.swift`, `macos/EngramService/Core/ServiceWebMetadataProducer.swift`, `macos/EngramCoreWrite/Database/EngramMigrations.swift`. +- **Verified by** - `macos/EngramRemoteServerCoreTests/WebAliasHTTPTests.swift` (testAliasActorIsBoundToSessionAndBrowserOverrideIsRejected_repro), `macos/EngramServiceCoreTests/WebAliasWriteTests.swift` (testAliasActorsReachAuditThroughCapabilitySocketAndDispatcher_repro, testAliasAuditRecordsOnlyTransitionsAndPublishedIdentities_repro, testAliasAuditInsertFailureRollsBackAddAndRemove_repro, testAliasAuditRetentionAndPruningFailureAreTransactional_repro), `macos/EngramServiceCoreTests/CollectorWebDemoTests.swift` (testAliasEditorThroughHTTPAndServiceIPC), `macos/EngramCoreTests/Database/MigrationRunnerTests.swift` (testWebAliasAuditMigrationIsAdditiveAndPreservesHistory). +- **Gate** - `none`. + +## Web AI Endpoint Changes Cannot Reuse Existing Credentials + +- **Statement** - The Web AI settings writer compares full effective base URLs under the settings-file lock before applying any field. Summary-to-embedding fallback, legacy title URL casing, defaults and dormant credentials participate; title-provider changes also require a credential-free state. Relevant stored values/markers, environment keys, runtime-sidecar keys, or non-absent Keychain metadata reject the entire patch. Unsafe/unreadable sidecars and indeterminate Keychain status fail closed. Keychain probing requests only attributes with interaction disabled; it does not retrieve secret data or migrate credentials. Environment URL overrides do not permit staging a different stored destination. Credential-free endpoint editing and unrelated edits remain allowed. The typed IPC rejection maps to HTTP 409; the UI retains a correctable draft and says nothing was saved. Changing a credential-bound endpoint requires a local server settings operation. This is a Web configuration boundary, not DNS/redirect pinning or isolation from a trusted local administrator. +- **Enforced by** - `macos/EngramService/Core/EngramServiceCommandHandler+WebAiSettings.swift`, `macos/Shared/Service/EngramServiceWebWriteClient.swift`, `macos/EngramRemoteServer/Core/WebWriteRoutes.swift`, `macos/EngramRemoteServer/Core/WebUIRoutes.swift`. +- **Verified by** - `macos/EngramServiceCoreTests/WebAiSettingsWriteTests.swift`, `macos/EngramRemoteServerCoreTests/WebAiSettingsHTTPTests.swift`, `tests/scripts/collector-web-ui.test.ts`. +- **Gate** - `none`. + ## Optional AI Maintenance Does Not Gate Required Index Readiness - **Statement** - Initial scan and periodic indexing do not await embedding providers. A separate bounded maintenance task starts after the required scan, checks provider configuration and cooldown before backlog queries, and retains existing retry/terminal semantics. Orderly shutdown explicitly cancels and joins that task before draining writers and checkpointing; optional work cannot retain writer ownership after the runner returns. @@ -227,9 +255,9 @@ Invariants are properties that must survive every change; each entry names where ## Collector Bootstrap Scan Finishes Past Vanished Directories -- **Statement** - A queued non-root directory that no longer exists, or is no longer a directory and is not a symlink, finishes as empty, and so do its queued descendants; locators observed under it earlier are left unchanged. Any other failure to open a directory still blocks the scan and records `last_scan_failure`, and a symlink swapped in for a component is still refused. A missing or replaced root keeps its unavailable-source handling. A root whose bootstrap stays blocked for five consecutive steps is reported once on stderr, by root ID, and once more when it recovers; a shorter block writes nothing. A blocked root is still retried every turn without back-off. +- **Statement** - A queued non-root directory that no longer exists, or is no longer a directory and is not a symlink, finishes as empty, and so do its queued descendants; locators observed under it earlier are left unchanged. Any other failure to open a directory still blocks the scan and records `last_scan_failure`, and a symlink swapped in for a component is still refused. A missing or replaced root keeps its unavailable-source handling. A root whose bootstrap stays blocked for five consecutive steps is reported once on stderr, by root ID, and once more when it recovers; a shorter block writes nothing. Blocked bootstrap retries use a process-local monotonic delay of 1, 2, 4, 8, 16, 32 and at most 60 seconds. Partial scan progress does not reset the delay; a completed scan does. Cooling roots cannot restart listeners, begin new capture or force capture turns; healthy roots and already staged uploads continue. Retry becomes eligible at the deadline and is noticed on the existing cheap-wake cadence; fresh process start retries immediately. - **Enforced by** - `macos/EngramCollectorCore/CollectorPOSIXRootEnumerator.swift`, `macos/EngramCollectorCore/CollectorBootstrapWalker.swift`, `macos/EngramCollectorCore/CollectorRuntime.swift`. -- **Verified by** - `macos/EngramCollectorCoreTests/CollectorPOSIXRootEnumeratorTests.swift` (testVanishedFrontierDirectoryFinishesScan_repro, testVanishedDirectoryDrainsQueuedDescendantsAndKeepsObservedLocators, testDirectoryVanishingMidEnumerationFinishesWithoutItsUnappliedEntries, testTargetedDirectoryThatNeverExistedFinishesEmpty, testFrontierDirectoryReplacedByFileFinishesEmpty, testMissingOrReplacedRootIsNotAVanishedDirectory, testPermissionDeniedFrontierDirectoryStillBlocks, testSymlinkSwappedForFrontierDirectoryIsStillRefused), `macos/EngramServiceCoreTests/CollectorRuntimeTests.swift` (testBlockedBootstrapRootIsReportedOncePerChange). +- **Verified by** - `macos/EngramCollectorCoreTests/CollectorPOSIXRootEnumeratorTests.swift` (testVanishedFrontierDirectoryFinishesScan_repro, testVanishedDirectoryDrainsQueuedDescendantsAndKeepsObservedLocators, testDirectoryVanishingMidEnumerationFinishesWithoutItsUnappliedEntries, testTargetedDirectoryThatNeverExistedFinishesEmpty, testFrontierDirectoryReplacedByFileFinishesEmpty, testMissingOrReplacedRootIsNotAVanishedDirectory, testPermissionDeniedFrontierDirectoryStillBlocks, testSymlinkSwappedForFrontierDirectoryIsStillRefused), `macos/EngramServiceCoreTests/CollectorRuntimeTests.swift` (testBlockedBootstrapRootIsReportedOncePerChange, testBlockedBootstrapBacksOffWithoutDelayingHealthyRootAndRecovers_repro, testBlockedBootstrapResidentWaitDoesNotSpinAndRetriesBeforeLongPoll_repro). - **Gate** - `none`. ## Cross-Machine Exact Duplicates Are Quarantined At Commit @@ -260,6 +288,13 @@ Invariants are properties that must survive every change; each entry names where - **Verified by** - `macos/EngramServiceCoreTests/CollectorPublicationWorkerTests.swift` (testNeverCapturedMissingSourceLeavesDirtySetAndRearmsOnReobservation_repro, testPreviouslyCapturedMissingSourceDefersOneSecondAndStaysUnacknowledged_repro, testMixedCapturedAndNeverCapturedMissingCodexShareOneBatchAndDeadlines_repro). - **Gate** - `none`. +## Human Keyword Ranking Uses All Search Terms + +- **Statement** - Service keyword search and the App's local search/searchWithSnippets require every searchable term at session scope and rank by the sum of each term's best row rank, summed in canonical token order. Short terms retain zero-rank LIKE; long Latin/CJK terms use the same scalar-count trigram routing. Recency descending and session ID ascending break ties before LIMIT and final projection. Existing filters and Service's short-token candidate cap remain; cap-limited recall, native MCP ranking and weighted quality/title scores are outside this contract. +- **Enforced by** - `macos/EngramService/Core/EngramServiceReadProvider.swift`, `macos/Engram/Core/Database.swift`, `macos/Shared/EngramCore/CJKTextHelpers.swift` (usesTrigramMatch). +- **Verified by** - `macos/EngramTests/DatabaseManagerTests.swift` (testKeywordRankingUsesEveryTermBeforeLimitAcrossReaders_repro, testKeywordRankingShortFirstKeepsLongTermRelevanceAcrossReaders_repro, testKeywordRankingBreaksTiesByRecencyThenIdentityAcrossReaders), `macos/EngramServiceCoreTests/EngramServiceIPCTests.swift` (testSQLiteReadProviderCJKTrigramTermsRankByRelevance_repro, testSQLiteReadProviderKeywordSearchIdentityIndexLayoutMatchesFallback_repro). +- **Gate** - `none`. + ## Unverified Anchors None. diff --git a/docs/reviews/2026-09-07-collector-source-retirement-checklist.md b/docs/reviews/2026-09-07-collector-source-retirement-checklist.md index 0c00da807..b9acfa135 100644 --- a/docs/reviews/2026-09-07-collector-source-retirement-checklist.md +++ b/docs/reviews/2026-09-07-collector-source-retirement-checklist.md @@ -1,8 +1,31 @@ # Collector source-retirement checklist -## Current status (2026-10-02) - -- The newest dated entry in this file is 2026-09-13. Nothing in the repo records +## Resource acceptance update (2026-10-25) + +The missing quiet-source 30-minute window passed on the deployed Daily +`collector-fb9538a-20261024` (PID 98680, unchanged 10-second poll). Between +07:16 and 07:46 UTC, 61 samples covered 1800.000 monotonic seconds: 34.57 CPU +seconds / 1800 seconds = **1.921%** of one core (<=2%, PASS), maximum sampled +RSS **58.5 MiB** (<=150 MiB, PASS). Independent recomputation checked all samples, +the stable process start/binary hash, and a maximum sample gap of 30.023 seconds. + +All 16 roots were converged throughout; no requested-revision changes, active +scan observations, new publication ACKs or log growth occurred. This supersedes +the missing quiet-window receipt below, without overturning historical active +workload failures. The 30-second CPU intervals ranged 1.133-2.733%; the acceptance +metric is the whole-window average. Active-write and startup costs, the small +CPU margin and per-source retirement verification remain separate follow-ups. + +Each replica retained 603 privacy-withheld pending publications. Dirty locators +remained 275; no queue/data cleanup was performed. No source transcript or +credential reads, process attachment, configuration change or service restart. +Evidence: `~/.engram-shadow-core-20260911/hq/state/collector-resource-20261025/` +(`samples.jsonl`, `verified-result.json`, metadata summaries and hashes). +Full narrative and next verifier: `CHANGELOG.md` 2026-10-25. + +## Historical status (2026-10-02) + +- At this reconciliation, the newest dated entry was 2026-09-13. No repo entry recorded the terminal result of resource observer 7119 or append wrapper 58055, or a resumed acceptance pass, so their outcome is unknown here. - PR #446 merged as `3859f788`; HQ has run the r18 index-role Service and diff --git a/docs/reviews/2026-10-23-development-reconciliation.md b/docs/reviews/2026-10-23-development-reconciliation.md new file mode 100644 index 000000000..34b160fb5 --- /dev/null +++ b/docs/reviews/2026-10-23-development-reconciliation.md @@ -0,0 +1,563 @@ +# Engram development reconciliation — 2026-10-23 + +## Baseline and counting rules + +- Baseline: local `main` and live `origin/main` both + `5216ff3a61202b4c113b1e20799a03ad7e7df706` (`git ls-remote`, 2026-10-23). +- Implementation checkout: `.worktrees/leader-loop-20261023`, branch + `codex/leader-loop-20261023`. The main checkout's untracked + `.grok/workflows/engram-session-status.rhai` is unrelated and preserved. +- Canonical ownership remains `docs/TODO.md` (engineering), + `docs/followups.md` (verification/residuals), and `docs/roadmap.md` (product). + This dated report supplies routing and evidence, not a second backlog. +- Inventory covers all 21 files in `docs/superpowers/plans`, 24 specs, root + design documents, the lifecycle/relaunch plans, current follow-ups, and + their explicitly referenced closeout/checklist records. Archives, generated + reports, fixtures, and historical review findings are not new work queues. +- The 397 literal unchecked markers in the 21 plans include procedural steps + and even checkbox examples. They are **not 397 outstanding deliverables**. + Count one named residual/product decision, deduplicate repeated IDs, and + retain deployment and real-host acceptance separately from implementation. +- Live GitHub query: 0 open issues and 6 open dependency PRs (#490–#495). + Baseline Tests [39366050861](https://github.com/bbingz/engram/actions/runs/39366050861) + and CodeQL [39366050927](https://github.com/bbingz/engram/actions/runs/39366050927) + succeeded at this baseline. They do not validate this branch. + +## Development loop + +Owner requested continuous project leadership on 2026-10-23. For each batch: +refresh current source and worktree; choose a bounded confirmed item; capture +a failing behavioral regression; make the minimum fix; run focused and +affected contract checks; review the diff; update canonical status and +CHANGELOG/MEMO; select the next item. A failed check is not completion. + +Local development and tests are authorized. Push, PR publication, merge, +deployment, service restart, production configuration/data writes, public +release, and reboot remain separate actions requiring task-specific owner +authorization. No local Docker. No production database is a test fixture. +The active goal is the continuation mechanism for this conversation; no +background daemon, scheduler, or unattended external-write loop was installed. + +Leadership default after reliability: finish the software side of collector +cutover and Web authority honesty, then improve human search and measured +value visibility, then evaluate opt-in memory automation. Write acceptance +criteria before schema/adapter-wide changes. Existing explicit exclusions +(native sqlite-vec, removed corpus mining, remote deletion, new release +channels) are not silently overturned. Routine implementation choices within +the selected local batch do not need repeated owner confirmation. + +## Delivery boundary after owner feedback + +The owner requested shorter delivery cycles instead of continued scope growth. +The implementation batch ends at `b17783c9`: 16 local commits since the baseline, +covering reliability/Web fixes, search ranking/value visibility and measurement +tooling. Preserve the individual commits for review; do not start another product +family before this delivery is reviewed. This is local completion, not a release. + +Cosine supersession remains a product decision: `saveInsight` is synchronous and +checks Web visibility in its transaction; the later embedding backfill has no +originating caller policy. An unconditional backfill replacement would bypass +that check. No semantic replacement code was added. Remaining decisions stay in +the product ledger; real-host acceptance stays in followups. Push/PR publication +is the next external authorization boundary, followed separately by deployment. + +## First engineering queue + +| Order | Canonical item | Current evidence / disposition | Completion verifier | +|---|---|---|---| +| 1 | `cutover-role-surface-1` | Fixed locally; 215 MCP executable tests passed after 8 expected RED assertions. `MCPToolRegistry.tools` still returned 27 definitions for collector/replica/invalid roles while calls were refused. | Executable legacy/modern `tools/list` returns an empty list; local/index/default catalogs remain usable; direct calls still fail before DB/service access. | +| 2 | `collector-settings-write-flake-1` | Fixed locally: complete sibling file, 0600 before rename; all 85 runtime tests passed. Hosted recurrence unmeasured. | Publish a complete owner-only fixture atomically; exercise invalid configuration reload without transient JSON/permission failures. | +| 3 | `collector-test-fixture-leak-1` | Fixed locally: deferred fixture removal; affected test passed. | Match fixture cleanup pattern and run the affected test. | +| 4 | `collector-blocked-root-backoff-1` | Fixed locally; final 87 runtime tests passed after behavioral RED and scheduler negative control. Per-root monotonic backoff, capped at 60 seconds, preserves healthy-root progress and recovery. | Deterministic clock/recovery tests, healthy-root progress, bounded blocked attempts. | +| 5 | Round-12 secrets cluster | Existing fixes verified locally for the summarized scope; part of the 210-test App GREEN below. No redundant production patch. | Filesystem/secret failure cases, empty-key policy and injected-home/socket routing. | +| 6 | `session-detail-id-*` | Live-navigation/favorite residuals closed locally. Mounted transcript verification found and fixed stale paging cleanup/duplicate scheduling; initial/result/index cancellation passed the tested interleavings. | Five mounted tests passed five iterations; final-harness negative control failed 17 expected assertions; restored final 72 transcript tests passed. No full UI/production claim. | +| 7 | `cutover-web-editor-hardening-1` | Project-operation actor propagation and endpoint/key reuse fixed locally. Alias audit storage now implemented below; credential throttle remains an accepted design residual. | Actor/dispatcher receipts below. Endpoint batch: 26 service, 24 HTTP/IPC/UI-route and 181 JS tests passed after RED; fake keys only. | +| 8 | Collector CPU work | One parent acceptance item, not separate counts for every hotspot. Publication, idle-turn, stuck-dirty and startup leads remain below. | Profile the changed path before tuning; retain CPU <=2% / RSS <=150 MiB real-host acceptance separately. | + +## Completed local batch: blocked-root retry backoff + +- Scope: runtime scheduling only; preserve coordinator loss persistence, scan + fences, source identities, fail-closed filesystem checks and upload workers. +- On blocked bootstrap, postpone that root's restart and new capture by 1, 2, + 4, 8, 16, 32, then at most 60 seconds on a monotonic clock. A healthy root + can continue its own scan/capture/publication. Existing durable publications + remain eligible for the independent upload loops. +- Partial re-walk progress does not reset the accumulated delay; a completed + scan does. Fresh process start retries immediately; no schema or persisted + deadline is introduced. Missing/replaced root handling keeps its own guard. +- A cooling root must not make the resident capture scheduler spin on its + pending loss/recovery signal. Retry becomes eligible at the deadline and is + observed at the existing cheap-wake cadence (at most one second), subject to + work already in progress. Cancellation still interrupts normal waits. +- Acceptance: deterministic clock tests prove bounded stream/revision churn, + delay cap, healthy-root ACKs, permission recovery, reset and restart; + resident-loop regression proves no hot capture spin during the delay. +- Verification: behavioral RED produced 24 expected assertions; omitting the + cooling-root mailbox filter made the resident regression fail. After restoring + the filter, all 87 runtime tests passed. Lint and the invariant ledger passed. + Logs and xcresults: `/tmp/engram-leader-backoff-20261023/`; final result is + `green-final.xcresult`. Next batch is current-source security/UI adjudication. +- Risk: a newly readable blocked root can wait up to the remaining 60-second + delay plus scheduler/work latency. No CPU target or host recovery claim is + made without a separately authorized real-host receipt. + +## Current-source adjudication: secrets and session identity + +The September `.grok/all-confirmed-remediation-ledger-2026-09-02.md` records +W1/W7/W9 remediation and explains why the August blanket-open text was stale. +That record was only a navigation lead; the decisions below use current source +and fresh tests at local commit `414b7c18`. + +| Summarized claim | Verdict / current evidence | +|---|---| +| Runtime-secret writes/cleanup follow caller-controlled paths or own unrelated siblings | OVERTURNED for the described current paths. `EngramServiceLauncher.writeRuntimeAISecrets` validates the runtime parent; `SecureRegularFile` pins directory FDs and uses no-follow read/write/unlink operations. Custom socket sidecars are namespaced. Tests exercise leaf/intermediate symlinks, hardlinks, failed publication and unrelated siblings. | +| AI key deletion/save failures are reported as successful | OVERTURNED. `APIKeyPersistencePolicy` checks settings writes/deletion; `reconcilingRuntimeBridgeRefresh` preserves marker failures and reports bridge failures. Empty unavailable keys are preserved until explicit clearing. Policy and call-site tests passed; no real Keychain was used. | +| Settings refresh/default test DB falls through to the login user's runtime | OVERTURNED for the reported entry paths. App/menu/main-window Settings pass the resolved socket to `AISettingsSection`; default DB uses `EngramUserDataDirectory.resolvedHomeDirectory`. Injected socket/source-wiring and temporary-home regression passed. This is not a fresh whole-suite filesystem-access audit. | +| Live navigation and delayed favorite reads lack identity guards | OVERTURNED. Popover/SourcePulse complete captured nil-resolution tokens; favorites compare generation and invalidate it on successful toggle/disappear. Current navigation/predicate/source-wiring tests passed. | +| Every transcript apply is identity-fenced | NARROWED by the mounted-view batch below. Existing cancellation checks protect the tested initial success/error, page/error and delayed index applications. The actual reproduced defect was unowned paging cleanup plus duplicate scheduling; both fixed locally. This is bounded concurrency evidence, not an exhaustive proof of every UI interleaving. | + +Fresh validation: `xcodebuild test`, scheme `EngramTests`, isolated +`CFFIXED_USER_HOME` and `TEST_RUNNER_CFFIXED_USER_HOME`, same DerivedData as +above, passed 210 tests: AISettingsSection 36, selected launcher 19, +UnixSocketTransport 32, injected-home DB 1, HomePopoverActions 43, +TodayWorkbenchScope 39 and TranscriptFind 40. No production code changed; +there is no newly claimed RED. Logs, exact selectors and xcresult: +`/tmp/engram-leader-residuals-20261023/app-{command.txt,residuals.log,residuals.xcresult}` +(command file is `app-command.txt`). No mounted UI/render, hosted or live +credential test was run. The next batch implemented Web project audit actor propagation (below); +the former fixed actor enum required a bounded identity value, not DTO +forwarding alone. + +## Web project audit actor batch + +The [phase-2 addendum](../superpowers/specs/2026-10-23-web-project-audit-actor-addendum.md) +recorded a baseline correction: alias add/remove had no audit-log writer +or actor field. The later alias-audit batch below closes that separate design. This batch +covers the four existing project audit paths only, with no schema migration. +RemoteServer injects the bound editor identity (credential fallback web-editor) +after browser-key validation, typed IPC carries it with the existing capability +token, and Service writes it through the native pipeline/batch override. Actor +is included in replay fingerprints. Native actor parsing/defaults stay intact. + +RED: 8 HTTP identity assertions, and 8 service assertions covering malformed +actor decoding, audit readback and cross-actor replay. GREEN: 60 Web HTTP/auth/IPC tests, 29 service/project-registry tests and 34 +native batch/store tests passed. Lint and invariant gates also passed. Evidence: `/tmp/engram-leader-web-actor-20261023/`. +Continuation found the dispatcher still rejected actor despite the component +GREEN at `6d30437a`. The four optional IPC field allowlists are now updated. +The audit/replay test now uses the real Web client/socket/dispatcher/storage: +it failed before the fix, then passed with all 18 selected service tests. +Evidence: `/tmp/engram-leader-web-dispatch-20261023/`. This supersedes the prior +component-only evidence for complete service dispatch. +No browser/render, hosted or production verification. Both RemoteServer and +Service must be updated before claiming attribution on a host; an old Service +may ignore the additive actor field. Existing historical rows are untouched. + +## Completed local batch: Web AI endpoint credential boundary + +Confirmed and reproduced the Web editor changing an endpoint while retaining +the service credential. The existing locked settings mutation now rejects the +whole patch if a changed effective destination can reuse an existing key. +Comparison includes full paths, summary-to-embedding fallback, legacy title +casing/defaults and title-provider changes; disabled providers and env URL +overrides do not bypass the guard. Relevant stored values/markers, env keys, +runtime sidecars and markerless Keychain metadata participate. Unknown secret +state fails closed. Keychain lookup requests attributes with interaction disabled, +not secret data. Credential-free endpoint editing and unrelated edits remain. + +The structured IPC refusal maps to HTTP 409, preserving the draft and explicitly +saying nothing was saved. Verification includes the real capability-protected Web +client/socket/dispatcher and unchanged file bytes, independent HTTP error mapping, +and executable page-JS correction/retry behavior. RED: two service assertions plus +one UI regression. GREEN: 26 service, 24 HTTP/client/UI-route, 181 JS tests; lint and +invariant gates passed. Evidence: `/tmp/engram-leader-web-endpoint-20261023/`. +No real Keychain, outbound AI request, rendered browser, all-target suite, hosted +or production acceptance. This is not DNS/redirect pinning or isolation from a +trusted local settings writer. + +Credential-throttle adjudication: current source and 47 fresh auth session/route +tests confirm the shared five-attempt window and its availability limitation. +The accepted [identity-auth design](../superpowers/specs/2026-10-08-web-tailscale-identity-auth-design.md) +explicitly parks per-client credential throttling. There is no trusted client +address contract in the credential route; arbitrary forwarded/identity headers +must not become throttle keys. Preserve this design item until client/proxy +provenance and a global guessing budget are defined. No auth-code change. +Receipt: `auth-adjudication.{log,xcresult}` in the same evidence directory. +The following batch completed the mounted-view transcript investigation. + +## Completed local batch: transcript paging ownership + +Mounted SessionDetailView tests held non-cooperative parsing/index completion +across A-B and A-B-A switches. Existing initial success/error and index-apply +cancellation guards passed. Old page/all/copy cleanup, however, cleared a new +session's busy state and let a duplicate action cancel its valid page. Two actions +before the task started also both entered. The minimal runtime change sets busy +before scheduling, assigns a fresh load token, invalidates it on switch/disappear, +and lets only its owner release busy/task state. + +The DEBUG-only instance hooks exercise actual mounted State and actions with +fake transcripts, mock service and captured clipboard output. Snapshot-observer +timeouts were diagnosed and replaced with on-demand mounted State reads; retain +those failures in the evidence directory. Final five mounted tests passed five +iterations (25 executions). Removing both runtime protections with that final +harness failed 17 expected assertions; restoring them passed all 72 selected +transcript tests. Lint/invariants/project-drift/diff checks passed. +Release arm64/x86_64 build passed; binary inspection confirms test hooks only +in the Debug dylib, not the Release executable (`build-receipt.json`). Evidence: +`/tmp/engram-leader-transcript-20261023/`; details in CHANGELOG. No hosted or +production acceptance, exhaustive UI stress or real-parser/clipboard end-to-end +claim. `session-detail-id-3` closes locally for this reproduced/verified scope. +The following batch completed `r11-dropped-2` transport-error mapping verification. + +## Completed local batch: legacy request-body error verification + +Replaced the source-string check with four production-router behavioral tests. +Injected streams prove partial read errors return 503, accumulated oversize bodies +return 413, and failed bundle uploads neither create blobs nor replace existing +ciphertext. Exact-limit uploads and MCP initialize pass. MCP error envelopes +separate transport failure, size failure and malformed complete JSON; auth/Origin +refusals do not read the stream. The router, body collector and storage are real; +only the inbound transport stream is controlled, with temporary stores/fake keys. + +Temporarily restoring the historical catch-all mapping failed five expected +assertions in the two error tests. Restoring unchanged production source passed +all 28 server tests. Lint and invariant/module/convention gates passed. Evidence: +`/tmp/engram-leader-body-20261023/` (`negative-control`, `green-final`, lint and +invariant logs/xcresults). No hosted/production or real socket-disconnect claim. +`r11-dropped-2` closes locally. The following batch measured and optimized a +Cursor legacy observation hotspot; quiet-host CPU/RSS acceptance remains +separately authorized. + +## Measured local batch: Cursor observation digest formatting + +An isolated Release XCTest used 64/512/2048 synthetic workspaces, pages of 1/64, +and three rounds of 50 pages. Process CPU excludes fixture setup. A five-second +sample during a 2048-workspace/64-item loop attributed 699/3738 stacks to digest +`String(format:)` calls. The shared SHA-256 function now converts the digest with +a fixed hex lookup table, retaining its exact lowercase 64-byte representation. +No directory cache, filesystem check, observer cadence or capture authority changed. + +The 2048-workspace/64-item median fell 6.824 -> 5.245 CPU ms/page (23.1%); the +64-workspace/64-item case fell 2.982 -> 1.536 (48.5%). At one item per page the +2048 case improved only 3.1%, preserving enumeration/stat work as a residual. +The optimized sample has zero formatting stacks among 3764 samples. These are +synthetic path measurements, not whole Collector CPU/RSS or production acceptance. +The 512-workspace baseline varied 3.49-6.37 ms; do not generalize its larger delta. +Temporary profile instrumentation was removed; its exact method, command, input +counts, logs, samples, binary hashes and comparison JSON remain under +`/tmp/engram-leader-cursor-perf-20261023/`. Correctness receipt is in CHANGELOG. +The following batch defined and implemented alias audit storage/transaction/ +retention semantics with failure-path acceptance. Host cutover +and the comparable 30-minute resource window remain external acceptance work. + +## Completed local batch: transactional Web alias audit + +The [alias design](../superpowers/specs/2026-10-23-web-alias-audit-design.md) +adds a dedicated diagnostic table, not project migration records. HTTP supplies +the editor session actor after browser-key validation; additive DTOs and the real +capability dispatcher forward it. Missing actor uses web-editor. A changed exact +pair records action, actor, published identities and UTC time in the mutation's +transaction. Repeated/no-op/refused writes create no rows. Audit insert or retention +failure rolls back the business mutation. The latest 10,000 transitions are kept +by insertion ID; raw paths and request bodies are not retained in this history. + +RED: Service 20 failures including four missing-table errors; HTTP four missing +actor assertions. GREEN: 13 Service tests (including real HTTP/socket/dispatcher/SQL +and capability-socket attribution), 46 HTTP/client/auth/project tests, 20 migration +tests. Insertion/pruning triggers prove rollback; 10,000 seeded rows prove retention +across clock rollback; old/repeated migration preserves aliases/history. Lint and +invariants passed. Receipt: `/tmp/engram-leader-alias-audit-20261023/`. +No all-target, hosted, rendered-browser or production upgrade claim. Upgrade Service +before RemoteServer: old Service rejects actor on aliases; old RemoteServer with +new Service works with web-editor attribution. Native alias and permanent audit +history are explicitly outside this log. Credential throttle remains the accepted +provenance/design residual. The following batch adjudicated the foreground UI +test-harness failure policy (`mcp-hermetic-1`), before product search. + +## UI harness failure-policy adjudication + +Current `TestLaunchConfig.configure` installs isolated HOME/fixed-home, settings, +secret paths and test mode before any fallible setup. All 27 configure call sites +have `continueAfterFailure = false` in the test or its setup. The old claim that +the suite still lacks fail-fast policy is stale; the helper uses XCTFail + return, +so a caller's XCTest failure policy remains part of the behavioral contract. + +A temporary injected missing-fixture test used a distinct App bundle identifier +and no actual app.launch statement. Xcode built it, but the UI runner failed to +initialize with LocalAuthentication Code=-4, "System authentication is running." +The test body never ran. This proves neither stop-before-launch nor continuation; +the negative control and normal foreground smoke were not run. No authentication +process was changed and no immediate retry was attempted. All four backed-up files +were checked against HEAD and restored byte-for-byte; the worktree was then clean. + +After restoration, both existing `MCPActivationOnboardingTests` source-policy +regressions passed in isolated fixed homes. They verify source ordering/wiring, +not foreground behavior. Evidence: `/tmp/engram-leader-ui-harness-20261023/`, +`failure-stop.{log,xcresult}`, `source-policy.{log,xcresult}` and original backups. +The residual stays open only for foreground acceptance when the runner can start. +The following product-search batch continued without changing this policy. + +## Completed local batch: human keyword ranking consistency + +Lifecycle 3.4's missing-BM25/CJK/facet premise was stale: Service already used +FTS rank, windowed highlights and CJK trigram routing; SearchPage already had +source/project/time/origin filters. The actual reproducible gaps were that both +readers ranked by only the first term, and App offline search routed all CJK to +zero-rank LIKE. A short-first query such as AI needle discarded needle relevance. + +The [ranking contract](../superpowers/specs/2026-10-23-keyword-ranking-design.md) +keeps session-level AND and sums each term's existing best-row rank in canonical +token order. Time descending and ID ascending break ties before the result limit +and final projection. App search/searchWithSnippets now use Service's scalar-count +trigram/LIKE routing. No schema/index rebuild, UI layout, embedding or new facet. +Service's short-term candidate cap remains an explicit recall limitation; native +MCP's separate ranking policy is outside this human-search contract. + +After fixing an initial missing-await test compile error, behavioral RED produced +72 expected ranking assertions across three real App/Service shared-fixture tests. +GREEN: 163 App data/search tests and 63 Service search/semantic/insight/Web tests, +zero failures. Six term permutations, cross-message matching, rank-before-limit, +stable ties, Latin/Chinese/Japanese/Korean, highlights, both Service FTS layouts, +filters, busy handling and semantic fallback passed. Lint/invariants passed. +Receipt: `/tmp/engram-leader-keyword-ranking-20261023/`. No all-target, rendered UI, +hosted, production-corpus relevance/latency or deployment acceptance. Lifecycle 3.4 +stays partial: title/quality/recency weighting and value/lifecycle facets need +separate contracts. Next product batch: adjudicate existing value-score/band UI +and access/outcome signals before selecting a measured visibility improvement. + +## Collector / cutover residuals (all named rows routed) + +| Item | Current disposition / next action | +|---|---| +| `cutover-feed-stalled-1` | Recovery implemented and deployment recorded; reboot/real-host recovery remains unverified this session. Do not label the October 2 outage current. | +| `cutover-hq-local-gap-1` | HQ collector/canary/central-index/Web acceptance remains operational work. Runbook R2–R10 includes credential/root steps and host mutations. | +| `codex-resume-window-1` | `CodexAdapter(modifiedSince:)` and repros exist. Old-host backlog recovery/deployment needs fresh host evidence; never restart the old stack merely to close this row. | +| `collector-stuck-dirty-locators-1` | Vanished never-captured half fixed. Present-but-uncapturable Copilot/Kimi members remain an unverified cause, not a proven parser defect. | +| `collector-blocked-root-backoff-1` | Closed locally; behavioral/runtime evidence above. Deployment and real-host CPU acceptance remain separate. | +| `collector-start-cost-1` | Full startup walk cost remains unmeasured; startup must not be confused with steady-state CPU. | +| `collector-epoch-change-scope-1` | Root-local suspension implemented in #481; Daily deployment recorded October 18. Retain real-host trigger verification separately. | +| `collector-epoch-producer-1` | Same-instance epoch rotation implemented in #481; manual inventory wipe still loses identity by accepted design. Real revision-bump recovery remains unexercised. | +| `collector-epoch-authority-file-1` | Existing owner decision is runbook-only: update `initialEpoch` before next restart. Revisit before first real approval; no silent authority relaxation. | +| `collector-revision-bump-staged-1` | Drain pending publications before revision bump. Old-revision staged publication handling remains a design/operational limitation. | +| `collector-publication-cost-1` | Shared privacy proof and `memchr` landed. Remaining byte scans/CAS reads are low-priority measured-performance candidates. | +| `collector-idle-turn-cost-1` | Frontier/ancestor/vanished-locator fixes landed. Local digest formatting optimization measured above; legacy enumeration/stat and modern-peer discovery costs plus comparable quiet-host receipt remain. | +| `collector-settings-write-flake-1` | Closed locally; hosted recurrence not claimed. | +| `collector-test-fixture-leak-1` | Closed locally; no historical temp cleanup. | +| `cutover-install-tooling-1` | P3 scripts/templates exist (#454); old claim of no tooling is stale. First real-host install/upgrade/rollback remains unverified. | +| `cutover-web-editor-hardening-1` | Project actor, alias audit and endpoint credential batches above; credential throttle remains an accepted design residual. Tailscale mode does not prove credential-mode behavior safe. | +| `cutover-role-surface-1` | First batch passed locally; no deployment claim. | + +Additional cutover gates: configure and verify `webReaderURL` on each intended +host; deploy/probe current HQ epoch CLI/Service together when authorized; +obtain per-source capture → dual receipt → HQ index → Web proof for the 19 +sources / 17 collector root formats; retain D3 cache/PB exceptions; decide M1 +collector inclusion after fresh source evidence. Local synthetic success does +not satisfy these host receipts. Daily's last comparable recorded CPU window +was 5.847% (October 18), while October 19's 34.6% active-workload window is not +a comparable quiet-state receipt. Neither is a current runtime measurement. + +## Other residuals (all nonterminal families routed) + +| Canonical family | Remaining scope | +|---|---| +| Live ingest SLA | Unique natural session through HQ publish and App/MCP search, including post-reboot path, before claiming 16-minute SLA. | +| Live enable / token / helper / first publish | Restart-on-first-enable contract, ad-hoc helper credentials, independent helper lifetime, and initial-scan gating are recorded operational constraints; refresh on the intended host. | +| Live launchd / boot / hardening | Old jobs were installed; reboot without console login, injected degraded sentinel, duplicate start, and executed rollback were not proven. Cutover may retire these jobs; decide applicability before testing. | +| `live-ingest-settings-repro-1` | Already closed locally; not a new implementation task. | +| Live hard-delete retract / manifest GC retry | Conditional: only promote if a periodic hard-delete producer or storage-hygiene need appears. | +| `live-session-scan-timeout-1` | Data path fixed historically; unlocked menu rendering acceptance remains. | +| `startup-initial-scan-latency-1` | Historical Daily local-Service startup symptom; check whether that retired role still applies before optimizing or probing it. | +| Project-move stale-break claim | TS reference-only crash recovery needs an ownership-safe delete design; do not weaken fail-closed locking. | +| Round-12 `mcp-hermetic-1` | Source policy present and two source tests passed; foreground failure-injection acceptance blocked before test execution by system authentication. Details above; no new product defect proven. | +| Round-12 `live-export-terminal-*` | Warp cleanup and Terminal/iTerm/Ghostty cancellation need real terminal integration testing. | +| Round-12 `workitem-localtime-1` / `xcodeproj-untracked-*` | Fixes are on the verified remote-main baseline (`00bb5809`); “local, not pushed” is stale. Tests are not rerun by a Git history check. | +| Round-12 `r11-dropped-2` | Closed locally: production-router injected body failures, limits, auth ordering and storage non-mutation passed; historical-mapping negative control failed five expected assertions. Details above; real transport-disconnect/host verification remains unclaimed. | +| `ALIAS-P2` | Conditional same-basename ghost removal; only reopen with a repro. OPS-ALIAS-001 is closed, not authorization to rewrite production aliases. | +| Blind-audit A1–D1 | Old stacked-PR table is implementation history, not an open-PR queue. Only a newly reproduced residual is promoted; no current open implementation PR is hidden there. | +| `ARCH-001` | A/B/C/D shipped. Full CoreRead pool/DTO migration remains structural investment. | +| `R6` | Writer-gate availability redesign deferred; current long FS+patch gate is an integrity choice. | +| Accepted security residuals | SEC-M4/M5/I1/I2/L4 and release/TS-only notes remain explicit product/threat-model decisions, not automatic remediation. | +| Archive discovery | ARCHIVE-DISCOVERY-001 needs durable restart/event-loss design; no safe prefix-limit patch. Collector inventory does not automatically fix the legacy archive coordinator. | +| Additional archive exporters | Each virtual/composite/database-backed source needs canonical source-set and delete-original/replay-equivalence proof. | +| Remote archive GC/erasure | Separate design, backup/restore and explicit destructive-operation authorization; API stays immutable. | +| UI restyle / launcher flake | October 19 deferred deeper Home/Search/Timeline/Activity restyle. Launcher full-suite timing failure was observed and passed alone; reproduce before changing it. | + +## Product roadmap reconciliation + +The historical table had 12 decision rows. One is already implemented +(`ai_audit_log` body redaction); five have partial implementation or prerequisites (score measurement, human search, +value-band UI, plugin integration and Grok/Pi/provider work). Lifecycle 3.2 was parked in its source plan but +missing from the table; it is now explicitly routed. The resulting table has +13 rows: **12 open or conditional product families, 1 implemented family**. +These are not 12 ready-to-code defects. + +1. Value score evolution, measured thresholds, and outcome/access signals. + Measurement tooling now exists: `scripts/db/measure-database.py`; six behavioral + tests passed and a copied repository fixture produced aggregate JSON. Current + production distribution and a revised factor/outcome contract remain open. +2. Human search beyond the all-term/CJK ranking batch and existing basic filters: + value-band filtering is now implemented locally (Service/offline, before + result and short-token candidate limits). Title/quality/recency weighting and + lifecycle facets remain. Receipt: `/tmp/engram-leader-value-filter-20261023/`. +3. Tool-result normalization plus structured summary job. +4. Value band UI now implemented locally in cards and Sessions table cells; + `value_override` remains open. Reused existing <=35 / >=60 bands and shared + score function; 31 App/model/read/render tests plus two score tests passed. + Four bilingual light/dark component renders inspected; no full-page interaction + or production claim. Receipt: `/tmp/engram-leader-value-bands-20261023/`. +5. Embedding-cosine insight supersession: decision pending on replacement timing + and authority scope; asynchronous embedding must not bypass Web visibility. +6. Opt-in automatic insight extraction. +7. Native sqlite-vec: conditional only on measured bottleneck and explicit + architecture decision; current repo instructions prohibit introducing it. +8. Corpus/rule mining: intentionally removed, not accidentally unfinished. +9. Plugin Stop/SessionEnd insight capture and distribution channels: + SessionStart hook already exists; no release/channel publication authorized. +10. Remaining provider taxonomy/capability choices: Grok/Pi adapters already + register in `SessionAdapterFactory`; do not rebuild them from archive. +11. Sources/Settings navigation consolidation. +12. Lifecycle-state/shadow-archive model (not Archive V2 source reclamation). + +`ServiceAIAuditRecorder.record` already writes via `ServiceWriterGate`, gates +bodies with `logBodies`, redacts before truncating, and has +`ServiceAIAuditRecordTests.testLogBodiesStoresRedactedTruncatedTextWhileWebKeepsFlagsOnly`. +This closes the old “no Swift writer/body-redaction” assumption, not a broad +security audit. Historical retention-cadence assumptions must be reconsidered +now that a live writer exists. + +## Dependency maintenance lane + +Live open PRs on October 23: #490 lint-staged 17.6.0, #491 pixelmatch 8.0.0, +#492 openai 7.38.0, #493 @types/node 26.6.5, #494 tsx 4.23.15, #495 CodeQL +Actions group. These are PR titles, not independently approved upgrade +recommendations. Inspect exact diffs and required checks before selecting a +batch; treat pixelmatch's major version and immutable Actions pins separately. +No dependency bump is bundled into the MCP fix. + +## Verification record + +- Read-only baseline: Git status/history/worktrees, live main SHA, issue/PR + inventory and main CI conclusions. +- Real-binary historical gate rechecked live: + [39098816343](https://github.com/bbingz/engram/actions/runs/39098816343), + success at `10f5ab21f876b73b768a4a940165d41cc0dfc3bd`; workflow and runner + exist on the baseline. This closes stale “no recorded gate” prose, not + current-branch or real-host acceptance. +- Current branch: 215 MCP executable tests and 94 service tests passed + (85 collector runtime, 1 publication, 8 AI audit). The MCP RED run first + failed eight expected assertions. `npm run lint` passed with 1 existing + warning / 40 infos; invariant-ledger/module-boundary/Swift-convention checks + and 72 local document links passed. Commands/logs are in `CHANGELOG.md`. + Other source-existence and historical receipts are not freshly run tests. +- No production state readback, host mutation, release, UI rendering, or full + all-target regression has been performed for this reconciliation. + +## Plan-by-plan routing appendix + +Each implementation plan is listed once. “Exists” is source/navigation evidence, +not proof that every historical acceptance criterion was rerun. + +| Plan | Current route | +|---|---| +| [2026-06-04-followup-remediation](../superpowers/plans/2026-06-04-followup-remediation.md) | Historical remediation; residuals use current followups, not checkbox count. | +| [2026-06-04-fts-table-swap-rebuild](../superpowers/plans/2026-06-04-fts-table-swap-rebuild.md) | Explicit completed header (#48); preserve current FTS swap invariants. | +| [2026-06-06-claude-qoder-dir-reconcile](../superpowers/plans/2026-06-06-claude-qoder-dir-reconcile.md) | Explicit completed header; current project-move tests are the verifier. | +| [2026-06-06-project-move-gemini-iflow](../superpowers/plans/2026-06-06-project-move-gemini-iflow.md) | Explicit completed header (#65); not an open implementation batch. | +| [2026-06-10-audit-complete-remediation](../superpowers/plans/2026-06-10-audit-complete-remediation.md) | Historical audit campaign; current residuals are in followups. | +| [2026-06-27-session-implementation-timeline](../superpowers/plans/2026-06-27-session-implementation-timeline.md) | Extractor, persistence and Timeline implementation exist; unchecked procedural steps are stale. | +| [2026-07-10-wave7-43-item-remediation](../superpowers/plans/2026-07-10-wave7-43-item-remediation.md) | Terminal Wave 8 closeout in followups; do not reopen 43 historical findings. | +| [2026-07-10-wave7-engineering-zero-closeout](../superpowers/plans/2026-07-10-wave7-engineering-zero-closeout.md) | Terminal Wave 8 closeout; zero was a dated campaign count, not current project count. | +| [2026-07-11-exact-source-dual-replica-archive](../superpowers/plans/2026-07-11-exact-source-dual-replica-archive.md) | Implemented; conditional discovery/exporter/remote-GC residuals remain. | +| [2026-07-12-archive-remote-telemetry](../superpowers/plans/2026-07-12-archive-remote-telemetry.md) | ArchiveRemoteTelemetryStore and tests exist; historical implementation, not new queue. | +| [2026-07-12-archive-settings-sync-status-localization](../superpowers/plans/2026-07-12-archive-settings-sync-status-localization.md) | ArchiveSettingsSection and service status exist; acceptance is historical. | +| [2026-07-12-archive-sync-observability](../superpowers/plans/2026-07-12-archive-sync-observability.md) | ArchiveV2ServiceCoordinator/status implementation exists; no new task from unchecked steps. | +| [2026-07-12-archive-v2-automatic-reclamation](../superpowers/plans/2026-07-12-archive-v2-automatic-reclamation.md) | ArchiveReclamationCoordinator and tests exist; never infer authorization to enable or delete. | +| [2026-07-12-archive-v2-production-deployment](../superpowers/plans/2026-07-12-archive-v2-production-deployment.md) | Historical operational closeout; fresh production mutation requires authorization. | +| [2026-07-13-archive-v2-backlog-drain](../superpowers/plans/2026-07-13-archive-v2-backlog-drain.md) | ArchiveV2BacklogDrainer and tests exist; discovery limitation remains separate. | +| [2026-07-13-archive-v2-drain-fairness-remediation](../superpowers/plans/2026-07-13-archive-v2-drain-fairness-remediation.md) | Historical replication/coordinator fix; no claim of current host drain health. | +| [2026-07-13-claude-profile-registry](../superpowers/plans/2026-07-13-claude-profile-registry.md) | ClaudeCodeProfileResolver/Service and tests exist; not a new feature. | +| [2026-08-24-hq-live-session-ingest](../superpowers/plans/2026-08-24-hq-live-session-ingest.md) | Implemented historically; SLA/boot and retirement applicability remain operational. | +| [2026-09-05-collector-server-web](../superpowers/plans/2026-09-05-collector-server-web.md) | Implemented core with cutover/performance residuals listed above; W4.1/W4.5 closed by owner. | +| [2026-09-13-web-native-parity](../superpowers/plans/2026-09-13-web-native-parity.md) | Web read/write surfaces implemented; current actor/security and real-host acceptance residuals above. | +| [2026-09-14-hq-web-residuals](../superpowers/plans/2026-09-14-hq-web-residuals.md) | Explicit landed header (#446); do not rerun six historical fixes. | + +### Specification inventory + +All 24 baseline specifications were inventoried; this loop added the Web +project-audit addendum, alias-audit and keyword-ranking designs below. Paired specs use the plan route above; +a Draft/Accepted label alone neither opens nor closes an implementation task. + +| Specification | Routing | +|---|---| +| [2026-10-23-keyword-ranking-design](../superpowers/specs/2026-10-23-keyword-ranking-design.md) | Local all-term ranking/offline CJK consistency implemented; 226 focused tests passed after behavioral RED. Broader search weighting/facets remain product decisions. | +| [2026-04-23-swift-single-stack-design](../superpowers/specs/2026-04-23-swift-single-stack-design.md) | Swift-only product runtime already authoritative; never revive Node product entrypoints. | +| [2026-06-01-today-workbench-design](../superpowers/specs/2026-06-01-today-workbench-design.md) | Paired plan/current follow-ups; historical status is not a fresh acceptance receipt. | +| [2026-06-04-fts-table-swap-rebuild-design](../superpowers/specs/2026-06-04-fts-table-swap-rebuild-design.md) | Paired plan/current follow-ups; historical status is not a fresh acceptance receipt. | +| [2026-06-06-claude-qoder-dir-reconcile-design](../superpowers/specs/2026-06-06-claude-qoder-dir-reconcile-design.md) | Paired plan/current follow-ups; historical status is not a fresh acceptance receipt. | +| [2026-07-10-wave7-43-item-remediation-design](../superpowers/specs/2026-07-10-wave7-43-item-remediation-design.md) | Paired plan/current follow-ups; historical status is not a fresh acceptance receipt. | +| [2026-07-11-exact-source-dual-replica-archive-design](../superpowers/specs/2026-07-11-exact-source-dual-replica-archive-design.md) | Paired plan/current follow-ups; historical status is not a fresh acceptance receipt. | +| [2026-07-12-archive-remote-telemetry-design](../superpowers/specs/2026-07-12-archive-remote-telemetry-design.md) | Paired plan/current follow-ups; historical status is not a fresh acceptance receipt. | +| [2026-07-12-archive-settings-sync-status-localization-design](../superpowers/specs/2026-07-12-archive-settings-sync-status-localization-design.md) | Paired plan/current follow-ups; historical status is not a fresh acceptance receipt. | +| [2026-07-12-archive-sync-observability-design](../superpowers/specs/2026-07-12-archive-sync-observability-design.md) | Paired plan/current follow-ups; historical status is not a fresh acceptance receipt. | +| [2026-07-12-archive-v2-automatic-reclamation-design](../superpowers/specs/2026-07-12-archive-v2-automatic-reclamation-design.md) | Paired plan/current follow-ups; historical status is not a fresh acceptance receipt. | +| [2026-07-13-archive-v2-backlog-drain-design](../superpowers/specs/2026-07-13-archive-v2-backlog-drain-design.md) | Paired plan/current follow-ups; historical status is not a fresh acceptance receipt. | +| [2026-07-13-archive-v2-drain-fairness-remediation-design](../superpowers/specs/2026-07-13-archive-v2-drain-fairness-remediation-design.md) | Paired plan/current follow-ups; historical status is not a fresh acceptance receipt. | +| [2026-07-13-claude-profile-registry-design](../superpowers/specs/2026-07-13-claude-profile-registry-design.md) | Paired plan/current follow-ups; historical status is not a fresh acceptance receipt. | +| [2026-07-16-typescript-7-upgrade-design](../superpowers/specs/2026-07-16-typescript-7-upgrade-design.md) | Explicit implemented/merged header; no dependency reimplementation. | +| [2026-08-24-hq-live-session-ingest-design](../superpowers/specs/2026-08-24-hq-live-session-ingest-design.md) | Paired plan/current follow-ups; historical status is not a fresh acceptance receipt. | +| [2026-09-05-collector-server-web-design](../superpowers/specs/2026-09-05-collector-server-web-design.md) | Paired plan/current follow-ups; historical status is not a fresh acceptance receipt. | +| [2026-10-02-collector-replay-loss-design](../superpowers/specs/2026-10-02-collector-replay-loss-design.md) | Draft header is stale: #447/#448 recovery fixes exist; real-host reboot proof remains separate. | +| [2026-10-02-collector-volume-identity-design](../superpowers/specs/2026-10-02-collector-volume-identity-design.md) | Draft header is stale: #447/#448 recovery fixes exist; real-host reboot proof remains separate. | +| [2026-10-02-hq-local-collector-cutover-design](../superpowers/specs/2026-10-02-hq-local-collector-cutover-design.md) | Active host cutover R1-R10 and D1-D12; engineering/ops boundaries above. | +| [2026-10-08-web-tailscale-identity-auth-design](../superpowers/specs/2026-10-08-web-tailscale-identity-auth-design.md) | Phase 1 implemented; project actor propagation and alias storage verified locally; host acceptance remains separate. | +| [2026-10-23-web-project-audit-actor-addendum](../superpowers/specs/2026-10-23-web-project-audit-actor-addendum.md) | Added by this loop; current implementation scope, compatibility and acceptance contract. | +| [2026-10-23-web-alias-audit-design](../superpowers/specs/2026-10-23-web-alias-audit-design.md) | Added by this loop; implemented bounded transactional Web alias history, attribution and additive migration. | +| [2026-10-17-collector-native-epoch-reconcile-design](../superpowers/specs/2026-10-17-collector-native-epoch-reconcile-design.md) | Source implemented; epoch/operator/real-host residuals above. | +| [2026-10-18-collector-epoch-rotation-and-root-scope-design](../superpowers/specs/2026-10-18-collector-epoch-rotation-and-root-scope-design.md) | Source implemented; epoch/operator/real-host residuals above. | +| [2026-10-18-d7-legacy-local-origin-purge-design](../superpowers/specs/2026-10-18-d7-legacy-local-origin-purge-design.md) | Source implemented; epoch/operator/real-host residuals above. | +| [2026-10-18-hq-collector-epoch-approval-design](../superpowers/specs/2026-10-18-hq-collector-epoch-approval-design.md) | Source implemented; epoch/operator/real-host residuals above. | + +### Root design inventory + +Root designs are retained reference specifications. Paths below exist on this +baseline and establish navigation only; full criterion-by-criterion re-audit +remains UNVERIFIED unless a specific result appears above. Product choices +route to the reconciled roadmap, implementation defects to failing repros. + +| Design | Existing source/test pointer | +|---|---| +| [adapter-format-drift-design-2026-07](../adapter-format-drift-design-2026-07.md) | `macos/Shared/EngramCore/Indexing/SessionTier.swift`; `macos/Shared/EngramCore/Adapters/Sources/ClaudeCodeAdapter.swift` | +| [archive-list-index-robustness-design](../archive-list-index-robustness-design.md) | `macos/EngramRemoteServer/Core/ArchiveStore.swift`; `macos/EngramRemoteServer/Core/EngramRemoteServerApp.swift` | +| [archive-windowed-read-design](../archive-windowed-read-design.md) | `macos/EngramRemoteServer/Core/EngramRemoteServerApp.swift`; `macos/EngramRemoteServer/Core/ArchiveStore.swift` | +| [build-provenance-perf-design-2026-07](../build-provenance-perf-design-2026-07.md) | `macos/Engram/App.swift`; `tests/scripts/build-release-script.test.ts` | +| [claude-workflow-subagents-design-2026-07](../claude-workflow-subagents-design-2026-07.md) | `macos/Shared/EngramCore/AI/SessionSemanticSearchPolicy.swift`; `macos/EngramCoreTests/Adapters/ClaudeCodeMultiRootAdapterTests.swift` | +| [codex-native-parentage-design-2026-07](../codex-native-parentage-design-2026-07.md) | `macos/EngramCoreWrite/Indexing/StartupBackfills.swift`; `macos/Shared/EngramCore/Adapters/Sources/CodexAdapter.swift` | +| [cursor-cwd-ownership-design](../cursor-cwd-ownership-design.md) | `macos/Shared/EngramCore/Adapters/Sources/CursorAdapter.swift` | +| [diagnostic-bundle-design-2026-07](../diagnostic-bundle-design-2026-07.md) | `macos/Engram/Views/Settings/AboutSettingsSection.swift`; `macos/Engram/Core/Database.swift` | +| [embedding-guardrails-design-2026-07](../embedding-guardrails-design-2026-07.md) | `macos/Shared/EngramCore/AI/EmbeddingClient.swift`; `macos/Shared/EngramCore/AI/EmbeddingSettings.swift` | +| [file-index-state-orphan-prune-design-2026-07](../file-index-state-orphan-prune-design-2026-07.md) | `macos/EngramCoreTests/FileIndexStateOrphanPruneTests.swift` | +| [human-driven-sessions-design-2026-06](../human-driven-sessions-design-2026-06.md) | `macos/EngramCoreWrite/Indexing/SwiftIndexer.swift`; `macos/Shared/EngramCore/Indexing/InstructionExtractor.swift` | +| [insight-supersede-filter-design-2026-07](../insight-supersede-filter-design-2026-07.md) | `macos/EngramService/Core/EngramServiceCommandHandler.swift`; `macos/Shared/Service/EngramCLIContextCommand.swift` | +| [mcp-2026-07-28-dual-era-design](../mcp-2026-07-28-dual-era-design.md) | `macos/EngramMCP/Core/MCPStdioServer.swift`; `macos/EngramMCPTests/EngramMCPExecutableTests.swift` | +| [mcp-activation-onboarding-design-2026-07](../mcp-activation-onboarding-design-2026-07.md) | `macos/Engram/App.swift`; `macos/Shared/Service/EngramCLIContextCommand.swift` | +| [mcp-cost-honesty-design-2026-07](../mcp-cost-honesty-design-2026-07.md) | `macos/EngramMCP/Core/MCPInsightsTool.swift`; `macos/EngramCoreWrite/Indexing/SessionSnapshotWriter.swift` | +| [mcp-protocol-alignment-design](../mcp-protocol-alignment-design.md) | `macos/EngramMCP/Core/MCPStdioServer.swift`; `macos/EngramRemoteServer/Core/EngramRemoteServerApp.swift` | +| [mcp-semantic-search-design-2026-07](../mcp-semantic-search-design-2026-07.md) | Current roadmap / semantic-memory residuals; review source before promotion. | +| [p1-semantic-memory-design-2026-06](../p1-semantic-memory-design-2026-06.md) | Current roadmap / semantic-memory residuals; review source before promotion. | +| [publish-readiness-design-2026-07](../publish-readiness-design-2026-07.md) | `macos/EngramTests/AppSearchServiceCutoverScanTests.swift`; `macos/EngramService/Core/IndexingSchedulePolicy.swift` | +| [remote-mcp-2026-07-28-design](../remote-mcp-2026-07-28-design.md) | `macos/EngramRemoteServer/Core/EngramRemoteServerApp.swift`; `macos/EngramRemoteServer/Core/EngramRemoteServerConfig.swift` | +| [resume-suppression-subagents-design-2026-07](../resume-suppression-subagents-design-2026-07.md) | `macos/EngramService/Core/EngramServiceReadProvider.swift`; `macos/Shared/Service/EngramServiceModels.swift` | +| [service-resilience-design-2026-07](../service-resilience-design-2026-07.md) | `macos/Engram/AppNotifications.swift`; `tests/fixtures/mcp-golden/stats.source.json` | +| [source-health-predicate-design-2026-07](../source-health-predicate-design-2026-07.md) | `macos/EngramService/Core/EngramServiceReadProvider.swift`; `macos/Engram/Views/Pages/SourcePulseView.swift` | +| [tail-parse-design-2026-07](../tail-parse-design-2026-07.md) | `macos/EngramCoreWrite/Indexing/IndexingWriteSink.swift`; `macos/EngramCoreWrite/Database/EngramMigrations.swift` | +| [transcript-find-rendering-design-2026-07](../transcript-find-rendering-design-2026-07.md) | `macos/Engram/Views/ContentSegmentViews.swift`; `macos/EngramTests/TranscriptLabelAndCopyTests.swift` | +| [transcript-paging-timing-design-2026-07](../transcript-paging-timing-design-2026-07.md) | `macos/EngramCoreTests/AdapterWindowedReadTests.swift`; `macos/EngramTests/MessageParserTests.swift` | +| [uiux-polish-a11y-design-2026-07](../uiux-polish-a11y-design-2026-07.md) | `scripts/invariant-gates.json`; `macos/EngramTests/PopoverUsageSectionTests.swift` | + +Additional retained planning surfaces: `docs/engram-lifecycle-upgrade-plan.md`, +`docs/competitive-relaunch-2026-06.md`, `docs/competitive-mirror-2026-07.md`, +`docs/backlog-audit-2026-05-24.md`, +`docs/reviews/2026-08-12-stewardship-queue.md`, and +`docs/reviews/2026-09-07-collector-source-retirement-checklist.md`. The old +stewardship queue explicitly terminated on August 16; its scheduler is not +silently re-enabled by this new conversation goal. Competitive research is +historical, not a verified October market comparison. diff --git a/docs/reviews/README.md b/docs/reviews/README.md index ca5de1888..e5a97675f 100644 --- a/docs/reviews/README.md +++ b/docs/reviews/README.md @@ -8,8 +8,10 @@ are not automatically current backlog items. Current routing and disposition entry points: -- [stewardship queue](2026-08-12-stewardship-queue.md): ranked routing for the - current remediation stream. +- [development reconciliation](2026-10-23-development-reconciliation.md): current + plan inventory and continuous-development routing. +- [stewardship queue](2026-08-12-stewardship-queue.md): historical campaign, + explicitly closed on 2026-08-16. - [follow-ups](../followups.md): canonical observations and deferred work. - [TODO](../TODO.md): canonical confirmed engineering tasks. - [roadmap](../roadmap.md): canonical product and cross-module direction. diff --git a/docs/roadmap.md b/docs/roadmap.md index 0cbbf3c66..5f3fc08cf 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -52,6 +52,15 @@ correctly because `PopoverUsageSection` remains gated on real usage data. ## Current roadmap +**Reconciled 2026-10-23:** see the [development inventory](reviews/2026-10-23-development-reconciliation.md) +for all plan/spec routes and the local development loop. The former 12-row +product table now has 13 rows: 12 open/conditional families and one already +implemented audit family; lifecycle 3.2 was missing and is restored below. +Grok/Pi and the plugin SessionStart hook are already present. P2/P3 and the +real-binary gate are implemented; alias/bootstrap were closed by owner on +October 18. Dated cutover descriptions below are history, not fresh runtime +measurements or authorization for host changes. + ### Collector / central index / Web cutover — open decisions (added 2026-10-02) PR #446 merged as `3859f788`; HQ has run the r18 index-role Service and @@ -126,32 +135,30 @@ not scheduled engineering work. data, secrets, client settings, or public release channel. Exact hashes, probes, warm-up behavior, and rollback handles are in the runbook. -## Decision pending (2026-07-09 plan-completion audit) - -**Engineering zero ≠ roadmap zero (2026-07-11).** Wave 8 + Round 4 docs closed -the Wave 7 defect ledger (43 terminal confirmed) and removed its open -engineering TODO/follow-ups. The public release baseline selected later is now -complete; the **12 rows below remain owner decisions** and are intentionally -**not** implemented, scheduled, or mislabeled as engineering defects. +## Product decision ledger (reconciled 2026-10-23) -Large product-decision items the audit confirmed **not done** and deliberately -**not** implemented in wave 6. Each needs an explicit product decision before -scheduling. +The July plan-completion audit originally parked 12 product families. Current +source closes the audit-body prerequisite and partially implements human search, +plugin and provider work. Lifecycle 3.2 is explicitly added from its source plan. The +remaining 12 families are product/design choices, not confirmed defects. +The continuous-development request selects prioritization and local work; +irreversible operations and new release channels remain separately authorized. | Item | Source | Audit status | Size estimate | Decision needed | |------|--------|--------------|---------------|-----------------| -| Lifecycle 3.1 multi-factor value score (+ P0-6 measurement-script prerequisite) | `docs/engram-lifecycle-upgrade-plan.md` §3.1 | not_done | L (2–3d + measurement) | Whether to evolve `quality_score` formula and expose bands in GUI | -| Lifecycle 3.4 BM25/CJK ranking + faceting | `docs/engram-lifecycle-upgrade-plan.md` §3.4 | not_done | L (3–4d) | Whether human search ranking upgrade is next after MCP semantic | +| Lifecycle 3.1 multi-factor value score (+ P0-6 measurement-script prerequisite) | `docs/engram-lifecycle-upgrade-plan.md` §3.1 / §3.6 | partial prerequisite | L (formula + real measurement) | Read-only `scripts/db/measure-database.py` now reports storage/FTS/tier/score distributions; six tests and repository-fixture readback passed. GUI bands exist. A current production distribution and the revised factor/outcome contract remain prerequisites to changing the formula. | +| Lifecycle 3.4 BM25/CJK ranking + faceting | `docs/engram-lifecycle-upgrade-plan.md` §3.4; [ranking contract](superpowers/specs/2026-10-23-keyword-ranking-design.md) | partial | re-scope | All-term/CJK ranking and source/project/time/origin/value-band filters implemented locally. Value filtering covers Service and offline queries before result limits and short-token candidate caps. Lifecycle facets and title/quality/recency weighting remain separate decisions. App and Service must be upgraded together for the additive filter field. | | Lifecycle 3.5 tool-result normalization + structured summary job | `docs/engram-lifecycle-upgrade-plan.md` §3.5 | not_done | XL (~15 adapters + summary job) | Accept adapter regression surface vs keep lossy tool results | -| P0.5 value-band badge on session cards + `value_override` | `docs/engram-lifecycle-upgrade-plan.md` §3.1 / P0.5 | not_done | M (~1d after score parity) | Whether to show quality bands before multi-factor scoring | -| Insight supersession via embedding cosine > 0.92 | `docs/p1-semantic-memory-design-2026-06.md` §d | partial (text match only) | M | Whether cosine supersession is required once embeddings are common | +| P0.5 value-band badge on session cards + `value_override` | `docs/engram-lifecycle-upgrade-plan.md` §3.1 / P0.5 | partial | re-scope | Session cards and table cells now display existing bands locally (33 tests and bilingual light/dark component renders passed). Existing score function/thresholds reused; user override remains unimplemented and needs mutation semantics. | +| Insight supersession via embedding cosine > 0.92 | `docs/p1-semantic-memory-design-2026-06.md` §d | partial (text match only); decision pending | M | Saving is synchronous; embeddings arrive in later maintenance. Web supersession checks source visibility inside the save transaction, while backfill has no originating caller policy. Choose whether approximate similarity should replace history, its timing and authority scope before implementation; do not add unconditional backfill supersession. | | P2 auto insight extraction | `docs/engram-lifecycle-upgrade-plan.md` §3.5; p1 design | not_done | L | Opt-in LLM mining of finished sessions into `insights` | | F2 sqlite-vec native target | `docs/p1-semantic-memory-design-2026-06.md` F2 | not_done / superseded by brute-force | XL (native dep + notarization) | Revisit only if brute-force KNN becomes a measured bottleneck | | F3/f corpus mining (`mined_rules`) | p1 design §f; removed feature-cut item 3 | not_done (surface deleted 2026-07-06) | L | Whether to reintroduce rule mining after feature-cut | -| Competitive-relaunch activation after the public baseline (Claude Code plugin; Homebrew/Sparkle channel expansion) | `docs/competitive-relaunch-2026-06.md` | not_done | XL | After `v1.0.5` baseline proof, decide whether plugin integration, Homebrew packaging, or Sparkle updates should be scheduled | -| `ai_audit_log` desensitization design | lifecycle §3.5 P0; no Swift writer today | not_done | M (design) then L | Design body redaction **before** any Swift audit-log writer lands (wave-6 task 9 descope) | -| Provider-branch valuable-missing features (Grok/Pi adapters, session taxonomy filter, runtime capability gates) | `docs/reviews/provider-audit-branch-reconciliation-2026-07.md` | not_done | L each | Branch `codex-provider-audit-remediation` deleted local+origin 2026-07-09 after three-model adjudication; recovery path is archive tag `archive/codex-provider-audit-remediation` (`285453d7`) + the reconciliation doc | +| Competitive-relaunch activation after the public baseline (Claude Code plugin; Homebrew/Sparkle channel expansion) | `docs/competitive-relaunch-2026-06.md` | partial | XL | SessionStart hook and plugin skills exist in `integrations/claude-code/engram`; Stop/SessionEnd insight capture and Homebrew/Sparkle remain choices. | +| `ai_audit_log` body redaction | lifecycle §3.5 P0 | implemented in source | done | `ServiceAIAuditRecorder` uses the writer gate, opt-in bodies and redaction-before-truncation; `ServiceAIAuditRecordTests` covers it. Historical no-writer assumption is stale; broader audit/retention is separate. | +| Provider-branch valuable-missing features (Grok/Pi adapters, session taxonomy filter, runtime capability gates) | `docs/reviews/provider-audit-branch-reconciliation-2026-07.md` | partial | re-scope | Grok/Pi adapters already register in `SessionAdapterFactory`; re-adjudicate taxonomy/capability residuals against current source before recovering anything from the archive tag. | | Sources-sync-3 nav consolidation | `docs/reviews/alignment-design-2026-06-14.md` ~:836,:896 | not_done (explicitly deferred) | M | Whether Sources/Settings nav consolidation is still wanted | +| Lifecycle 3.2 lifecycle-state / shadow archive | `docs/engram-lifecycle-upgrade-plan.md` §3.2 | not_done / design decision | L | Define lifecycle transitions and user-state semantics before schema changes; existing Archive V2 reclamation is a different feature. | ## Closed on 2026-06-20 diff --git a/docs/superpowers/specs/2026-10-08-web-tailscale-identity-auth-design.md b/docs/superpowers/specs/2026-10-08-web-tailscale-identity-auth-design.md index 5cac1013f..4e906e3e2 100644 --- a/docs/superpowers/specs/2026-10-08-web-tailscale-identity-auth-design.md +++ b/docs/superpowers/specs/2026-10-08-web-tailscale-identity-auth-design.md @@ -2,7 +2,7 @@ - **Status**: Accepted. Phase 1 merged in PR #457 (`1e3b5c34`) and deployed to the HQ receiver on 2026-10-08 (package - `remote-server-main-1e3b5c34-20261008`); phase 2 (§6) not started. + `remote-server-main-1e3b5c34-20261008`); phase 2 project-operation attribution and alias audit storage implemented locally on 2026-10-23 (see the [project addendum](2026-10-23-web-project-audit-actor-addendum.md) and [alias audit design](2026-10-23-web-alias-audit-design.md)); host acceptance remains separate. - **Owner**: RemoteServer / Web maintainers - **Date**: 2026-10-08 - **Related**: `docs/superpowers/specs/2026-09-05-collector-server-web-design.md` diff --git a/docs/superpowers/specs/2026-10-23-keyword-ranking-design.md b/docs/superpowers/specs/2026-10-23-keyword-ranking-design.md new file mode 100644 index 000000000..c8b3bc66f --- /dev/null +++ b/docs/superpowers/specs/2026-10-23-keyword-ranking-design.md @@ -0,0 +1,62 @@ +# Human keyword ranking consistency + +Status: implemented and verified locally, 2026-10-23; not pushed/deployed. + +## Baseline and scope + +Lifecycle 3.4's blanket missing-BM25/CJK/facet description is stale. Service +keyword search already uses FTS5 rank, best row per session/token, session-level +AND matching and windowed highlights. SearchPage has source/project/time/origin +filters and calls Service with keyword mode; on failure it uses DatabaseManager +searchWithSnippets. Both readers rank only by the first token. The App also routes +all CJK terms through zero-rank LIKE, unlike Service's scalar-count trigram path. + +This batch corrects these two ranking inconsistencies. No schema, index rebuild, +embedding dependency, scoring preference, UI layout or new facet is introduced. +Value/lifecycle facets and title/quality/recency weighting need separate contracts. +The native MCP's separate ranking policy is outside this human-search change. + +## Acceptance contract + +- Every searchable term must still occur somewhere within a session; matching + terms in separate messages remains valid. Existing tier/visibility/scope and + source/project/date/origin filters apply before the result limit. +- Rank sessions by the sum of each term's existing best-row FTS5 rank. Short + terms keep zero-rank LIKE matching; placing a short term first must not discard + the relevance contribution of later long terms. Sum in canonical token order + so input permutation cannot change floating-point addition order. Preserve + query order for snippets. Repeated terms retain their existing query semantics. +- Ties sort by start_time descending, then session ID ascending. Apply the same + order before LIMIT and in Service's final snippet projection. +- App search and searchWithSnippets use the same scalar-count MATCH/LIKE routing + as Service: three or more Unicode scalars use trigram MATCH, shorter searchable + terms use LIKE. Verify Chinese, Japanese and Korean matching/highlighting. +- Keep Service's bounded short-token candidates and optimized identity-indexed + FTS plan. This does not promise exhaustive recall beyond the existing cap or + full App/Service parity when that cap excludes old short-token matches. +- Compare real Service and both App queries over one temporary database. Capture + RED for Latin multi-term relevance, short-first mixed queries and CJK fallback; + then verify rank-before-limit, permutations, cross-message conjunction, ties, + filters, snippets, both Service FTS layouts and existing syntax/semantic tests. + +## Failure handling and residuals + +Failed tests leave the batch open. Use isolated temporary databases/fixed homes; +no production index, service restart or host mutation. Local tests do not prove +HQ-corpus latency or subjective relevance quality. Record actual acceptance in +the roadmap/reconciliation and CHANGELOG/MEMO; larger lifecycle 3.4 remains open. + +## Local verification receipt + +The first test build caught a missing await in the new fixture setup; corrected +before behavioral RED. Three new App/Service shared-fixture tests then produced +72 expected ordering failures. After the minimal two-reader fix, 163 App data/search +tests and 63 Service search/semantic/insight/Web admission tests passed, zero failures. +They cover six permutations of three terms, relevance before LIMIT, cross-message +AND, ties, mixed short/long Latin/Chinese/Japanese/Korean queries, highlights, +identity-indexed/fallback layouts, visibility/origin/date/tool filters and busy/ +semantic fallback. The query-plan regression retains FTS-driven execution. +Lint (one pre-existing warning, 40 infos) and invariant/module/convention gates passed. +Logs, xcresults and exact Service selectors: `/tmp/engram-leader-keyword-ranking-20261023/`. +No full all-target, rendered UI, hosted CI, production-corpus latency or host rollout +acceptance. No new Swift source files or project generation were needed. diff --git a/docs/superpowers/specs/2026-10-23-web-alias-audit-design.md b/docs/superpowers/specs/2026-10-23-web-alias-audit-design.md new file mode 100644 index 000000000..cb22c8754 --- /dev/null +++ b/docs/superpowers/specs/2026-10-23-web-alias-audit-design.md @@ -0,0 +1,83 @@ +# Web alias mutation audit + +Status: implemented and verified locally, 2026-10-23; not pushed/deployed. +Completes the alias storage gap identified by the +[project actor addendum](2026-10-23-web-project-audit-actor-addendum.md). + +## Current source and scope + +Web aliases use ServiceWebMetadataProducer's exact raw-pair mutations, separately +from native manageProjectAlias's basename normalization. At the pre-change +baseline they had no durable actor record. Project migration_log is an undo/recovery journal for filesystem +operations and must not receive synthetic alias migrations. + +Add bounded, service-owned history for successful Web alias state transitions. +Native alias changes, project-move aliases, authentication attempts and unchanged +requests remain outside this log's claim. No audit browser API or UI is introduced. + +## Contract + +- Both Web alias DTOs carry optional actor metadata. RemoteServer obtains it + from the authenticated editor session and injects it after the existing browser + body allowlist. A later Tailscale header cannot replace the bound session actor; + browser actor fields remain invalid. Credential mode and older RemoteServer + requests without the field use web-editor. Reuse the project audit actor syntax: + 1-254 printable non-whitespace ASCII bytes excluding quote and backslash. +- The capability-protected dispatcher admits the optional field. Validation fails + before mutation. Actor metadata grants no additional authority; existing project + visibility, exact-pair resolution, writer gate and CSRF checks stay active. +- Add web_alias_audit_log through the normal idempotent Swift schema path: + monotonically increasing integer id, UTC created_at, action (add/remove), actor, + published alias and published canonical identities. Store neither raw paths, + transcript content, credentials nor whole request bodies. +- Insert exactly one row when an authorized request changes a pair. Repeated adds, + absent removes and refused requests produce no history. Removal keeps its record + after the alias row disappears. Opaque published identities allow correlation + without retaining a removed raw path; this is not an anonymous audit. +- Mutation, audit insertion and retention execute in one writer.write transaction. + Any audit insertion or pruning failure rolls the entire mutation back and + surfaces the existing generic service-unavailable error. No false success. +- Retain the newest 10,000 transitions by insertion id. Prune within the same + transaction after recording each new transition. Ordering does not depend on + wall-clock monotonicity. This bounded local diagnostic history is not a permanent + compliance ledger. No historical backfill or timer is added. +- Existing databases gain the table without rewriting aliases. Repeat migration + preserves history. Older binaries can ignore the additive table but do not + provide audit guarantees; deployment acceptance must verify Service and + RemoteServer together. An older Service rejects the newly supplied actor key; + upgrade Service before RemoteServer. An older RemoteServer with the new Service + remains usable with web-editor attribution. Existing business responses remain + wire-compatible; rolling back Service alone can interrupt new Web alias writes. +- The log belongs to the local service database, not the user-state import/export + contract. Whole-database backups retain it; a user-data-only restore does not + promise to carry diagnostic history. No new remote data surface is authorized. + +## Acceptance and failure handling + +1. Behavioral RED: existing code loses actor metadata/has no history; test via + DTO decoding, authenticated HTTP and the actual service dispatcher. +2. Verify add/remove actor readback, omission fallback, no-op/refusal silence, + published identities only, and sibling raw-pair preservation. +3. Fail audit insertion and pruning with SQLite triggers and verify rollback of + the alias row and history. Seed 10,000 rows and verify exact retention order. +4. Verify old-schema migration plus repeat migration preserve business data/history. +5. Exercise real HTTP -> Web client -> capability socket -> dispatcher -> writer + -> SQL readback, alongside identity-mode HTTP actor binding and browser override + rejection. Run affected DTO, route, migration and alias tests. +6. Update followups, CHANGELOG and MEMO with actual checks and limitations. Failed + gates leave the item open; no hosted or production claim from local tests. + +## Local verification receipt + +RED: 12 selected Service tests reported 20 failures (four missing-table errors +and 16 invalid-actor assertions); HTTP reported four missing-actor assertions. +GREEN: 13 Service tests, including real HTTP/socket/dispatcher/SQL and real +capability-socket attribution/error readback; 46 HTTP/client/auth/project tests; +20 migration tests. All passed with zero failures. Audit insert/prune triggers +proved rollback, and 10,000 seeded transitions proved clock-independent retention. +Lint, invariant/module/convention and diff checks passed. Evidence: +`/tmp/engram-leader-alias-audit-20261023/` (`service-red`, `http-red`, +`service-green`, `http-green`, `migrations-green` logs/xcresults). +No all-target suite, hosted CI, browser render, production schema upgrade or +mixed-version host rollout was exercised. No source files were added, so project +generation was unnecessary. Both deployment components still need a host receipt. diff --git a/docs/superpowers/specs/2026-10-23-web-project-audit-actor-addendum.md b/docs/superpowers/specs/2026-10-23-web-project-audit-actor-addendum.md new file mode 100644 index 000000000..681649b9a --- /dev/null +++ b/docs/superpowers/specs/2026-10-23-web-project-audit-actor-addendum.md @@ -0,0 +1,72 @@ +# Web project audit actor propagation + +Status: implemented and verified locally, 2026-10-23; not pushed/deployed. Addendum to +[identity auth phase 2](2026-10-08-web-tailscale-identity-auth-design.md#6-phase-2-actor-propagation-separate-pr). + +## Current-source corrections and scope + +At the pre-change baseline, Web project commands overwrote attribution with +`mcp`; batch execution wrote `batch` even when its request carried another +actor. The migration log already stored unrestricted TEXT, but the Swift actor +enum only represented cli/mcp/swift-ui/batch. Forwarding a new DTO field alone is +insufficient. Alias add/remove have neither an actor field nor an audit-log +write in the current service, contrary to the older design assumption. Alias +history was a separate follow-up; the subsequent [alias audit batch](2026-10-23-web-alias-audit-design.md) +now implements and verifies its storage/retention contract locally. The original +project-operation batch below did not include alias auditing. + +## Contract + +- Scope: move, archive, undo and move-batch attribution through RemoteServer, + capability-protected IPC and the existing migration log; no schema changes. +- RemoteServer derives the value from the authenticated editor session: the + bound identity login, or `web-editor` for credential mode. It never trusts a + browser JSON actor or a later identity header. Existing browser key allowlists, + CSRF checks, viewer restrictions and capability-token requirements stay active. +- Web DTOs accept an optional actor for wire compatibility. An absent actor + from an older RemoteServer means `web-editor`, never `mcp`. Nonempty supplied + actors must match the existing identity syntax (1-254 printable ASCII bytes, + no whitespace, quote or backslash). Invalid data fails before filesystem work. +- Use an explicit internal audit override on native command handlers so the + native CLI/MCP actor parsing contract is unchanged. The override participates + in operation fingerprints, including batches, and reaches every item log. +- Audit attribution is metadata, not new authority. Existing local capability + holders remain trusted; no extra IPC command or broader allowlist is added. +- Deploy both service and RemoteServer before claiming identity attribution on + a host: an older service may ignore the additive DTO field. Historical rows + are not rewritten. Existing binaries remain rollback candidates; only newly + emitted actor values change, and the existing schema stores them as text. + +## Acceptance and failure handling + +1. HTTP tests fail on current source for all four project routes in credential + and identity mode. A forged subsequent identity header cannot change the + session actor; JSON actor injection is rejected before the writer. +2. Isolated real project move/archive/undo/batch fixtures fail on current source + when their migration-log actors are read back. Missing actor becomes + `web-editor`; malformed actors fail decoding. Cross-actor reuse of one + operation ID must conflict instead of returning another actor's result. +3. Exercise encoded DTOs across the local capability-protected socket and run + affected Web, registry and native batch regressions. Keep hosted, production + and UI/render acceptance separate; no runtime UI layout changes are involved. +4. A failed check leaves the item open. Record RED/GREEN and any revised scope + in the canonical follow-up and CHANGELOG/MEMO before a local commit. + +## Local verification receipt + +HTTP RED: 1 test / 8 expected attribution assertions. Service RED: 2 tests / +8 expected assertions (invalid actor decoding, audit readback and replay). +Final GREEN: 60 RemoteServer HTTP/auth/IPC tests; 29 service/operation-registry +and long-operation IPC tests; 34 native batch/migration-store tests, all with +zero failures. Lint and invariants passed. Evidence lives under +`/tmp/engram-leader-web-actor-20261023/`. These runs do not prove hosted CI, +production rollout or alias auditing. No schema migration or historical-row +rewrite occurred. + +Continuation correction: the first component GREEN did not traverse the real +Service dispatcher, whose optional payload-key allowlists omitted actor. The +existing audit/replay regression now uses the real Web client, socket listener +and dispatcher before SQL readback. It reproduced invalid at `6d30437a`; after +adding actor to all four IPC allowlists, 18 related service tests passed. +Evidence: `/tmp/engram-leader-web-dispatch-20261023/`. Browser actor injection +remains rejected by the unchanged HTTP body allowlists. diff --git a/macos/Engram.xcodeproj/project.pbxproj b/macos/Engram.xcodeproj/project.pbxproj index 29046016a..659da493b 100644 --- a/macos/Engram.xcodeproj/project.pbxproj +++ b/macos/Engram.xcodeproj/project.pbxproj @@ -46,6 +46,7 @@ 0850F1F77DFD06E476AEB5CE /* TranscriptExportServiceTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = B9EA0FC1FBCAE6B2E1982FFF /* TranscriptExportServiceTests.swift */; }; 086035EC5216FC2238E84AAB /* HumanDrivenFilterTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 04F796D7BB52854AAD26F0A7 /* HumanDrivenFilterTests.swift */; }; 08A25C8BDABCAB8A827FAC63 /* KeychainSecretStore.swift in Sources */ = {isa = PBXBuildFile; fileRef = D091C7C867B2067D91B9E18A /* KeychainSecretStore.swift */; }; + 08FD0BE0CC748290D2A89CA9 /* TranscriptSessionSwitchTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 4AC3F18ABAC749E3C1271AB1 /* TranscriptSessionSwitchTests.swift */; }; 091D285C6C7B58287731D0E1 /* UserDataBackupTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 39C8207CB87F26BE8E440FDA /* UserDataBackupTests.swift */; }; 093633F371A00C17682F0224 /* EngramServiceWebUsageModels.swift in Sources */ = {isa = PBXBuildFile; fileRef = 78487750268B9A2D9BCFCBF8 /* EngramServiceWebUsageModels.swift */; }; 09C19469DBA8A1AC506C51A0 /* ArchiveCursorLegacyCaptureTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = F997F4B70FCDE6CF3DAF46EA /* ArchiveCursorLegacyCaptureTests.swift */; }; @@ -1536,6 +1537,7 @@ 4A4BE77C3F2998DF1CB6A1B7 /* EngramServiceCommandHandler.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = EngramServiceCommandHandler.swift; sourceTree = ""; }; 4A84EE165301392E05DA4055 /* EngramServiceTransport.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = EngramServiceTransport.swift; sourceTree = ""; }; 4A9A02BE2AF9345C8C607A86 /* AliasSheet.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AliasSheet.swift; sourceTree = ""; }; + 4AC3F18ABAC749E3C1271AB1 /* TranscriptSessionSwitchTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TranscriptSessionSwitchTests.swift; sourceTree = ""; }; 4BE8934A97ED99ED63F65BFA /* EngramRemoteServerConfig.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = EngramRemoteServerConfig.swift; sourceTree = ""; }; 4C02528A606FCBB10D813715 /* EngramServiceIPCTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = EngramServiceIPCTests.swift; sourceTree = ""; }; 4D08551975F49EE054CA81AE /* ExplicitCredentialFile.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ExplicitCredentialFile.swift; sourceTree = ""; }; @@ -3140,6 +3142,7 @@ 67C03A12BF8F24EA3BC60214 /* TranscriptFindTests.swift */, E548F540BC0AB1B87C3E7D65 /* TranscriptLabelAndCopyTests.swift */, 1586C9B579B7B0A1183D516F /* TranscriptPagingTimingTests.swift */, + 4AC3F18ABAC749E3C1271AB1 /* TranscriptSessionSwitchTests.swift */, 022A79E53FE22B1D2B622933 /* UIUXPolishWiringTests.swift */, 5F067E1E7AA90D1466F0533E /* UnixSocketTransportTests.swift */, F6CC5E6E6447F051C0B38BD4 /* UsagePressureNotifierTests.swift */, @@ -4833,6 +4836,7 @@ FC63B6E092B62FB290C964A5 /* TranscriptFindTests.swift in Sources */, EAF647BE47D9F63E39CB7F08 /* TranscriptLabelAndCopyTests.swift in Sources */, 6A87E0205EA0A984D40F5BCF /* TranscriptPagingTimingTests.swift in Sources */, + 08FD0BE0CC748290D2A89CA9 /* TranscriptSessionSwitchTests.swift in Sources */, 4F2D7949AAFD84F8836FA144 /* UIUXPolishWiringTests.swift in Sources */, 4BE254288FB83031E03FED4D /* UnixSocketTransportTests.swift in Sources */, 66BA3D6C14B8A99DBC9191B6 /* UsagePressureNotifierTests.swift in Sources */, diff --git a/macos/Engram/Components/SessionCard.swift b/macos/Engram/Components/SessionCard.swift index d89436bc2..83a96b267 100644 --- a/macos/Engram/Components/SessionCard.swift +++ b/macos/Engram/Components/SessionCard.swift @@ -17,6 +17,8 @@ struct SessionCard: View { .lineLimit(1) .truncationMode(.tail) + SessionValueBadge(session: session) + Spacer() if let project = session.project { @@ -57,3 +59,32 @@ struct SessionCard: View { .buttonStyle(.plain) } } + +/// The existing search value bands, with a readable cue beyond color alone. +struct SessionValueBadge: View { + let session: Session + + private var title: LocalizedStringKey { + switch session.valueBand { + case .high: return "High value" + case .medium: return "Medium value" + case .low: return "Low value" + case .unknown: return "" + } + } + + var body: some View { + if let score = session.qualityScore { + Text(title) + .font(.caption2) + .foregroundStyle(session.valueBand == .high ? Theme.green : Theme.secondaryText) + .padding(.horizontal, 6) + .padding(.vertical, 2) + .background(Theme.surfaceHighlight, in: Capsule()) + .fixedSize() + .help(Text("Estimated from session activity. Score: \(score).")) + .accessibilityLabel(Text(title) + Text(", score \(score)")) + .accessibilityIdentifier("session_valueBand") + } + } +} diff --git a/macos/Engram/Core/Database.swift b/macos/Engram/Core/Database.swift index bc989ea0f..daa238eba 100644 --- a/macos/Engram/Core/Database.swift +++ b/macos/Engram/Core/Database.swift @@ -593,13 +593,15 @@ final class DatabaseManager: @unchecked Sendable { sources: Set, projects: Set, origin: String?, - since: String? + since: String?, + valueBand: String? ) { let clauses = SearchFilterPredicates.clauses( sources: Array(sources), projects: Array(projects), origin: origin, - since: since + since: since, + valueBand: valueBand ) for clause in clauses { parts.append("AND \(clause.sql)") @@ -617,8 +619,8 @@ final class DatabaseManager: @unchecked Sendable { /// somewhere in the session. This replaces the old correlated shape that /// probed MATCH once per `sessions` row (audit #25: 80-100s vs 81ms for the /// identical query). `withSnippet` adds the FTS5 `` snippet used by the - /// offline-fallback GUI path. Ordering (`m0.rank, s.start_time DESC`) and the - /// hidden/tier/source/project/since filters are preserved exactly. + /// offline-fallback GUI path. All terms contribute to rank, with recency and + /// session ID as tie-breakers; hidden/tier/source/project/since filters stay. static func keywordSearchSQL( rawTokens: [String], termMatches: [String], @@ -627,7 +629,8 @@ final class DatabaseManager: @unchecked Sendable { since: String?, limit: Int, withSnippet: Bool, - origin: String? = nil + origin: String? = nil, + valueBand: String? = nil ) -> (sql: String, args: [DatabaseValueConvertible]) { var ctes: [String] = [] var joins: [String] = [] @@ -638,7 +641,7 @@ final class DatabaseManager: @unchecked Sendable { // mixed-token-1: classify the original whitespace token. Peeling // the escaped MATCH phrase first turns quoted `"I"` into `%I%` and // empty quotes into a broad `%%` LIKE. - if CJKText.containsCJK(rawTerm) || rawTerm.count < 3 { + if !CJKText.usesTrigramMatch(rawTerm) { if withSnippet { ctes.append(""" \(alias)_hits AS ( @@ -724,7 +727,8 @@ final class DatabaseManager: @unchecked Sendable { sources: sources, projects: projects, origin: origin, - since: since + since: since, + valueBand: valueBand ) parts.append(""" ORDER BY s.start_time DESC @@ -752,10 +756,14 @@ final class DatabaseManager: @unchecked Sendable { sources: sources, projects: projects, origin: origin, - since: since + since: since, + valueBand: valueBand ) + // Match Service's canonical addition order without reordering snippets. + let rankExpression = rawTokens.indices.sorted { rawTokens[$0] < rawTokens[$1] } + .map { "m\($0).rank" }.joined(separator: " + ") parts.append(""" - ORDER BY m0.rank, s.start_time DESC + ORDER BY \(rankExpression), s.start_time DESC, s.id LIMIT ? """) args.append(limit) @@ -768,7 +776,8 @@ final class DatabaseManager: @unchecked Sendable { sources: Set = [], projects: Set = [], since: String? = nil, - origin: String? = nil + origin: String? = nil, + valueBand: String? = nil ) throws -> [Session] { let rawTokens = CJKText.searchableTerms(query) guard !rawTokens.isEmpty || query.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty else { @@ -785,7 +794,8 @@ final class DatabaseManager: @unchecked Sendable { since: since, limit: limit, withSnippet: false, - origin: origin + origin: origin, + valueBand: valueBand ) return try Session.fetchAll( db, @@ -805,7 +815,8 @@ final class DatabaseManager: @unchecked Sendable { sources: Set = [], projects: Set = [], since: String? = nil, - origin: String? = nil + origin: String? = nil, + valueBand: String? = nil ) throws -> [(session: Session, snippet: String)] { let rawTokens = CJKText.searchableTerms(query) guard !rawTokens.isEmpty || query.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty else { @@ -827,7 +838,8 @@ final class DatabaseManager: @unchecked Sendable { since: since, limit: limit, withSnippet: true, - origin: origin + origin: origin, + valueBand: valueBand ) let rows = try Row.fetchAll(db, sql: built.sql, arguments: StatementArguments(built.args)) return try rows.map { row in diff --git a/macos/Engram/Models/Session.swift b/macos/Engram/Models/Session.swift index 419f61d5a..46097be9f 100644 --- a/macos/Engram/Models/Session.swift +++ b/macos/Engram/Models/Session.swift @@ -103,8 +103,8 @@ struct Session: FetchableRecord, Decodable, Identifiable { /// non-skip; min 0 / max 74 / avg 42; scores cluster in 30–39): <=35 low, /// >=60 high — NOT 33/67-of-100, since scores top out near 74. nil → unknown. enum ValueBand: String { case high, medium, low, unknown } - static let valueBandLowMax = 35 - static let valueBandHighMin = 60 + static let valueBandLowMax = SearchFilterPredicates.valueBandLowMax + static let valueBandHighMin = SearchFilterPredicates.valueBandHighMin var valueBand: ValueBand { guard let score = qualityScore else { return .unknown } if score >= Self.valueBandHighMin { return .high } diff --git a/macos/Engram/Resources/Localizable.xcstrings b/macos/Engram/Resources/Localizable.xcstrings index b0c2ea517..41d5838ba 100644 --- a/macos/Engram/Resources/Localizable.xcstrings +++ b/macos/Engram/Resources/Localizable.xcstrings @@ -1,6 +1,54 @@ { "sourceLanguage" : "en", "strings" : { + "Any value" : { + "localizations" : { + "en" : { "stringUnit" : { "state" : "translated", "value" : "Any value" } }, + "zh-Hans" : { "stringUnit" : { "state" : "translated", "value" : "所有价值" } } + } + }, + "Not scored" : { + "localizations" : { + "en" : { "stringUnit" : { "state" : "translated", "value" : "Not scored" } }, + "zh-Hans" : { "stringUnit" : { "state" : "translated", "value" : "未评分" } } + } + }, + "Value" : { + "localizations" : { + "en" : { "stringUnit" : { "state" : "translated", "value" : "Value" } }, + "zh-Hans" : { "stringUnit" : { "state" : "translated", "value" : "价值" } } + } + }, + "High value" : { + "localizations" : { + "en" : { "stringUnit" : { "state" : "translated", "value" : "High value" } }, + "zh-Hans" : { "stringUnit" : { "state" : "translated", "value" : "高价值" } } + } + }, + "Medium value" : { + "localizations" : { + "en" : { "stringUnit" : { "state" : "translated", "value" : "Medium value" } }, + "zh-Hans" : { "stringUnit" : { "state" : "translated", "value" : "中价值" } } + } + }, + "Low value" : { + "localizations" : { + "en" : { "stringUnit" : { "state" : "translated", "value" : "Low value" } }, + "zh-Hans" : { "stringUnit" : { "state" : "translated", "value" : "低价值" } } + } + }, + "Estimated from session activity. Score: %lld." : { + "localizations" : { + "en" : { "stringUnit" : { "state" : "translated", "value" : "Estimated from session activity. Score: %lld." } }, + "zh-Hans" : { "stringUnit" : { "state" : "translated", "value" : "根据会话活动估算，评分：%lld。" } } + } + }, + ", score %lld" : { + "localizations" : { + "en" : { "stringUnit" : { "state" : "translated", "value" : ", score %lld" } }, + "zh-Hans" : { "stringUnit" : { "state" : "translated", "value" : "，评分 %lld" } } + } + }, "%lld files · %@ · %lld indexed · %lld archived · %lld empty ignored" : { "localizations" : { "en" : { "stringUnit" : { "state" : "translated", "value" : "%1$lld files · %2$@ · %3$lld indexed · %4$lld archived · %5$lld empty ignored" } }, diff --git a/macos/Engram/Views/Pages/SearchPageView.swift b/macos/Engram/Views/Pages/SearchPageView.swift index 37e3c30f9..e9da4f8af 100644 --- a/macos/Engram/Views/Pages/SearchPageView.swift +++ b/macos/Engram/Views/Pages/SearchPageView.swift @@ -91,7 +91,39 @@ enum SearchOutcome: Equatable { } } +enum SearchValueFilter: String, CaseIterable { + case all, high, medium, low, unknown + + var requestValueBand: String? { self == .all ? nil : rawValue } + + var label: String { + switch self { + case .all: return "Any value" + case .high: return "High value" + case .medium: return "Medium value" + case .low: return "Low value" + case .unknown: return "Not scored" + } + } +} + +struct SearchValueFilterPicker: View { + @Binding var selection: SearchValueFilter + + var body: some View { + Picker("Value", selection: $selection) { + ForEach(SearchValueFilter.allCases, id: \.self) { filter in + Text(LocalizedStringKey(filter.label)).tag(filter) + } + } + .pickerStyle(.menu) + .fixedSize() + .accessibilityIdentifier("search_valueFilter") + } +} + struct SearchPageView: View { + @Environment(\.locale) private var locale @Environment(DatabaseManager.self) var db @Environment(\.engramServiceClient) var serviceClient @Environment(\.engramFixedDate) var fixedDate @@ -105,6 +137,7 @@ struct SearchPageView: View { @State private var selectedSourceFilter: String? @State private var selectedTimeFilter: SearchTimeFilter = .all @State private var selectedOriginFilter: SearchOriginFilter = .all + @State private var selectedValueFilter: SearchValueFilter = .all /// Keyboard focus on result rows: `.plain` card buttons draw no focus ring, /// so the row overlay below renders one explicitly (accent, 2pt). @FocusState private var focusedResultId: String? @@ -132,6 +165,7 @@ struct SearchPageView: View { private var hasClearableFilters: Bool { selectedSourceFilter != nil || selectedTimeFilter != .all || selectedOriginFilter != .all + || selectedValueFilter != .all || (lockedProject == nil && selectedProjectFilter != nil) } @@ -182,6 +216,7 @@ struct SearchPageView: View { .onChange(of: selectedSourceFilter) { _, _ in triggerSearchIfReady() } .onChange(of: selectedTimeFilter) { _, _ in triggerSearchIfReady() } .onChange(of: selectedOriginFilter) { _, _ in triggerSearchIfReady() } + .onChange(of: selectedValueFilter) { _, _ in triggerSearchIfReady() } .onDisappear { searchTask?.cancel(); searchTask = nil } } @@ -342,12 +377,15 @@ struct SearchPageView: View { } private var filterBar: some View { - HStack(spacing: 8) { - projectFilterControl - sourceFilterControl - timeFilterControl - originFilterControl - Spacer() + VStack(alignment: .leading, spacing: 8) { + HStack(spacing: 8) { + projectFilterControl + sourceFilterControl + timeFilterControl + originFilterControl + Spacer() + } + SearchValueFilterPicker(selection: $selectedValueFilter) } .controlSize(.small) .accessibilityIdentifier("search_filters") @@ -367,6 +405,9 @@ struct SearchPageView: View { if selectedOriginFilter != .all { parts.append(selectedOriginFilter.label) } + if selectedValueFilter != .all { + parts.append(String(localized: String.LocalizationValue(selectedValueFilter.label), locale: locale)) + } return parts.joined(separator: " · ") } @@ -377,6 +418,7 @@ struct SearchPageView: View { selectedSourceFilter = nil selectedTimeFilter = .all selectedOriginFilter = .all + selectedValueFilter = .all } /// The navigation notification shared by result tap and keyboard Enter/Space. @@ -503,6 +545,7 @@ struct SearchPageView: View { let requestedSource = selectedSourceFilter let requestedTime = selectedTimeFilter let requestedOrigin = selectedOriginFilter + let requestedValueBand = selectedValueFilter.requestValueBand guard SearchQueryReadiness.classify(query: requestedQuery) == .ready else { results = [] searchModes = [] @@ -524,7 +567,8 @@ struct SearchPageView: View { project: requestedProject, source: requestedSource, since: since, - origin: requestedOrigin.requestOrigin + origin: requestedOrigin.requestOrigin, + valueBand: requestedValueBand ) ) @@ -552,7 +596,8 @@ struct SearchPageView: View { sources: fallbackSources, projects: fallbackProjects, since: fallbackSince, - origin: requestedOrigin.requestOrigin + origin: requestedOrigin.requestOrigin, + valueBand: requestedValueBand ) }.value guard !Task.isCancelled else { return } diff --git a/macos/Engram/Views/Pages/SessionsPageView.swift b/macos/Engram/Views/Pages/SessionsPageView.swift index 476fc409a..40a0b566b 100644 --- a/macos/Engram/Views/Pages/SessionsPageView.swift +++ b/macos/Engram/Views/Pages/SessionsPageView.swift @@ -904,7 +904,7 @@ struct SessionsPageView: View { /// The "Session" column cell: disclosure chevron + badges + title, plus the /// full row context menu. Suggested-child rows also carry inline /// Confirm/Dismiss buttons. -private struct SessionTableCell: View { +struct SessionTableCell: View { let row: SessionTableRow var onToggleExpand: (() -> Void)? = nil var onResume: ((Session) -> Void)? = nil @@ -972,6 +972,8 @@ private struct SessionTableCell: View { .foregroundStyle(row.isSuggestedChild ? Theme.tertiaryText : Theme.primaryText) .accessibilityIdentifier("sessions_row_\(row.index)") + SessionValueBadge(session: session) + if row.isSuggestedChild { Button("Confirm") { onConfirmSuggestion?(session) } .font(.system(size: 10)) diff --git a/macos/Engram/Views/PopoverView.swift b/macos/Engram/Views/PopoverView.swift index e1eddddcc..78f012e2d 100644 --- a/macos/Engram/Views/PopoverView.swift +++ b/macos/Engram/Views/PopoverView.swift @@ -181,7 +181,7 @@ struct PopoverView: View { } else { ScrollView { VStack(alignment: .leading, spacing: 2) { - let groups = groupedByDate(data.recentSessions) + let groups = Self.groupedByDate(data.recentSessions) ForEach(groups) { group in Text(group.key) .font(.caption2).fontWeight(.semibold) @@ -385,13 +385,14 @@ struct PopoverView: View { return f }() - private struct DateGroup: Identifiable { + struct DateGroup: Identifiable { let key: String let sessions: [Session] - var id: String { key } + // Activity ordering can split one date into multiple nonempty groups. + var id: String { sessions[0].id } } - private func groupedByDate(_ sessions: [Session]) -> [DateGroup] { + static func groupedByDate(_ sessions: [Session]) -> [DateGroup] { let cal = Calendar.current var groups: [(String, [Session])] = [] var currentKey = "" diff --git a/macos/Engram/Views/SessionDetailView.swift b/macos/Engram/Views/SessionDetailView.swift index a6ad67ac3..f58b61934 100644 --- a/macos/Engram/Views/SessionDetailView.swift +++ b/macos/Engram/Views/SessionDetailView.swift @@ -19,6 +19,51 @@ struct SessionDetailView: View { /// Prime the find bar with this query when the session opens from keyword /// search (nil for every other entry point — find bar stays closed/empty). var searchTerm: String? = nil + #if DEBUG + // Instance-scoped hooks let mounted-view tests hold uncancellable work and + // observe the real SwiftUI state without touching live transcripts or IPC. + var transcriptTestHooks: TranscriptTestHooks? = nil + struct TranscriptTestHooks { + var parse: (String, Int, Int?) async throws -> ( + messages: [ChatMessage], producedCount: Int, truncated: Bool, parseFailed: Bool + ) + var attach: (String, @escaping () -> TranscriptTestState, @escaping (TranscriptTestAction) -> Void) -> Void + var beforeIndexedApply: (() async -> Void)? = nil + var copy: ((String) -> Void)? = nil + } + enum TranscriptTestAction { case page, all, copy } + struct TranscriptTestState: Equatable { + let sessionID: String? + let messages: [String] + let indexed: [String] + let loading: Bool + let paging: Bool + let hasMore: Bool + let failed: Bool + } + private var transcriptTestState: TranscriptTestState { + TranscriptTestState(sessionID: displayedSessionId, messages: messages.map(\.content), + indexed: indexedMessages.map { $0.message.content }, loading: isLoadingMessages, + paging: isLoadingMore, hasMore: hasMoreToLoad, failed: transcriptParseFailed) + } + @ViewBuilder private var transcriptTestProbe: some View { + if transcriptTestHooks != nil { + Color.clear.onAppear { attachTranscriptTestHooks() } + .onChange(of: session.id) { _, _ in attachTranscriptTestHooks() } + } + } + private func attachTranscriptTestHooks() { + // Read the mounted State locations on demand. Publishing snapshots via + // onChange/task can lag coalesced render passes in an offscreen window. + transcriptTestHooks?.attach(session.id, { transcriptTestState }) { action in + switch action { + case .page: loadMoreMessages(all: false) + case .all: loadMoreMessages(all: true) + case .copy: copyAllTranscript() + } + } + } + #endif @Environment(DatabaseManager.self) var db @Environment(\.engramServiceClient) var serviceClient @Environment(\.accessibilityReduceMotion) private var reduceMotion @@ -99,6 +144,7 @@ struct SessionDetailView: View { @State private var isLoadingMore = false @State private var loadedProducedCount = 0 @State private var transcriptLoadTask: Task? = nil + @State private var transcriptLoadToken: UUID? @State private var displayFilterTask: Task? = nil @State private var displayFilterSessionId: String? = nil // Bumped whenever `displayIndexed` is recomputed; the match-scan task is keyed @@ -969,6 +1015,7 @@ struct SessionDetailView: View { loadedProducedCount = 0 transcriptLoadTask?.cancel() transcriptLoadTask = nil + transcriptLoadToken = nil // Clear transcript-derived + nav state too, or a stale per-type position // from the previous session can index out of the new (shorter) one. displayIndexed = [] @@ -1051,9 +1098,13 @@ struct SessionDetailView: View { parentMutationTask = nil relatedTask?.cancel(); relatedTask = nil transcriptLoadTask?.cancel(); transcriptLoadTask = nil + transcriptLoadToken = nil favoriteLoadGeneration = nil displayedSessionId = nil } + #if DEBUG + .background { transcriptTestProbe } + #endif } // MARK: - Related Sessions @@ -1643,9 +1694,11 @@ struct SessionDetailView: View { return } transcriptLoadTask?.cancel() + let loadToken = UUID() + transcriptLoadToken = loadToken + isLoadingMore = true transcriptLoadTask = Task { - isLoadingMore = true - defer { isLoadingMore = false } + defer { finishTranscriptLoad(loadToken) } await appendMessages(all: true) if Task.isCancelled { return } guard !transcriptParseFailed else { @@ -1662,6 +1715,9 @@ struct SessionDetailView: View { private func copyLoadedTranscript() { let text = TranscriptText.conversationText(indexedMessages) + #if DEBUG + if let copy = transcriptTestHooks?.copy { copy(text); return } + #endif NSPasteboard.general.clearContents() NSPasteboard.general.setString(text, forType: .string) } @@ -1700,6 +1756,11 @@ struct SessionDetailView: View { offset: Int, limit: Int? ) async throws -> (messages: [ChatMessage], producedCount: Int, truncated: Bool, parseFailed: Bool) { + #if DEBUG + if let hooks = transcriptTestHooks { + return try await hooks.parse(session.id, offset, limit) + } + #endif let path = resolvedTranscriptPath() let source = session.source let sessionId = session.id @@ -1746,6 +1807,9 @@ struct SessionDetailView: View { } return IndexedMessage.build(from: fullSnapshot) }.value + #if DEBUG + if let beforeApply = transcriptTestHooks?.beforeIndexedApply { await beforeApply() } + #endif // The detached build can outlive a session switch; don't clobber the reset. if Task.isCancelled { return } indexedMessages = built.messages @@ -1841,13 +1905,24 @@ struct SessionDetailView: View { private func loadMoreMessages(all: Bool) { guard !isLoadingMore else { return } transcriptLoadTask?.cancel() + let loadToken = UUID() + transcriptLoadToken = loadToken + isLoadingMore = true transcriptLoadTask = Task { - isLoadingMore = true - defer { isLoadingMore = false } + defer { finishTranscriptLoad(loadToken) } await appendMessages(all: all) } } + private func finishTranscriptLoad(_ loadToken: UUID) { + // Cancelled parse/build work may finish after switching away and back. + // Only the task that owns the current page may release its busy state. + guard transcriptLoadToken == loadToken else { return } + isLoadingMore = false + transcriptLoadToken = nil + transcriptLoadTask = nil + } + @ViewBuilder private var transcriptLoadMoreFooter: some View { VStack(spacing: 8) { diff --git a/macos/EngramCollectorCore/CollectorRuntime.swift b/macos/EngramCollectorCore/CollectorRuntime.swift index 063e248da..0935af71f 100644 --- a/macos/EngramCollectorCore/CollectorRuntime.swift +++ b/macos/EngramCollectorCore/CollectorRuntime.swift @@ -92,6 +92,7 @@ public actor CollectorRuntime { private var closed = false private let diagnostics: @Sendable (String) -> Void private let eventStreamFactory: EventStreamFactory + private let retryClock: @Sendable () -> ContinuousClock.Instant private var identitySuspendedRoots = Set() private var reportedIdentitySuspendedRoots = Set() /// Roots whose stored FSEvents checkpoint belongs to another native epoch. Kept for @@ -105,6 +106,11 @@ public actor CollectorRuntime { private var blockedStreaks: [Int: Int] = [:] private var blockedRoots = Set() private var reportedBlockedRoots = Set() + private struct BlockedRootRetry { + let delaySeconds: Int + let deadline: ContinuousClock.Instant + } + private var blockedRetries: [Int: BlockedRootRetry] = [:] /// Awaited immediately before a background capture turn checks `stopping`. /// Tests hold this until `stop` has entered; production leaves it nil. private var testCaptureTurnBarrier: (@Sendable () async -> Void)? @@ -128,7 +134,8 @@ public actor CollectorRuntime { owner: CollectorInventoryOwner, catalog: ArchiveCatalog, worker: CollectorPublicationWorker, uploader: CollectorPublicationWorker, diagnostics: @escaping @Sendable (String) -> Void, - eventStreamFactory: @escaping EventStreamFactory) { + eventStreamFactory: @escaping EventStreamFactory, + retryClock: @escaping @Sendable () -> ContinuousClock.Instant) { self.settingsURL = settingsURL self.configuration = configuration self.owner = owner @@ -137,6 +144,7 @@ public actor CollectorRuntime { self.uploader = uploader self.diagnostics = diagnostics self.eventStreamFactory = eventStreamFactory + self.retryClock = retryClock } public static func open( @@ -156,7 +164,8 @@ public actor CollectorRuntime { diagnostics: @escaping @Sendable (String) -> Void, eventStreamFactory: @escaping EventStreamFactory = { request, budget in CollectorNativeEventStream(request: request, budget: budget) - } + }, + retryClock: @escaping @Sendable () -> ContinuousClock.Instant = { ContinuousClock.now } ) throws -> CollectorRuntime? { try Task.checkCancellation() guard let configuration = try CollectorRuntimeConfiguration.load(at: settingsURL) else { return nil } @@ -184,7 +193,7 @@ public actor CollectorRuntime { configuration: configuration, replicas: replicas, settingsURL: settingsURL) return CollectorRuntime(settingsURL: settingsURL, configuration: configuration, owner: owner, catalog: catalog, worker: worker, uploader: uploader, diagnostics: diagnostics, - eventStreamFactory: eventStreamFactory) + eventStreamFactory: eventStreamFactory, retryClock: retryClock) } catch { try catalog.close() throw error @@ -252,6 +261,7 @@ public actor CollectorRuntime { var steps: [(Int, CollectorEventCoordinatorStep)] = [] for index in coordinators.keys.sorted() { try Task.checkCancellation() + guard !isBackingOff(index) else { continue } guard let coordinator = coordinators[index] else { continue } do { let observed = started.observations[Data(configuration.rootConfigurations[index].rootID.utf8)] @@ -260,12 +270,20 @@ public actor CollectorRuntime { steps.append((index, step)) if let bootstrap = step.bootstrap { if case .blocked = bootstrap.outcome { + let delay = min((blockedRetries[index]?.delaySeconds ?? 0) * 2, 60) + let seconds = max(delay, 1) + blockedRetries[index] = .init(delaySeconds: seconds, + deadline: retryClock().advanced(by: .seconds(seconds))) + captureRootIDs.remove(Data(configuration.rootConfigurations[index].rootID.utf8)) let streak = min((blockedStreaks[index] ?? 0) + 1, Self.blockedReportThreshold) blockedStreaks[index] = streak if streak == Self.blockedReportThreshold { blockedRoots.insert(index) } } else { blockedStreaks[index] = nil blockedRoots.remove(index) + // Reopening the first directory is not recovery: a later + // directory can block again in this same multi-turn scan. + if bootstrap.outcome == .finished { blockedRetries[index] = nil } } } } catch where Self.isUnavailableSource(error) { @@ -279,6 +297,8 @@ public actor CollectorRuntime { private func mailboxSamples(historyWaiting: [Int: Bool]) throws -> [CollectorCaptureScheduleSample] { try coordinators.keys.sorted().compactMap { index in + // A pending loss on a cooling root must not force extra capture turns. + guard !isBackingOff(index) else { return nil } guard let coordinator = coordinators[index] else { return nil } let snap = try coordinator.snapshot() return CollectorCaptureScheduleSample( @@ -480,6 +500,7 @@ public actor CollectorRuntime { _ = try owner.activateStoredRootForPublication(configuration.rootConfigurations[index]) blockedStreaks[index] = nil blockedRoots.remove(index) + blockedRetries[index] = nil if let coordinator = coordinators[index] { do { try coordinator.stop() } catch where Self.isUnavailableSource(error) { @@ -499,6 +520,7 @@ public actor CollectorRuntime { var live: [(Int, CollectorRootConfiguration)] = [] for (index, root) in configuration.rootConfigurations.enumerated() { try Task.checkCancellation() + guard !isBackingOff(index) else { continue } if let coordinator = coordinators[index], try coordinator.snapshot().phase != .recoveryRequired { live.append((index, root)) @@ -519,6 +541,7 @@ public actor CollectorRuntime { } for (index, root) in configuration.rootConfigurations.enumerated() { try Task.checkCancellation() + guard !isBackingOff(index) else { continue } if live.contains(where: { $0.0 == index }) || epochSuspendedRoots.contains(index) { continue } do { // Missing or recovery-required coordinators still enroll and start. @@ -598,8 +621,13 @@ public actor CollectorRuntime { diagnostics(line) } + private func isBackingOff(_ index: Int) -> Bool { + guard let retry = blockedRetries[index] else { return false } + return retryClock() < retry.deadline + } + /// docs/invariants.md "Collector Bootstrap Scan Finishes Past Vanished Directories": a root whose bootstrap walk is blocked - /// (enumerationUnavailable or unsafeEntry) is retried every turn. Report one line, + /// (enumerationUnavailable or unsafeEntry) retries after bounded backoff. Report one line, /// root IDs only, each time the set of persistently blocked roots changes. private func reportBlockedRootsIfChanged() { guard blockedRoots != reportedBlockedRoots else { return } diff --git a/macos/EngramCoreTests/ArchiveV2/ArchiveModelTests.swift b/macos/EngramCoreTests/ArchiveV2/ArchiveModelTests.swift index 6fe48b98c..91ca53592 100644 --- a/macos/EngramCoreTests/ArchiveV2/ArchiveModelTests.swift +++ b/macos/EngramCoreTests/ArchiveV2/ArchiveModelTests.swift @@ -17,6 +17,19 @@ final class ArchiveModelTests: XCTestCase { ArchiveV2Hash.sha256(Data("abc".utf8)), "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad" ) + XCTAssertEqual( + ArchiveV2Hash.sha256(Data("286".utf8)), + "00328ce57bbc14b33bd6695bc8eb32cdf2fb5f3a7d89ec14a42825e15d39df60", + "Hex encoding must preserve leading zero bytes" + ) + XCTAssertEqual( + ArchiveV2Hash.sha256(Data((0...255).map(UInt8.init))), + "40aff2e9d2d8922e47afd4648e6967497158785fbd1da870e7110266bf944880" + ) + XCTAssertEqual( + ArchiveV2Hash.sha256(Data(repeating: 0xff, count: 65)), + "855c9a28fff2bbbd71327165554a00c6fdd44d5d58c1f12b8fdff29b5d67f001" + ) XCTAssertTrue(ArchiveV2Hash.isValidSHA256(sourceDigest)) XCTAssertFalse(ArchiveV2Hash.isValidSHA256(String(repeating: "A", count: 64))) XCTAssertFalse(ArchiveV2Hash.isValidSHA256(String(repeating: "a", count: 63))) diff --git a/macos/EngramCoreTests/Database/MigrationRunnerTests.swift b/macos/EngramCoreTests/Database/MigrationRunnerTests.swift index 5d36ebc7d..959a034f4 100644 --- a/macos/EngramCoreTests/Database/MigrationRunnerTests.swift +++ b/macos/EngramCoreTests/Database/MigrationRunnerTests.swift @@ -143,6 +143,34 @@ final class MigrationRunnerTests: XCTestCase { ]) } + func testWebAliasAuditMigrationIsAdditiveAndPreservesHistory() throws { + let writer = try EngramDatabaseWriter(path: databasePath("alias-audit.sqlite")) + try writer.migrate() + XCTAssertTrue(try writer.read { try $0.tableExists("web_alias_audit_log") }) + // Model an existing database from before this additive table. + try writer.write { db in + try db.execute(sql: """ + DROP TABLE IF EXISTS web_alias_audit_log; + INSERT INTO project_aliases(alias, canonical) VALUES ('/old/path', '/new/path'); + """) + } + try writer.migrate() + XCTAssertEqual(try writer.read { try Int.fetchOne($0, sql: "SELECT COUNT(*) FROM web_alias_audit_log") }, 0) + XCTAssertEqual(try writer.read { + try String.fetchOne($0, sql: "SELECT canonical FROM project_aliases WHERE alias = '/old/path'") + }, "/new/path") + try writer.write { db in + try db.execute(sql: """ + INSERT INTO web_alias_audit_log(action, actor, alias, canonical) + VALUES ('add', 'editor@example.com', 'old', 'new'); + """) + } + try writer.migrate() + XCTAssertEqual(try writer.read { + try String.fetchOne($0, sql: "SELECT actor FROM web_alias_audit_log") + }, "editor@example.com") + } + func testMigrationIsIdempotentAcrossRepeatedRuns() throws { let writer = try EngramDatabaseWriter(path: databasePath("idempotent.sqlite")) diff --git a/macos/EngramCoreTests/ImplementationDigestExtractorTests.swift b/macos/EngramCoreTests/ImplementationDigestExtractorTests.swift index 116a7b8ff..54633e82e 100644 --- a/macos/EngramCoreTests/ImplementationDigestExtractorTests.swift +++ b/macos/EngramCoreTests/ImplementationDigestExtractorTests.swift @@ -1,10 +1,12 @@ +import Darwin import EngramCoreRead import XCTest final class ImplementationDigestExtractorTests: XCTestCase { func testLongAssistantKeepsTailCompletionMarkersWithoutSlowCharacterScanning() { let content = String(repeating: "x", count: 26 * 1024 * 1024) + "\nchecks run: success" - let start = Date() + // Measure CPU work rather than time descheduled on a shared CI runner. + let start = clock() let beats = ImplementationDigestExtractor.extract(messages: [ NormalizedMessage(role: .user, content: "Implement the complete history reader", timestamp: "2026-09-07T00:00:00Z"), NormalizedMessage(role: .assistant, content: content, timestamp: "2026-09-07T00:00:01Z"), @@ -12,7 +14,8 @@ final class ImplementationDigestExtractorTests: XCTestCase { XCTAssertEqual(beats.count, 1) XCTAssertEqual(beats.first?.status, .completed) XCTAssertEqual(beats.first?.kind, .implementation) - XCTAssertLessThan(Date().timeIntervalSince(start), 10) + let cpuSeconds = Double(clock() - start) / Double(CLOCKS_PER_SEC) + XCTAssertLessThan(cpuSeconds, 10) } func testExtractsCompletionReportAndFiltersMachineTurns() { diff --git a/macos/EngramCoreWrite/Database/EngramMigrations.swift b/macos/EngramCoreWrite/Database/EngramMigrations.swift index 53574bb8d..5972da12b 100644 --- a/macos/EngramCoreWrite/Database/EngramMigrations.swift +++ b/macos/EngramCoreWrite/Database/EngramMigrations.swift @@ -142,6 +142,15 @@ enum EngramMigrations { PRIMARY KEY (alias, canonical) ); + CREATE TABLE IF NOT EXISTS web_alias_audit_log ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')), + action TEXT NOT NULL CHECK (action IN ('add', 'remove')), + actor TEXT NOT NULL, + alias TEXT NOT NULL, + canonical TEXT NOT NULL + ); + CREATE TABLE IF NOT EXISTS session_local_state ( session_id TEXT PRIMARY KEY REFERENCES sessions(id) ON DELETE CASCADE, hidden_at TEXT, diff --git a/macos/EngramCoreWrite/ProjectMove/Batch.swift b/macos/EngramCoreWrite/ProjectMove/Batch.swift index b712574ba..ec785b443 100644 --- a/macos/EngramCoreWrite/ProjectMove/Batch.swift +++ b/macos/EngramCoreWrite/ProjectMove/Batch.swift @@ -117,6 +117,7 @@ public enum BatchError: Error, Equatable { } public struct BatchOverrides: Sendable { + public var actor: MigrationLogActor public var homeDirectory: URL public var lockPath: String? public var force: Bool @@ -127,11 +128,13 @@ public struct BatchOverrides: Sendable { homeDirectory: URL = FileManager.default.homeDirectoryForCurrentUser, lockPath: String? = nil, force: Bool = false, + actor: MigrationLogActor = .batch, runOperation: (@Sendable (BatchOperation, String, String) async throws -> PipelineResult)? = nil ) { self.homeDirectory = homeDirectory self.lockPath = lockPath self.force = force + self.actor = actor self.runOperation = runOperation } } @@ -302,7 +305,7 @@ public enum Batch { force: overrides.force, archived: op.archive, auditNote: op.note, - actor: .batch, + actor: overrides.actor, homeDirectory: overrides.homeDirectory, lockPath: overrides.lockPath, shouldCancel: shouldCancel, diff --git a/macos/EngramCoreWrite/ProjectMove/MigrationLogStore.swift b/macos/EngramCoreWrite/ProjectMove/MigrationLogStore.swift index 5b0fb9e26..a243bda94 100644 --- a/macos/EngramCoreWrite/ProjectMove/MigrationLogStore.swift +++ b/macos/EngramCoreWrite/ProjectMove/MigrationLogStore.swift @@ -15,11 +15,19 @@ import Foundation import EngramCoreRead import GRDB -public enum MigrationLogActor: String, Sendable { - case cli - case mcp - case swiftUI = "swift-ui" - case batch +public struct MigrationLogActor: RawRepresentable, Hashable, Sendable { + public let rawValue: String + public static let cli = Self(rawValue: "cli")! + public static let mcp = Self(rawValue: "mcp")! + public static let swiftUI = Self(rawValue: "swift-ui")! + public static let batch = Self(rawValue: "batch")! + + /// Web attribution is a bounded identity value, never filesystem authority. + public init?(rawValue: String) { + guard !rawValue.isEmpty, rawValue.utf8.count <= 254, + rawValue.utf8.allSatisfy({ (33...126).contains($0) && $0 != 34 && $0 != 92 }) else { return nil } + self.rawValue = rawValue + } } public struct StartMigrationInput: Sendable { diff --git a/macos/EngramMCP/Core/MCPToolRegistry.swift b/macos/EngramMCP/Core/MCPToolRegistry.swift index 807f508e5..543176340 100644 --- a/macos/EngramMCP/Core/MCPToolRegistry.swift +++ b/macos/EngramMCP/Core/MCPToolRegistry.swift @@ -72,8 +72,8 @@ enum MCPToolRegistry { /// tools/list + argument validation surface. Search `mode` enum is gated by /// `SessionVectorSearchAvailability` on the given database path. static func tools(dbPath: String, runtimeRole: EngramRuntimeRole = .local) -> [MCPToolDefinition] { - let semanticUsable = runtimeRole.allowsLocalIndex - && FileManager.default.fileExists(atPath: dbPath) + guard runtimeRole.allowsLocalIndex else { return [] } + let semanticUsable = FileManager.default.fileExists(atPath: dbPath) && ((try? MCPDatabase(path: dbPath).vectorAvailabilityForAdvertising().isUsable) ?? false) return allToolDefinitions.compactMap { definition in guard !unavailableNativeProjectOperationTools.contains(definition.name) else { diff --git a/macos/EngramMCPTests/EngramMCPExecutableTests.swift b/macos/EngramMCPTests/EngramMCPExecutableTests.swift index d7e1ac3a8..b75599703 100644 --- a/macos/EngramMCPTests/EngramMCPExecutableTests.swift +++ b/macos/EngramMCPTests/EngramMCPExecutableTests.swift @@ -72,8 +72,8 @@ func makeHermeticRPCEnvironment( } final class EngramMCPExecutableTests: XCTestCase { - func testRuntimeRoleDiscoveryKeepsStaticKeywordToolsWithoutLocalCapabilities_repro() throws { - for document in [#"{"runtimeRole":"collector"}"#, #"{"runtimeRole":"replica"}"#, #"{"runtimeRole":null}"#] { + func testRuntimeRoleDiscoveryOmitsUnavailableTools_repro() throws { + for document in [#"{"runtimeRole":"collector"}"#, #"{"runtimeRole":"replica"}"#, #"{"runtimeRole":null}"#, "not-json"] { let requests = [ #"{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25"}}"#, #"{"jsonrpc":"2.0","id":2,"method":"tools/list"}"#, @@ -94,9 +94,7 @@ final class EngramMCPExecutableTests: XCTestCase { XCTAssertNil(response["error"]) XCTAssertNil(response["result"]?["isError"]) let tools = try XCTUnwrap(response["result"]?["tools"]?.arrayValue) - XCTAssertEqual(tools.count, 27) - let search = try XCTUnwrap(tools.first { $0["name"]?.stringValue == "search" }) - XCTAssertEqual(search["inputSchema"]?["properties"]?["mode"]?["enum"]?.arrayValue?.compactMap(\.stringValue), ["keyword"]) + XCTAssertTrue(tools.isEmpty, "Unavailable roles must not advertise callable tools: \(document)") if id == 3 { XCTAssertEqual(response["result"]?["resultType"]?.stringValue, "complete") XCTAssertNotNil(response["result"]?["ttlMs"]) @@ -167,6 +165,22 @@ final class EngramMCPExecutableTests: XCTestCase { } } + func testRuntimeRoleLocalIndexAndMissingKeyRetainToolCatalog_repro() throws { + for document in [#"{"runtimeRole":"local"}"#, #"{"runtimeRole":"index"}"#, #"{"unrelated":true}"#] { + let responses = try rpcSession([ + #"{"jsonrpc":"2.0","id":1,"method":"tools/list"}"#, + ], prepare: { sandbox in + try self.prepareRuntimeRole(document, in: sandbox) + try self.seedUsableSessionEmbeddings(at: XCTUnwrap(sandbox.environment["ENGRAM_MCP_DB_PATH"])) + }) + let tools = try XCTUnwrap(responses.first?["result"]?["tools"]?.arrayValue) + XCTAssertEqual(tools.count, 27) + let search = try XCTUnwrap(tools.first { $0["name"]?.stringValue == "search" }) + XCTAssertEqual(search["inputSchema"]?["properties"]?["mode"]?["enum"]?.arrayValue?.compactMap(\.stringValue), + ["keyword", "semantic", "hybrid"]) + } + } + private func prepareRuntimeRole(_ document: String, in sandbox: HermeticRPCEnvironment) throws { let settings = URL(fileURLWithPath: try XCTUnwrap(sandbox.environment["ENGRAM_SETTINGS_PATH"])) try Data(document.utf8).write(to: settings) diff --git a/macos/EngramRemoteServer/Core/EngramRemoteServerApp.swift b/macos/EngramRemoteServer/Core/EngramRemoteServerApp.swift index 3d4fb505f..bb8e53968 100644 --- a/macos/EngramRemoteServer/Core/EngramRemoteServerApp.swift +++ b/macos/EngramRemoteServer/Core/EngramRemoteServerApp.swift @@ -307,7 +307,7 @@ public final class EngramRemoteServerApp: Sendable { let boundary = WebRequestBoundary(configuration: configuration) WebAuthRoutes.mount(on: router, boundary: boundary, sessions: sessions) WebReadRoutes.mount(on: router, surface: surface) - WebWriteRoutes.mount(on: router, surface: writeSurface) + WebWriteRoutes.mount(on: router, surface: writeSurface, boundary: boundary, sessions: sessions) WebUIRoutes.mount(on: router, tailscaleServeIdentity: configuration.usesTailscaleServeIdentity) middleware = WebRequestBoundary.Middleware(boundary: boundary, sessions: sessions) } else { diff --git a/macos/EngramRemoteServer/Core/WebAuthSessionStore.swift b/macos/EngramRemoteServer/Core/WebAuthSessionStore.swift index 0fad7c2f8..53713dd6a 100644 --- a/macos/EngramRemoteServer/Core/WebAuthSessionStore.swift +++ b/macos/EngramRemoteServer/Core/WebAuthSessionStore.swift @@ -115,6 +115,12 @@ actor WebAuthSessionStore { validSession(sessionToken: sessionToken)?.login } + /// Resolve editor attribution and authority from the same live session. + func writeActor(sessionToken: String) -> String? { + guard let session = validSession(sessionToken: sessionToken), session.canWrite else { return nil } + return session.login ?? "web-editor" + } + private func validSession(sessionToken: String) -> Session? { let instant = monotonicNow() purgeExpired(at: instant) diff --git a/macos/EngramRemoteServer/Core/WebUIRoutes.swift b/macos/EngramRemoteServer/Core/WebUIRoutes.swift index d341c28f5..ea5d2809c 100644 --- a/macos/EngramRemoteServer/Core/WebUIRoutes.swift +++ b/macos/EngramRemoteServer/Core/WebUIRoutes.swift @@ -1363,6 +1363,10 @@ enum WebUIRoutes { } catch (error) { if (version !== aiConfigVersion || activePage !== "settings") return; if (error.status === 401) { expireSession(); return; } + if (error.status === 409) { + setText(status, "Changes were not saved. Change AI endpoints and credentials in the server's local settings, or restore the previous endpoint before saving other edits."); + return; + } if (error.status === 403) settingsCanWrite = false; aiConfigUncertain = true; setText(status, "Could not confirm saving. Your draft is retained. Reload saved settings before retrying."); diff --git a/macos/EngramRemoteServer/Core/WebWriteRoutes.swift b/macos/EngramRemoteServer/Core/WebWriteRoutes.swift index 6d72a7be3..66ab9e553 100644 --- a/macos/EngramRemoteServer/Core/WebWriteRoutes.swift +++ b/macos/EngramRemoteServer/Core/WebWriteRoutes.swift @@ -94,15 +94,30 @@ enum WebWriteRoutes { ) } - static func mount(on router: Router, surface: Surface) { + static func mount( + on router: Router, surface: Surface, + boundary: WebRequestBoundary, sessions: WebAuthSessionStore + ) { router.post("/web/api/settings/aliases") { request, _ in - try await mutate(request, keys: addKeys) { data in - try await surface.addAlias(try JSONDecoder().decode(EngramServiceWebAddAliasRequest.self, from: data)) + guard let token = boundary.sessionToken(in: request), + let actor = await sessions.writeActor(sessionToken: token) else { + return Response(status: .unauthorized) + } + return try await mutate(request, keys: addKeys) { data in + try await surface.addAlias( + try JSONDecoder().decode(EngramServiceWebAddAliasRequest.self, from: attributed(data, actor: actor)) + ) } } router.delete("/web/api/settings/aliases") { request, _ in - try await mutate(request, keys: removeKeys) { data in - try await surface.removeAlias(try JSONDecoder().decode(EngramServiceWebRemoveAliasRequest.self, from: data)) + guard let token = boundary.sessionToken(in: request), + let actor = await sessions.writeActor(sessionToken: token) else { + return Response(status: .unauthorized) + } + return try await mutate(request, keys: removeKeys) { data in + try await surface.removeAlias( + try JSONDecoder().decode(EngramServiceWebRemoveAliasRequest.self, from: attributed(data, actor: actor)) + ) } } router.post("/web/api/settings/sources") { request, _ in @@ -208,37 +223,53 @@ enum WebWriteRoutes { } } router.post("/web/api/projects/move") { request, _ in - try await mutate( + guard let token = boundary.sessionToken(in: request), + let actor = await sessions.writeActor(sessionToken: token) else { + return Response(status: .unauthorized) + } + return try await mutate( request, required: moveKeys, allowed: moveKeys.union(moveOptionalKeys) ) { data in - try await surface.projectMove(try JSONDecoder().decode(EngramServiceWebProjectMoveRequest.self, from: data)) + try await surface.projectMove(try JSONDecoder().decode(EngramServiceWebProjectMoveRequest.self, from: attributed(data, actor: actor))) } } router.post("/web/api/projects/archive") { request, _ in - try await mutate( + guard let token = boundary.sessionToken(in: request), + let actor = await sessions.writeActor(sessionToken: token) else { + return Response(status: .unauthorized) + } + return try await mutate( request, required: archiveKeys, allowed: archiveKeys.union(archiveOptionalKeys) ) { data in try await surface.projectArchive( - try JSONDecoder().decode(EngramServiceWebProjectArchiveRequest.self, from: data) + try JSONDecoder().decode(EngramServiceWebProjectArchiveRequest.self, from: attributed(data, actor: actor)) ) } } router.post("/web/api/projects/undo") { request, _ in - try await mutate( + guard let token = boundary.sessionToken(in: request), + let actor = await sessions.writeActor(sessionToken: token) else { + return Response(status: .unauthorized) + } + return try await mutate( request, required: undoKeys, allowed: undoKeys.union(undoOptionalKeys) ) { data in - try await surface.projectUndo(try JSONDecoder().decode(EngramServiceWebProjectUndoRequest.self, from: data)) + try await surface.projectUndo(try JSONDecoder().decode(EngramServiceWebProjectUndoRequest.self, from: attributed(data, actor: actor))) } } router.post("/web/api/projects/move-batch") { request, _ in - try await mutate( + guard let token = boundary.sessionToken(in: request), + let actor = await sessions.writeActor(sessionToken: token) else { + return Response(status: .unauthorized) + } + return try await mutate( request, required: batchKeys, allowed: batchKeys.union(batchOptionalKeys), maximumBodyBytes: maximumProjectBodyBytes ) { data in try await surface.projectMoveBatch( - try JSONDecoder().decode(EngramServiceWebProjectMoveBatchRequest.self, from: data) + try JSONDecoder().decode(EngramServiceWebProjectMoveBatchRequest.self, from: attributed(data, actor: actor)) ) } } @@ -251,6 +282,15 @@ enum WebWriteRoutes { } } + /// Called only after the browser key allowlist has rejected an actor field. + private static func attributed(_ data: Data, actor: String) throws -> Data { + guard var body = try JSONSerialization.jsonObject(with: data) as? [String: Any] else { + throw EngramServiceWebWriteClientError.invalid + } + body["actor"] = try EngramServiceWebProjectValidation.auditActor(actor) + return try JSONSerialization.data(withJSONObject: body) + } + private struct ParentIdBody: Decodable { let parentId: String } @@ -323,6 +363,7 @@ enum WebWriteRoutes { private static func writeFailure(_ error: EngramServiceWebWriteClientError) -> Response { switch error { + case .aiEndpointRequiresLocalUpdate: return Response(status: .conflict) case .invalid: return Response(status: .badRequest) case .stale: return Response(status: .conflict) case .notFound: return Response(status: .notFound) diff --git a/macos/EngramRemoteServerCoreTests/EngramRemoteServerTests.swift b/macos/EngramRemoteServerCoreTests/EngramRemoteServerTests.swift index 20af75cb5..7487eee41 100644 --- a/macos/EngramRemoteServerCoreTests/EngramRemoteServerTests.swift +++ b/macos/EngramRemoteServerCoreTests/EngramRemoteServerTests.swift @@ -1,5 +1,10 @@ import XCTest import CryptoKit +import Hummingbird +import HTTPTypes +import Logging +import NIOCore +import NIOEmbedded @testable import EngramRemoteServerCore import EngramCoreWrite @@ -32,6 +37,43 @@ private final class PortWaiter: @unchecked Sendable { } } +private actor LegacyRequestStream { + enum Failure: Error { case disconnected } + private let chunks: [ByteBuffer] + private let fails: Bool + private(set) var reads = 0 + + init(chunks: [ByteBuffer], fails: Bool = false) { + self.chunks = chunks + self.fails = fails + } + + func next() throws -> ByteBuffer? { + let index = reads + reads += 1 + if index < chunks.count { return chunks[index] } + if fails { throw Failure.disconnected } + return nil + } +} + +private actor LegacyResponseBytes { + private(set) var data = Data() + func append(_ buffer: ByteBuffer) { data.append(contentsOf: buffer.readableBytesView) } +} + +private struct LegacyResponseWriter: ResponseBodyWriter { + let storage: LegacyResponseBytes + mutating func write(_ buffer: ByteBuffer) async throws { await storage.append(buffer) } + consuming func finish(_ trailingHeaders: HTTPFields?) async throws {} +} + +private func legacyResponseData(_ response: Response) async throws -> Data { + let storage = LegacyResponseBytes() + try await response.body.write(LegacyResponseWriter(storage: storage)) + return await storage.data +} + final class EngramRemoteServerTests: XCTestCase { private var tempDir: URL! @@ -261,25 +303,122 @@ final class EngramRemoteServerTests: XCTestCase { XCTAssertEqual((response as? HTTPURLResponse)?.statusCode, 413) } - func testLegacyBodyCollectionMapsOnlySizeErrorsTo413_repro() throws { - let macosRoot = URL(fileURLWithPath: #filePath) - .deletingLastPathComponent() - .deletingLastPathComponent() - let source = try String( - contentsOf: macosRoot.appendingPathComponent("EngramRemoteServer/Core/EngramRemoteServerApp.swift"), - encoding: .utf8 + func testLegacyBundleBodyErrorsDoNotCreateOrReplaceBlobs_repro() async throws { + let (app, store) = try bodyCollectionApp() + let original = Data("original".utf8) + try store.put("existing.bundle", plaintext: original) + let before = try regularFileBytes(under: tempDir.appendingPathComponent("legacy")) + + for key in ["new.bundle", "existing.bundle"] { + // Two individually legal chunks cross the limit only when accumulated. + for (chunks, failure, status) in [ + (["part"], true, 503), + (["1234", "56789"], false, 413), + ] { + let body = LegacyRequestStream(chunks: chunks.map { ByteBuffer(string: $0) }, fails: failure) + let response = try await legacyResponse(app, method: .put, path: "/v1/bundles/\(key)", body: body) + XCTAssertEqual(response.status.code, status, key) + let reads = await body.reads + XCTAssertEqual(reads, 2, "The route must actually consume the stream") + XCTAssertFalse(try store.exists("new.bundle")) + XCTAssertEqual(try store.get("existing.bundle"), original) + XCTAssertEqual(try regularFileBytes(under: tempDir.appendingPathComponent("legacy")), before) + } + } + + let complete = LegacyRequestStream(chunks: [ByteBuffer(string: "1234"), ByteBuffer(string: "5678")]) + let response = try await legacyResponse(app, method: .put, path: "/v1/bundles/new.bundle", body: complete) + XCTAssertEqual(response.status.code, 201, "A complete body exactly at the limit remains valid") + XCTAssertEqual(try store.get("new.bundle"), Data("12345678".utf8)) + } + + func testLegacyMCPBodyCollectionMapsOnlySizeErrorsTo413_repro() async throws { + let (app, _) = try bodyCollectionApp() + let halfLimit = ByteBuffer(repeating: 0x20, count: 524_288) + let cases: [([ByteBuffer], Bool, Int, Int, String)] = [ + ([ByteBuffer(string: #"{"jsonrpc":"2.0","#)], true, 503, -32603, "Request body unavailable"), + ([halfLimit, halfLimit, ByteBuffer(string: " ")], false, 413, -32600, "Request body too large"), + ([ByteBuffer(string: "{")], false, 400, -32700, "Parse error"), + ] + for (chunks, fails, status, code, message) in cases { + let stream = LegacyRequestStream(chunks: chunks, fails: fails) + let response = try await legacyResponse(app, method: .post, path: "/mcp", token: "mcp-token", body: stream) + XCTAssertEqual(response.status.code, status) + let data = try await legacyResponseData(response) + let json = try XCTUnwrap(try JSONSerialization.jsonObject(with: data) as? [String: Any]) + XCTAssertEqual(json["jsonrpc"] as? String, "2.0") + XCTAssertTrue(json["id"] is NSNull) + let error = try XCTUnwrap(json["error"] as? [String: Any]) + XCTAssertEqual(error["code"] as? Int, code) + XCTAssertEqual(error["message"] as? String, message) + XCTAssertNil(json["result"]) + } + } + + func testLegacyMCPCompleteStreamAtLimitStillInitializes() async throws { + let (app, _) = try bodyCollectionApp() + let json = ByteBuffer(string: #"{"jsonrpc":"2.0","id":"complete","method":"initialize"}"#) + let stream = LegacyRequestStream(chunks: [ + json, ByteBuffer(repeating: 0x20, count: 1_048_576 - json.readableBytes), + ]) + let response = try await legacyResponse(app, method: .post, path: "/mcp", token: "mcp-token", body: stream) + XCTAssertEqual(response.status.code, 200) + let data = try await legacyResponseData(response) + let result = try XCTUnwrap(try JSONSerialization.jsonObject(with: data) as? [String: Any]) + XCTAssertEqual(result["id"] as? String, "complete") + XCTAssertNotNil(result["result"] as? [String: Any]) + XCTAssertNil(result["error"]) + } + + func testLegacyBodyIsNotReadBeforeAuthorizationAndOriginChecks() async throws { + let (app, store) = try bodyCollectionApp() + let cases: [(HTTPRequest.Method, String, String, String?, Int)] = [ + (.put, "/v1/bundles/denied.bundle", "wrong-token", nil, 401), + (.post, "/mcp", "wrong-token", nil, 401), + (.post, "/mcp", "mcp-token", "https://untrusted.example", 403), + ] + for (method, path, token, origin, status) in cases { + let stream = LegacyRequestStream(chunks: [], fails: true) + let response = try await legacyResponse(app, method: method, path: path, token: token, origin: origin, body: stream) + XCTAssertEqual(response.status.code, status) + let reads = await stream.reads + XCTAssertEqual(reads, 0, "Rejected requests must not consume their body") + } + XCTAssertFalse(try store.exists("denied.bundle")) + } + + private func bodyCollectionApp() throws -> (EngramRemoteServerApp, BlobStore) { + let key = SymmetricKey(size: .bits256) + let root = tempDir.appendingPathComponent("legacy") + let config = EngramRemoteServerConfig( + host: "127.0.0.1", port: 0, storeRoot: root, + bearerToken: "legacy-token", atRestKey: key, maxBundleBytes: 8, + archiveV2: EngramRemoteArchiveConfig( + serverID: "hq", root: tempDir.appendingPathComponent("archive"), + bearerToken: "archive-token", atRestKey: SymmetricKey(size: .bits256) + ), + mcp: EngramRemoteMCPConfig(bearerToken: "mcp-token") ) - let putStart = try XCTUnwrap(source.range(of: "router.put(\"/v1/bundles/:key\")")) - let putEnd = try XCTUnwrap(source.range(of: "router.delete(\"/v1/bundles/:key\")", range: putStart.upperBound.. Response { + var fields: HTTPFields = [.authorization: "Bearer \(token)"] + if let origin { fields[HTTPField.Name("Origin")!] = origin } + let request = Request( + head: HTTPRequest(method: method, scheme: "http", authority: "localhost", path: path, headerFields: fields), + body: RequestBody(asyncSequence: AsyncThrowingStream { try await body.next() }) + ) + let channel = EmbeddedChannel() + defer { _ = try? channel.finish() } + let context = BasicRequestContext(source: ApplicationRequestContextSource( + channel: channel, logger: Logger(label: "legacy-body-test") + )) + return try await app.buildRouter().buildResponder().respond(to: request, context: context) } // MCP retro F12: validates the legacy initialize handshake over a real bound socket. diff --git a/macos/EngramRemoteServerCoreTests/WebAiSettingsHTTPTests.swift b/macos/EngramRemoteServerCoreTests/WebAiSettingsHTTPTests.swift index 5812eb50d..3db316a6a 100644 --- a/macos/EngramRemoteServerCoreTests/WebAiSettingsHTTPTests.swift +++ b/macos/EngramRemoteServerCoreTests/WebAiSettingsHTTPTests.swift @@ -125,6 +125,21 @@ final class WebAiSettingsHTTPTests: XCTestCase { XCTAssertEqual(allowed.summaryPrompt, "Decisions\nNext steps") } + func testEndpointCredentialRejectionMapsToConflictWithoutLeakingSecrets() async throws { + try await withServer { server, recorder in + recorder.writeError = .aiEndpointRequiresLocalUpdate + let editor = try await Self.login(server, credential: Self.editor) + let response = try await server.request( + "POST", "/web/api/settings/ai", + headers: Self.headers + [("Cookie", editor)], + body: Data(#"{"aiBaseURL":"https://other.example/v1"}"#.utf8) + ) + XCTAssertEqual(response.status, 409) + XCTAssertTrue(response.body.isEmpty) + XCTAssertEqual(recorder.patches.count, 1) + } + } + private func withServer( editorCredential: String? = editor, failOnWrite: Bool = false, diff --git a/macos/EngramRemoteServerCoreTests/WebAliasHTTPTests.swift b/macos/EngramRemoteServerCoreTests/WebAliasHTTPTests.swift index 95c2fdbfb..f80b495b3 100644 --- a/macos/EngramRemoteServerCoreTests/WebAliasHTTPTests.swift +++ b/macos/EngramRemoteServerCoreTests/WebAliasHTTPTests.swift @@ -102,12 +102,47 @@ final class WebAliasHTTPTests: XCTestCase { XCTAssertEqual(recorder.adds.count, 0) } + func testAliasActorIsBoundToSessionAndBrowserOverrideIsRejected_repro() async throws { + for identityMode in [false, true] { + let recorder = AliasRecorder() + try await withServer(write: recorder.surface(), identityMode: identityMode) { server in + let cookie: String + if identityMode { + let login = try await server.request("POST", "/web/api/auth", + headers: Self.writeHeaders + [("Tailscale-User-Login", "editor@example.com")], + body: Data("{}".utf8)) + XCTAssertEqual(login.status, 204) + cookie = try XCTUnwrap(login.header("set-cookie")?.split(separator: ";").first.map(String.init)) + } else { + cookie = try await Self.login(server, credential: Self.editor) + } + for method in ["POST", "DELETE"] { + let headers = Self.writeHeaders + [("Cookie", cookie), ("Tailscale-User-Login", "forged@example.com")] + let response = try await server.request(method, "/web/api/settings/aliases", + headers: headers, body: Data(#"{"canonical":"new","alias":"old"}"#.utf8)) + XCTAssertEqual(response.status, 503) // recording surface has no backing service + let encoded = try method == "POST" + ? JSONEncoder().encode(XCTUnwrap(recorder.adds.last)) + : JSONEncoder().encode(XCTUnwrap(recorder.removes.last)) + let payload = try XCTUnwrap(JSONSerialization.jsonObject(with: encoded) as? [String: Any]) + XCTAssertEqual(payload["actor"] as? String, identityMode ? "editor@example.com" : "web-editor") + let rejected = try await server.request(method, "/web/api/settings/aliases", headers: headers, + body: Data(#"{"canonical":"new","alias":"old","actor":"forged@example.com"}"#.utf8)) + XCTAssertEqual(rejected.status, 400) + } + } + XCTAssertEqual(recorder.adds.count, 1) + XCTAssertEqual(recorder.removes.count, 1) + } + } + private func withServer( write: WebWriteRoutes.Surface, + identityMode: Bool = false, operation: (D4HTTPServer) async throws -> Void ) async throws { let app = try EngramRemoteServerApp( - config: config(), + config: config(identityMode: identityMode), webReadClientFactory: { _ in WebReadRoutes.messagesOnly({ _ in throw EngramServiceWebReadClientError.unavailable }) }, @@ -121,11 +156,13 @@ final class WebAliasHTTPTests: XCTestCase { try await server.stop() } - private func config() throws -> EngramRemoteServerConfig { + private func config(identityMode: Bool = false) throws -> EngramRemoteServerConfig { try EngramRemoteServerConfig( host: "127.0.0.1", port: 0, storeRoot: directory.appendingPathComponent("legacy"), bearerToken: "d4-bearer", atRestKey: SymmetricKey(data: Data(repeating: 1, count: 32)), - web: try EngramRemoteWebConfig( + web: try identityMode + ? EngramRemoteWebConfig(origin: Self.origin, tailscaleServeViewers: "reader@example.com", editors: "editor@example.com") + : EngramRemoteWebConfig( origin: Self.origin, viewerCredential: Self.viewer, serverBearerCredentials: ["d4-bearer"], editorCredential: Self.editor ), @@ -160,7 +197,8 @@ private final class AliasRecorder: @unchecked Sendable { addAlias: { request in if failOnCall { XCTFail("Write surface must not run") } self.lock.lock(); self.addCalls.append(request); self.lock.unlock() - return try XCTUnwrap(self.addResult) + guard let result = self.addResult else { throw EngramServiceWebWriteClientError.unavailable } + return result }, removeAlias: { request in if failOnCall { XCTFail("Write surface must not run") } diff --git a/macos/EngramRemoteServerCoreTests/WebAuthSessionTests.swift b/macos/EngramRemoteServerCoreTests/WebAuthSessionTests.swift index a86d13ed9..4d9379050 100644 --- a/macos/EngramRemoteServerCoreTests/WebAuthSessionTests.swift +++ b/macos/EngramRemoteServerCoreTests/WebAuthSessionTests.swift @@ -18,6 +18,10 @@ final class WebAuthSessionTests: XCTestCase { let editorCanWrite = await store.canWrite(sessionToken: editorToken) XCTAssertFalse(viewerCanWrite) XCTAssertTrue(editorCanWrite) + let viewerWriteActor = await store.writeActor(sessionToken: viewerToken) + let editorWriteActor = await store.writeActor(sessionToken: editorToken) + XCTAssertNil(viewerWriteActor) + XCTAssertEqual(editorWriteActor, "web-editor") for rawCredential in [viewer, "fixture-editor", "fixture-service", "", String(repeating: "A", count: 43)] { let canWrite = await store.canWrite(sessionToken: rawCredential) XCTAssertFalse(canWrite) @@ -26,6 +30,8 @@ final class WebAuthSessionTests: XCTestCase { let revokedCanWrite = await store.canWrite(sessionToken: editorToken) let viewerStillValid = await store.isAuthenticated(sessionToken: viewerToken) XCTAssertFalse(revokedCanWrite) + let revokedWriteActor = await store.writeActor(sessionToken: editorToken) + XCTAssertNil(revokedWriteActor) XCTAssertTrue(viewerStillValid) guard let secondEditor = token(await store.login(credential: "fixture-editor")) else { return } clock.advance(seconds: 900) @@ -33,6 +39,8 @@ final class WebAuthSessionTests: XCTestCase { let expiredCanRead = await store.isAuthenticated(sessionToken: secondEditor) XCTAssertFalse(expiredCanWrite) XCTAssertFalse(expiredCanRead) + let expiredWriteActor = await store.writeActor(sessionToken: secondEditor) + XCTAssertNil(expiredWriteActor) } func testViewerOnlyConfigurationNeverGrantsWriteAuthority() async throws { @@ -282,6 +290,10 @@ final class WebAuthSessionTests: XCTestCase { let editorActor = await store.actor(sessionToken: editor) XCTAssertEqual(readerActor, "reader@example.com") XCTAssertEqual(editorActor, "editor@example.com") + let readerWriteActor = await store.writeActor(sessionToken: reader) + let editorWriteActor = await store.writeActor(sessionToken: editor) + XCTAssertNil(readerWriteActor) + XCTAssertEqual(editorWriteActor, "editor@example.com") for login in ["Reader@example.com", "reader@example.com ", "reader@example.co", "nobody@example.com", ""] { let outcome = await store.login(identity: WebIdentity(login: login)) XCTAssertEqual(outcome, .forbidden, login) diff --git a/macos/EngramRemoteServerCoreTests/WebProjectMigrationHTTPTests.swift b/macos/EngramRemoteServerCoreTests/WebProjectMigrationHTTPTests.swift index 84f766033..87e13590f 100644 --- a/macos/EngramRemoteServerCoreTests/WebProjectMigrationHTTPTests.swift +++ b/macos/EngramRemoteServerCoreTests/WebProjectMigrationHTTPTests.swift @@ -98,9 +98,46 @@ final class WebProjectMigrationHTTPTests: XCTestCase { } } + func testProjectAuditActorComesFromSessionAndRejectsBrowserOverride_repro() async throws { + for identityMode in [false, true] { + try await withServer(identityMode: identityMode) { server, recorder in + let cookie: String + if identityMode { + let login = try await server.request("POST", "/web/api/auth", + headers: Self.headers + [("Tailscale-User-Login", "editor@example.com")], + body: Data("{}".utf8)) + XCTAssertEqual(login.status, 204) + cookie = try XCTUnwrap(login.header("set-cookie")?.split(separator: ";").first.map(String.init)) + } else { + cookie = try await Self.login(server, credential: Self.editor) + } + let cases: [(String, [String: Any])] = [ + ("move", ["src": "/tmp/a", "dst": "/tmp/b", "dry_run": true]), + ("archive", ["src": "/tmp/a", "dry_run": true]), + ("undo", ["migration_id": "fixture-migration"]), + ("move-batch", ["yaml": "{\"version\":1,\"operations\":[]}", "dry_run": true]), + ] + for (route, fields) in cases { + var payload = fields + payload["operation_id"] = UUID().uuidString + let headers = Self.headers + [("Cookie", cookie), ("Tailscale-User-Login", "forged@example.com")] + let response = try await server.request("POST", "/web/api/projects/" + route, + headers: headers, body: JSONSerialization.data(withJSONObject: payload)) + XCTAssertEqual(response.status, 503) // recording surface has no backing service + XCTAssertEqual(recorder.actors[route], identityMode ? "editor@example.com" : "web-editor", route) + payload["actor"] = "forged@example.com" + let rejected = try await server.request("POST", "/web/api/projects/" + route, + headers: headers, body: JSONSerialization.data(withJSONObject: payload)) + XCTAssertEqual(rejected.status, 400) + } + } + } + } + private func withServer( editorCredential: String? = editor, failOnWrite: Bool = false, + identityMode: Bool = false, operation: (D4HTTPServer, ProjectMigrationRecorder) async throws -> Void ) async throws { let directory = URL(fileURLWithPath: "/tmp/eg-d15-\(UUID().uuidString.prefix(8))", isDirectory: true) @@ -124,16 +161,30 @@ final class WebProjectMigrationHTTPTests: XCTestCase { writes.projectMove = { request in if failOnWrite { XCTFail("Rejected request reached project move writer") } recorder.record(request) + try recorder.recordActor("move", request) + throw EngramServiceWebWriteClientError.unavailable + } + writes.projectArchive = { request in + try recorder.recordActor("archive", request) throw EngramServiceWebWriteClientError.unavailable } + writes.projectUndo = { request in + try recorder.recordActor("undo", request) + throw EngramServiceWebWriteClientError.unavailable + } + writes.projectMoveBatch = { request in + try recorder.recordActor("move-batch", request) + throw EngramServiceWebWriteClientError.unavailable + } + let web = try identityMode + ? EngramRemoteWebConfig(origin: Self.origin, tailscaleServeViewers: "reader@example.com", editors: "editor@example.com") + : EngramRemoteWebConfig(origin: Self.origin, viewerCredential: Self.viewer, + serverBearerCredentials: ["project-migration-bearer"], editorCredential: editorCredential) let app = try EngramRemoteServerApp( config: EngramRemoteServerConfig( host: "127.0.0.1", port: 0, storeRoot: directory.appendingPathComponent("store"), bearerToken: "project-migration-bearer", atRestKey: SymmetricKey(data: Data(repeating: 1, count: 32)), - web: EngramRemoteWebConfig( - origin: Self.origin, viewerCredential: Self.viewer, - serverBearerCredentials: ["project-migration-bearer"], editorCredential: editorCredential - ), + web: web, webServiceSocketPath: directory.appendingPathComponent("service.sock").path ), webReadClientFactory: { _ in reads }, @@ -185,6 +236,15 @@ final class WebProjectMigrationHTTPTests: XCTestCase { private final class ProjectMigrationRecorder: @unchecked Sendable { private let lock = NSLock() private var calls: [EngramServiceWebProjectMoveRequest] = [] + private var auditActors: [String: String] = [:] + var actors: [String: String] { + lock.lock(); defer { lock.unlock() }; return auditActors + } + func recordActor(_ route: String, _ request: T) throws { + let object = try XCTUnwrap(JSONSerialization.jsonObject(with: JSONEncoder().encode(request)) as? [String: Any]) + lock.lock(); defer { lock.unlock() } + auditActors[route] = object["actor"] as? String + } var cwds: EngramServiceWebProjectCwdsResponse? var history: EngramServiceWebProjectMigrationsResponse? var historyReads = 0 diff --git a/macos/EngramRemoteServerCoreTests/WebWriteClientTests.swift b/macos/EngramRemoteServerCoreTests/WebWriteClientTests.swift index a89f778fd..2e5b87422 100644 --- a/macos/EngramRemoteServerCoreTests/WebWriteClientTests.swift +++ b/macos/EngramRemoteServerCoreTests/WebWriteClientTests.swift @@ -72,6 +72,47 @@ final class WebWriteClientTests: XCTestCase { XCTAssertFalse(String(decoding: http, as: UTF8.self).contains(token)) } + func testProjectAuditActorCrossesCapabilityProtectedIPC_repro() async throws { + for (index, command) in ["webProjectMove", "webProjectArchive", "webProjectUndo", "webProjectMoveBatch"].enumerated() { + XCTAssertTrue(ServiceCapabilityToken.requiresToken(command)) + let path = directory.appendingPathComponent("actor-\(index).sock").path + try writeToken("actor-capability-fixture", socketPath: path) + let server = try WriteSocketFixture(path: path) { fd, bytes in + let incoming = try JSONDecoder().decode(EngramServiceRequestEnvelope.self, from: bytes) + XCTAssertEqual(incoming.command, command) + XCTAssertEqual(incoming.capabilityToken, "actor-capability-fixture") + let payload = try XCTUnwrap(JSONSerialization.jsonObject(with: XCTUnwrap(incoming.payload)) as? [String: Any]) + XCTAssertEqual(payload["actor"] as? String, "editor@example.com") + let response = EngramServiceResponseEnvelope.failure(requestId: incoming.requestId, + error: EngramServiceErrorEnvelope(name: "ServiceUnavailable", message: "fixture", retryPolicy: "later")) + try EngramServiceSocketIO.writeFrame(JSONEncoder().encode(response), to: fd, requestTimeout: 1) + } + defer { server.stop() } + let client = try EngramServiceWebWriteClient(socketPath: path) + let id = UUID().uuidString + do { + switch command { + case "webProjectMove": + _ = try await client.projectMove(EngramServiceWebProjectMoveRequest( + src: "/tmp/a", dst: "/tmp/b", dryRun: true, operationId: id, actor: "editor@example.com")) + case "webProjectArchive": + _ = try await client.projectArchive(EngramServiceWebProjectArchiveRequest( + src: "/tmp/a", dryRun: true, operationId: id, actor: "editor@example.com")) + case "webProjectUndo": + _ = try await client.projectUndo(EngramServiceWebProjectUndoRequest( + migrationId: "fixture", operationId: id, actor: "editor@example.com")) + default: + _ = try await client.projectMoveBatch(EngramServiceWebProjectMoveBatchRequest( + yaml: "{}", dryRun: true, operationId: id, actor: "editor@example.com")) + } + XCTFail("fixture must return unavailable after inspecting IPC") + } catch { + XCTAssertEqual(error as? EngramServiceWebWriteClientError, .unavailable) + } + XCTAssertEqual(server.requestCount, 1) + } + } + func testMissingTokenDoesNotOpenIPC() async throws { let path = directory.appendingPathComponent("missing.sock").path let server = try WriteSocketFixture(path: path) { _, _ in diff --git a/macos/EngramService/Core/EngramServiceCommandHandler+ProjectMigration.swift b/macos/EngramService/Core/EngramServiceCommandHandler+ProjectMigration.swift index d40cc2472..24e2c8816 100644 --- a/macos/EngramService/Core/EngramServiceCommandHandler+ProjectMigration.swift +++ b/macos/EngramService/Core/EngramServiceCommandHandler+ProjectMigration.swift @@ -40,16 +40,17 @@ extension EngramServiceCommandHandler { static func projectMove( _ request: EngramServiceProjectMoveRequest, writerGate: ServiceWriterGate, + auditActor: MigrationLogActor? = nil, hooks: ProjectMoveLongOpHooks = .none ) async throws -> ServiceWriterGateResult { let trimmedSrc = request.src.trimmingCharacters(in: .whitespacesAndNewlines) let trimmedDst = request.dst.trimmingCharacters(in: .whitespacesAndNewlines) ServiceLogger.notice( - "projectMove requested actor=\(request.actor ?? "mcp") dryRun=\(request.dryRun) force=\(request.force) src=\(trimmedSrc) dst=\(trimmedDst) operationId=\(request.operationId ?? "")", + "projectMove requested actor=\(auditActor?.rawValue ?? request.actor ?? "mcp") dryRun=\(request.dryRun) force=\(request.force) src=\(trimmedSrc) dst=\(trimmedDst) operationId=\(request.operationId ?? "")", category: .writer ) let operationId = normalizeOperationId(request.operationId) - let actor = parseActor(request.actor) ?? .mcp + let actor = auditActor ?? parseActor(request.actor) ?? .mcp let fingerprint = ProjectMoveOperationFingerprint.encode([ "kind": "move", "src": trimmedSrc, @@ -114,14 +115,15 @@ extension EngramServiceCommandHandler { static func projectArchive( _ request: EngramServiceProjectArchiveRequest, writerGate: ServiceWriterGate, + auditActor: MigrationLogActor? = nil, hooks: ProjectMoveLongOpHooks = .none ) async throws -> ServiceWriterGateResult { ServiceLogger.notice( - "projectArchive requested actor=\(request.actor ?? "mcp") dryRun=\(request.dryRun) force=\(request.force) src=\(request.src) archiveTo=\(request.archiveTo ?? "") operationId=\(request.operationId ?? "")", + "projectArchive requested actor=\(auditActor?.rawValue ?? request.actor ?? "mcp") dryRun=\(request.dryRun) force=\(request.force) src=\(request.src) archiveTo=\(request.archiveTo ?? "") operationId=\(request.operationId ?? "")", category: .writer ) let operationId = normalizeOperationId(request.operationId) - let actor = parseActor(request.actor) ?? .mcp + let actor = auditActor ?? parseActor(request.actor) ?? .mcp // Fingerprint uses src + archive category; dst resolved after reservation. let fingerprint = ProjectMoveOperationFingerprint.encode([ "kind": "archive", @@ -198,14 +200,15 @@ extension EngramServiceCommandHandler { static func projectUndo( _ request: EngramServiceProjectUndoRequest, writerGate: ServiceWriterGate, + auditActor: MigrationLogActor? = nil, hooks: ProjectMoveLongOpHooks = .none ) async throws -> ServiceWriterGateResult { ServiceLogger.notice( - "projectUndo requested actor=\(request.actor ?? "mcp") force=\(request.force) migrationId=\(request.migrationId) operationId=\(request.operationId ?? "")", + "projectUndo requested actor=\(auditActor?.rawValue ?? request.actor ?? "mcp") force=\(request.force) migrationId=\(request.migrationId) operationId=\(request.operationId ?? "")", category: .writer ) let operationId = normalizeOperationId(request.operationId) - let actor = parseActor(request.actor) ?? .mcp + let actor = auditActor ?? parseActor(request.actor) ?? .mcp let fingerprint = ProjectMoveOperationFingerprint.encode([ "kind": "undo", "migrationId": request.migrationId, @@ -248,6 +251,7 @@ extension EngramServiceCommandHandler { static func projectMoveBatch( _ request: EngramServiceProjectMoveBatchRequest, writerGate: ServiceWriterGate, + auditActor: MigrationLogActor? = nil, hooks: ProjectMoveLongOpHooks = .none ) async throws -> ServiceWriterGateResult { ServiceLogger.notice( @@ -260,7 +264,7 @@ extension EngramServiceCommandHandler { "yaml": request.yaml, "dryRun": String(request.dryRun), "force": String(request.force), - "actor": request.actor ?? "", + "actor": auditActor?.rawValue ?? request.actor ?? "", ]) if let cached = try await resolveExistingOperation( @@ -299,7 +303,8 @@ extension EngramServiceCommandHandler { writer: writer, overrides: BatchOverrides( homeDirectory: homeDirectoryURL(), - force: request.force + force: request.force, + actor: auditActor ?? .batch ), shouldCancel: { Task.isCancelled } ) @@ -323,7 +328,8 @@ extension EngramServiceCommandHandler { } let overrides = BatchOverrides( homeDirectory: homeDirectoryURL(), - force: request.force + force: request.force, + actor: auditActor ?? .batch ) if let batchOverride = hooks.batchRunOverride { return await batchOverride(document, writer, overrides) diff --git a/macos/EngramService/Core/EngramServiceCommandHandler+WebAiSettings.swift b/macos/EngramService/Core/EngramServiceCommandHandler+WebAiSettings.swift index c2b5c3804..74e5d9758 100644 --- a/macos/EngramService/Core/EngramServiceCommandHandler+WebAiSettings.swift +++ b/macos/EngramService/Core/EngramServiceCommandHandler+WebAiSettings.swift @@ -1,6 +1,8 @@ import Darwin import Foundation import EngramCoreRead +import LocalAuthentication +import Security extension EngramServiceCommandHandler { static func webAiSettings( @@ -14,18 +16,101 @@ extension EngramServiceCommandHandler { static func webPatchAiSettings( _ request: EngramServiceWebPatchAiSettingsRequest, - settingsURL: URL + settingsURL: URL, + environment: [String: String] = ProcessInfo.processInfo.environment, + keychainMayContain: (String) -> Bool = { webAiKeychainMayContain($0) } ) throws -> EngramServiceWebAiSettingsResponse { do { try SecureSettingsFileWriter.mutateJSON(at: settingsURL) { object in - applyWebAiSettingsPatch(request, to: &object) + var proposed = object + applyWebAiSettingsPatch(request, to: &proposed) + try requireSafeWebAiEndpointChange( + from: object, to: proposed, settingsURL: settingsURL, + environment: environment, keychainMayContain: keychainMayContain + ) + object = proposed } + } catch let error as EngramServiceError { + throw error } catch { throw webAiSettingsUnavailable } return try webAiSettings(settingsURL: settingsURL) } + // Check dormant credentials too: disabling a provider or using an env URL + // must not permit staging a new destination for its saved key. These are + // full base URLs, including paths, rather than just host names. + private static func requireSafeWebAiEndpointChange( + from current: [String: Any], to proposed: [String: Any], settingsURL: URL, + environment: [String: String], keychainMayContain: (String) -> Bool + ) throws { + func summaryURL(_ object: [String: Any]) -> String { + text(object["aiBaseURL"]) ?? "https://api.openai.com" + } + func embeddingURL(_ object: [String: Any]) -> String { + text(object["embeddingBaseURL"]) ?? text(object["aiBaseURL"]) ?? "https://api.openai.com/v1" + } + func titleProvider(_ object: [String: Any]) -> String { + text(object["titleProvider"]) ?? "ollama" + } + func titleURL(_ object: [String: Any]) -> String { + text(object["titleBaseUrl"]) ?? text(object["titleBaseURL"]) + ?? (titleProvider(object) == "ollama" ? "http://localhost:11434" : "https://api.openai.com") + } + var accounts = Set() + if summaryURL(current) != summaryURL(proposed) { accounts.insert("aiApiKey") } + if titleURL(current) != titleURL(proposed) || titleProvider(current) != titleProvider(proposed) { + accounts.insert("titleApiKey") + } + let embeddingChanged = embeddingURL(current) != embeddingURL(proposed) + if embeddingChanged { accounts.formUnion(["aiApiKey", "embeddingApiKey"]) } + guard !accounts.isEmpty else { return } + + // Markers are protected even when the corresponding store is locked or + // temporarily empty. Nothing here migrates or reads Keychain secret data. + let storedKey = accounts.contains { (current[$0] as? String)?.isEmpty == false } + let environmentKey = embeddingChanged && environment["ENGRAM_EMBEDDING_API_KEY"]?.isEmpty == false + let sidecarPath = environment["ENGRAM_RUNTIME_AI_SECRETS_PATH"] + ?? settingsURL.deletingLastPathComponent().appendingPathComponent("run/ai-secrets.json").path + if storedKey || environmentKey + || webAiRuntimeMayContain(accounts, path: sidecarPath) + || accounts.sorted().contains(where: keychainMayContain) { + throw EngramServiceError.commandFailed( + name: "WebAiEndpointRequiresLocalUpdate", + message: "AI endpoint changes with saved or unavailable credentials require a local server settings update.", + retryPolicy: "never", details: nil + ) + } + } + + private static func webAiRuntimeMayContain(_ accounts: Set, path: String) -> Bool { + var info = stat() + if lstat(path, &info) != 0 { return errno != ENOENT } + guard let data = SecureRegularFile.read(atPath: path, maximumBytes: 1024 * 1024), + let object = try? JSONSerialization.jsonObject(with: data) as? [String: Any] else { + return true // Unreadable/unsafe is not evidence of credential absence. + } + return accounts.contains { (object[$0] as? String)?.isEmpty == false } + } + + static func webAiKeychainMayContain( + _ account: String, + lookup: (CFDictionary) -> OSStatus = { SecItemCopyMatching($0, nil) } + ) -> Bool { + let context = LAContext() + context.interactionNotAllowed = true + let query: [String: Any] = [ + kSecClass as String: kSecClassGenericPassword, + kSecAttrService as String: KeychainSecretStore.service, + kSecAttrAccount as String: account, + kSecMatchLimit as String: kSecMatchLimitOne, + kSecReturnAttributes as String: true, + kSecUseAuthenticationContext as String: context, + ] + return lookup(query as CFDictionary) != errSecItemNotFound + } + private static func readWebAiSettingsObject(at url: URL) throws -> [String: Any] { var info = stat() if lstat(url.path, &info) != 0 { diff --git a/macos/EngramService/Core/EngramServiceCommandHandler+WebProject.swift b/macos/EngramService/Core/EngramServiceCommandHandler+WebProject.swift index 4c7bee17e..5bd8471a9 100644 --- a/macos/EngramService/Core/EngramServiceCommandHandler+WebProject.swift +++ b/macos/EngramService/Core/EngramServiceCommandHandler+WebProject.swift @@ -28,10 +28,11 @@ extension EngramServiceCommandHandler { dryRun: request.dryRun, force: request.force, auditNote: request.auditNote, - actor: "mcp", + actor: request.actor ?? "web-editor", operationId: try EngramServiceWebProjectValidation.namespacedOperationId(request.operationId) ), - writerGate: writerGate + writerGate: writerGate, + auditActor: try webProjectAuditActor(request.actor) ) } } @@ -48,10 +49,11 @@ extension EngramServiceCommandHandler { dryRun: request.dryRun, force: request.force, auditNote: request.auditNote, - actor: "mcp", + actor: request.actor ?? "web-editor", operationId: try EngramServiceWebProjectValidation.namespacedOperationId(request.operationId) ), - writerGate: writerGate + writerGate: writerGate, + auditActor: try webProjectAuditActor(request.actor) ) } } @@ -65,10 +67,11 @@ extension EngramServiceCommandHandler { EngramServiceProjectUndoRequest( migrationId: request.migrationId, force: request.force, - actor: "mcp", + actor: request.actor ?? "web-editor", operationId: try EngramServiceWebProjectValidation.namespacedOperationId(request.operationId) ), - writerGate: writerGate + writerGate: writerGate, + auditActor: try webProjectAuditActor(request.actor) ) } } @@ -96,10 +99,11 @@ extension EngramServiceCommandHandler { yaml: request.yaml, dryRun: request.dryRun, force: request.force, - actor: "mcp", + actor: request.actor ?? "web-editor", operationId: try EngramServiceWebProjectValidation.namespacedOperationId(request.operationId) ), - writerGate: writerGate + writerGate: writerGate, + auditActor: try webProjectAuditActor(request.actor) ) let wrapped = try EngramServiceWebProjectMoveBatchResponse( operationId: request.operationId, @@ -111,6 +115,13 @@ extension EngramServiceCommandHandler { } } + private static func webProjectAuditActor(_ value: String?) throws -> MigrationLogActor { + guard let actor = MigrationLogActor(rawValue: value ?? "web-editor") else { + throw EngramServiceError.invalidRequest(message: "Invalid Web audit actor.") + } + return actor + } + static func webCancelProjectMoveBatch( _ request: EngramServiceWebCancelProjectMoveBatchRequest ) throws -> EngramServiceWebCancelProjectMoveBatchResponse { diff --git a/macos/EngramService/Core/EngramServiceCommandHandler.swift b/macos/EngramService/Core/EngramServiceCommandHandler.swift index 146cbe946..78a90fba4 100644 --- a/macos/EngramService/Core/EngramServiceCommandHandler.swift +++ b/macos/EngramService/Core/EngramServiceCommandHandler.swift @@ -535,7 +535,7 @@ final class EngramServiceCommandHandler: @unchecked Sendable { try requireAllowedPayloadKeys( request, required: ["src", "dst", "dry_run", "operation_id"], - optional: ["force", "audit_note"] + optional: ["force", "audit_note", "actor"] ) let payload = try decodePayload(EngramServiceWebProjectMoveRequest.self, from: request) let result = try await Self.webProjectMove(payload, writerGate: writerGate) @@ -548,7 +548,7 @@ final class EngramServiceCommandHandler: @unchecked Sendable { try requireAllowedPayloadKeys( request, required: ["src", "dry_run", "operation_id"], - optional: ["archive_to", "force", "audit_note"] + optional: ["archive_to", "force", "audit_note", "actor"] ) let payload = try decodePayload(EngramServiceWebProjectArchiveRequest.self, from: request) let result = try await Self.webProjectArchive(payload, writerGate: writerGate) @@ -561,7 +561,7 @@ final class EngramServiceCommandHandler: @unchecked Sendable { try requireAllowedPayloadKeys( request, required: ["migration_id", "operation_id"], - optional: ["force"] + optional: ["force", "actor"] ) let payload = try decodePayload(EngramServiceWebProjectUndoRequest.self, from: request) let result = try await Self.webProjectUndo(payload, writerGate: writerGate) @@ -574,7 +574,7 @@ final class EngramServiceCommandHandler: @unchecked Sendable { try requireAllowedPayloadKeys( request, required: ["yaml", "dry_run", "operation_id"], - optional: ["force"] + optional: ["force", "actor"] ) let payload = try decodePayload(EngramServiceWebProjectMoveBatchRequest.self, from: request) let result = try await Self.webProjectMoveBatch(payload, writerGate: writerGate) @@ -630,7 +630,7 @@ final class EngramServiceCommandHandler: @unchecked Sendable { databaseGeneration: result.databaseGeneration ) case "webAddProjectAlias": - try requireExactPayloadKeys(request, allowed: [["canonical", "alias"]]) + try requireAllowedPayloadKeys(request, required: ["canonical", "alias"], optional: ["actor"]) let payload = try decodePayload(EngramServiceWebAddAliasRequest.self, from: request) let result = try await writerGate.performWriteCommand(name: request.command) { writer in try self.addWebProjectAlias(payload, writer: writer) @@ -641,7 +641,7 @@ final class EngramServiceCommandHandler: @unchecked Sendable { databaseGeneration: result.databaseGeneration ) case "webRemoveProjectAlias": - try requireExactPayloadKeys(request, allowed: [["alias", "canonical"]]) + try requireAllowedPayloadKeys(request, required: ["alias", "canonical"], optional: ["actor"]) let payload = try decodePayload(EngramServiceWebRemoveAliasRequest.self, from: request) let result = try await writerGate.performWriteCommand(name: request.command) { writer in try self.removeWebProjectAlias(payload, writer: writer) diff --git a/macos/EngramService/Core/EngramServiceReadProvider.swift b/macos/EngramService/Core/EngramServiceReadProvider.swift index 19ea0ae79..02952922a 100644 --- a/macos/EngramService/Core/EngramServiceReadProvider.swift +++ b/macos/EngramService/Core/EngramServiceReadProvider.swift @@ -1231,7 +1231,7 @@ struct SQLiteEngramServiceReadProvider: EngramServiceReadProvider { } args.append(termMatches[index]) } else { - ctes.append(try shortTokenHitsCTE(alias: alias, db: db)) + ctes.append(try shortTokenHitsCTE(alias: alias, db: db, valueBand: request.valueBand)) args.append("%\(CJKText.escapeLikePattern(token))%") args.append(Self.shortQueryHitCap(limit: limit)) } @@ -1246,9 +1246,13 @@ struct SQLiteEngramServiceReadProvider: EngramServiceReadProvider { candidateJoins.append("JOIN \(alias) ON \(alias).session_id = m0.session_id") } } + // Every term contributes its best row rank. Canonical addition + // order keeps query permutations stable even for floating-point ties. + let rankExpression = tokens.indices.sorted { tokens[$0] < tokens[$1] } + .map { "m\($0).rank" }.joined(separator: " + ") var topParts = [""" top AS MATERIALIZED ( - SELECT s.*, m0.rank AS search_rank + SELECT s.*, \(rankExpression) AS search_rank FROM m0 \(candidateJoins.joined(separator: " ")) CROSS JOIN sessions s ON s.id = m0.session_id @@ -1258,7 +1262,7 @@ struct SQLiteEngramServiceReadProvider: EngramServiceReadProvider { """] appendSearchFilters(for: request, scope: scope, to: &topParts, args: &args) topParts.append(""" - ORDER BY m0.rank, s.start_time DESC + ORDER BY search_rank, s.start_time DESC, s.id LIMIT ? ) """) @@ -1334,7 +1338,7 @@ struct SQLiteEngramServiceReadProvider: EngramServiceReadProvider { SELECT top.*, \(snippetExpression) AS snippet FROM top \(bestJoins.joined(separator: " ")) - ORDER BY top.search_rank, top.start_time DESC + ORDER BY top.search_rank, top.start_time DESC, top.id """ let rows = try Row.fetchAll(db, sql: sql, arguments: StatementArguments(args)) return EngramServiceSearchResponse( @@ -2784,8 +2788,9 @@ struct SQLiteEngramServiceReadProvider: EngramServiceReadProvider { max(256, min(limit * 32, 2_048)) } - private func shortTokenHitsCTE(alias: String, db: GRDB.Database) throws -> String { + private func shortTokenHitsCTE(alias: String, db: GRDB.Database, valueBand: String?) throws -> String { let mapped = try tableExists("fts_map", db: db) && indexExists("idx_sessions_activity_time", db: db) + let valueFilter = SearchFilterPredicates.clauses(valueBand: valueBand).first?.sql if mapped { return """ \(alias)_hits AS MATERIALIZED ( @@ -2796,14 +2801,17 @@ struct SQLiteEngramServiceReadProvider: EngramServiceReadProvider { WHERE s.hidden_at IS NULL AND (s.tier IS NULL OR s.tier != 'skip') AND f.content LIKE ? ESCAPE '\\' + AND \(valueFilter ?? "1 = 1") LIMIT ?) """ } return """ \(alias)_hits AS MATERIALIZED ( - SELECT rowid AS fts_rowid, session_id, 0.0 AS rank + SELECT sessions_fts.rowid AS fts_rowid, session_id, 0.0 AS rank FROM sessions_fts + \(valueFilter == nil ? "" : "JOIN sessions s ON s.id = sessions_fts.session_id") WHERE content LIKE ? ESCAPE '\\' + AND \(valueFilter ?? "1 = 1") LIMIT ?) """ } @@ -3054,7 +3062,8 @@ struct SQLiteEngramServiceReadProvider: EngramServiceReadProvider { sources: request.source.map { [$0] } ?? [], projects: request.project.map { [$0] } ?? [], origin: request.origin, - since: request.since + since: request.since, + valueBand: request.valueBand ) for clause in clauses { parts.append("AND \(clause.sql)") diff --git a/macos/EngramService/Core/ServiceWebMetadataProducer.swift b/macos/EngramService/Core/ServiceWebMetadataProducer.swift index 99d84431f..c9ca366f1 100644 --- a/macos/EngramService/Core/ServiceWebMetadataProducer.swift +++ b/macos/EngramService/Core/ServiceWebMetadataProducer.swift @@ -4080,8 +4080,13 @@ final class ServiceWebMetadataProducer: ServiceWebMetadataProviding, @unchecked sql: "SELECT COUNT(*) FROM project_aliases WHERE alias = ? AND canonical = ?", arguments: [request.alias, rawCanonical] ) ?? 0 + let changed = after > before ? 1 : 0 + if changed > 0 { + try Self.recordAliasAudit(db, action: "add", actor: request.actor, + alias: publishedAlias, canonical: request.canonical) + } return try EngramServiceWebAliasMutationResponse( - action: "add", alias: publishedAlias, canonical: request.canonical, changed: after > before ? 1 : 0 + action: "add", alias: publishedAlias, canonical: request.canonical, changed: changed ) } } @@ -4111,12 +4116,32 @@ final class ServiceWebMetadataProducer: ServiceWebMetadataProviding, @unchecked arguments: [pair.rawAlias, pair.rawCanonical] ) let changed = db.changesCount > 0 ? 1 : 0 + if changed > 0 { + try Self.recordAliasAudit(db, action: "remove", actor: request.actor, + alias: request.alias, canonical: request.canonical) + } return try EngramServiceWebAliasMutationResponse( action: "remove", alias: request.alias, canonical: request.canonical, changed: changed ) } } + /// Called inside the alias mutation transaction; failure rolls back both. + /// Published keys avoid retaining removed raw paths in diagnostic history. + private static func recordAliasAudit( + _ db: Database, action: String, actor: String?, alias: String, canonical: String + ) throws { + try db.execute( + sql: "INSERT INTO web_alias_audit_log (action, actor, alias, canonical) VALUES (?, ?, ?, ?)", + arguments: [action, actor ?? "web-editor", alias, canonical] + ) + // Bound local history by insertion order, independent of clock changes. + try db.execute(sql: """ + DELETE FROM web_alias_audit_log + WHERE id < (SELECT id FROM web_alias_audit_log ORDER BY id DESC LIMIT 1 OFFSET 9999) + """) + } + /// Pins a cancelled metadata-read control so tests can prove alias writes /// do not share that relay. func withPinnedCancelledReadRelay(_ body: () throws -> T) throws -> T { diff --git a/macos/EngramServiceCoreTests/CollectorPublicationWorkerTests.swift b/macos/EngramServiceCoreTests/CollectorPublicationWorkerTests.swift index c3d17e755..227c5f468 100644 --- a/macos/EngramServiceCoreTests/CollectorPublicationWorkerTests.swift +++ b/macos/EngramServiceCoreTests/CollectorPublicationWorkerTests.swift @@ -3388,6 +3388,7 @@ final class CollectorPublicationWorkerTests: XCTestCase { func testFreshOnlyReservationDoesNotReuseAnExistingCapture() throws { let f = try PublicationFixture() + defer { f.remove() } let vscode = try f.installNativeVSCodePrimary(rootID: "daily-vscode") let observed = try EngramCollectorCore.CollectorVSCodeSource.observe( rootPath: vscode.configuration.rootPath, primaryRelative: vscode.relativePath, diff --git a/macos/EngramServiceCoreTests/CollectorRuntimeTests.swift b/macos/EngramServiceCoreTests/CollectorRuntimeTests.swift index 1a7e6bd1b..9b7f0efcd 100644 --- a/macos/EngramServiceCoreTests/CollectorRuntimeTests.swift +++ b/macos/EngramServiceCoreTests/CollectorRuntimeTests.swift @@ -539,21 +539,27 @@ final class CollectorRuntimeTests: XCTestCase { WHERE root_id = 'runtime-codex' AND last_scan_failure = 'enumerationUnavailable' """ var active: Runtime? + let retryClock = RuntimeLocked(ContinuousClock.now) var now: Int64 = 100 do { try f.writeSettings(f.document(replicas: replicas)) - active = try XCTUnwrap(Runtime.open(settingsURL: f.settings, secretLoader: f.secret, diagnostics: report)) + active = try XCTUnwrap(Runtime.open(settingsURL: f.settings, secretLoader: f.secret, diagnostics: report, + retryClock: { retryClock.value })) // (a) and flapping: each turn takes at most one bootstrap step per root, so // threshold - 1 turns hold fewer than threshold blocked steps. Block, recover // (the scan finishes and clears last_scan_failure), block again: no line. for _ in 0..<3 { XCTAssertEqual(chmod(locked.path, 0), 0) - for _ in 0..<(threshold - 1) { _ = try await active!.runOnce(now: now); now += 1 } + for _ in 0..<(threshold - 1) { + _ = try await active!.runOnce(now: now); now += 1 + retryClock.update { $0 = $0.advanced(by: .seconds(60)) } + } XCTAssertEqual(try f.integer(failureSQL), 1, "the walk must actually block in this cycle") XCTAssertEqual(chmod(locked.path, 0o700), 0) var recoveryTurns = 0 while try f.integer(failureSQL) == 1, recoveryTurns < 20 { _ = try await active!.runOnce(now: now); now += 1; recoveryTurns += 1 + retryClock.update { $0 = $0.advanced(by: .seconds(60)) } } XCTAssertEqual(try f.integer(failureSQL), 0, "the walk must recover in this cycle") } @@ -563,17 +569,25 @@ final class CollectorRuntimeTests: XCTestCase { var turns = 0 while lines.value.isEmpty, turns < 30 { _ = try await active!.runOnce(now: now); now += 1; turns += 1 + retryClock.update { $0 = $0.advanced(by: .seconds(60)) } } XCTAssertGreaterThanOrEqual(turns, threshold) - for _ in 0..<3 { _ = try await active!.runOnce(now: now); now += 1 } + for _ in 0..<3 { + _ = try await active!.runOnce(now: now); now += 1 + retryClock.update { $0 = $0.advanced(by: .seconds(60)) } + } XCTAssertEqual(lines.value, [blocked]) // (c) the first non-blocked step clears it: exactly one 0/1 line. XCTAssertEqual(chmod(locked.path, 0o700), 0) var clearTurns = 0 while lines.value.count == 1, clearTurns < 20 { _ = try await active!.runOnce(now: now); now += 1; clearTurns += 1 + retryClock.update { $0 = $0.advanced(by: .seconds(60)) } + } + for _ in 0..<3 { + _ = try await active!.runOnce(now: now); now += 1 + retryClock.update { $0 = $0.advanced(by: .seconds(60)) } } - for _ in 0..<3 { _ = try await active!.runOnce(now: now); now += 1 } XCTAssertEqual(lines.value, [blocked, cleared]) try await active!.stop(); active = nil await replicas.stop() @@ -584,6 +598,160 @@ final class CollectorRuntimeTests: XCTestCase { } } + func testBlockedBootstrapBacksOffWithoutDelayingHealthyRootAndRecovers_repro() async throws { + try XCTSkipIf(geteuid() == 0, "root bypasses the fixture's DAC permission denial") + let f = try RuntimeFixture(); defer { f.remove() } + let replicas = try await RuntimeReplicas.start(parent: f.base) + let blockedRoot = f.base.appendingPathComponent("blocked-source") + let locked = blockedRoot.appendingPathComponent("locked") + try FileManager.default.createDirectory(at: locked, withIntermediateDirectories: true) + defer { _ = chmod(locked.path, 0o700) } + XCTAssertEqual(chmod(locked.path, 0), 0) + let clock = RuntimeLocked(ContinuousClock.now) + let starts = RuntimeLocked<[String: Int]>([:]) + let streams = RuntimeLocked<[String: RuntimeSignalStream]>([:]) + let factory: Runtime.EventStreamFactory = { request, _ in + starts.update { $0[request.binding.configuration.rootID, default: 0] += 1 } + let stream = RuntimeSignalStream() + streams.update { $0[request.binding.configuration.rootID] = stream } + return stream + } + var document = f.document(replicas: replicas) + var collector = try XCTUnwrap(document["collector"] as? [String: Any]) + collector["roots"] = [ + ["rootID": "blocked-codex", "source": "codex", "rootPath": blockedRoot.path, "revision": 1], + ["rootID": "runtime-codex", "source": "codex", "rootPath": f.sources.path, "revision": 1], + ] + document["collector"] = collector + let failureSQL = "SELECT count(*) FROM collector_roots WHERE root_id = 'blocked-codex' AND last_scan_failure = 'enumerationUnavailable'" + let revisionSQL = "SELECT requested_revision FROM collector_roots WHERE root_id = 'blocked-codex'" + var active: Runtime? + do { + try f.writeSettings(document) + try f.writeTranscript("healthy root still publishes during bootstrap backoff") + active = try XCTUnwrap(Runtime.open(settingsURL: f.settings, secretLoader: f.secret, + diagnostics: { _ in }, eventStreamFactory: factory, retryClock: { clock.value })) + for _ in 0..<10 { + _ = try await active!.runOnce(now: 100) + if try f.integer(failureSQL) == 1, + try f.integer("SELECT count(*) FROM collector_publication_replicas WHERE state = 'acknowledged'") == 2 { break } + } + XCTAssertEqual(try f.integer(failureSQL), 1) + XCTAssertEqual(try f.integer("SELECT count(*) FROM collector_publication_replicas WHERE state = 'acknowledged'"), 2) + XCTAssertEqual(starts.value["runtime-codex"], 1) + + // Partial progress at the start of each re-walk must not reset the delay. + for delay in [1, 2, 4, 8, 16, 32, 60, 60] { + let revision = try f.integer(revisionSQL) + let startCount = starts.value["blocked-codex", default: 0] + for wallTime: Int64 in [1, 9_000_000_000, 2] { + let cycle = try await active!.runOnce(now: wallTime) + XCTAssertGreaterThanOrEqual(cycle.deferred, 1) + } + clock.update { $0 = $0.advanced(by: .milliseconds(delay * 1_000 - 1)) } + _ = try await active!.runOnce(now: 100) + XCTAssertEqual(try f.integer(revisionSQL), revision, "no reconciliation churn before retry deadline") + XCTAssertEqual(starts.value["blocked-codex"], startCount) + clock.update { $0 = $0.advanced(by: .milliseconds(1)) } + for _ in 0..<10 { + _ = try await active!.runOnce(now: 100) + if starts.value["blocked-codex", default: 0] > startCount, + try f.integer(failureSQL) == 1 { break } + } + XCTAssertEqual(starts.value["blocked-codex"], startCount + 1) + XCTAssertEqual(try f.integer(failureSQL), 1) + } + XCTAssertEqual(starts.value["runtime-codex"], 1, "healthy listener never restarts with the blocked root") + XCTAssertEqual(chmod(locked.path, 0o700), 0) + clock.update { $0 = $0.advanced(by: .seconds(60)) } + let convergedSQL = "SELECT count(*) FROM collector_roots WHERE root_id = 'blocked-codex' AND completed_revision = requested_revision AND last_scan_failure IS NULL" + for _ in 0..<10 { + _ = try await active!.runOnce(now: 100) + if try f.integer(convergedSQL) == 1 { break } + } + XCTAssertEqual(try f.integer(convergedSQL), 1, "permission recovery completes automatically") + + // A later failure starts at one second again, even after reaching the cap. + XCTAssertEqual(chmod(locked.path, 0), 0) + _ = streams.value["blocked-codex"]?.emit(.loss(.continuityLoss)) + for _ in 0..<10 { + _ = try await active!.runOnce(now: 100) + if try f.integer(failureSQL) == 1 { break } + } + let afterRecovery = starts.value["blocked-codex", default: 0] + clock.update { $0 = $0.advanced(by: .seconds(1)) } + for _ in 0..<10 { + _ = try await active!.runOnce(now: 100) + if starts.value["blocked-codex", default: 0] > afterRecovery, + try f.integer(failureSQL) == 1 { break } + } + XCTAssertEqual(starts.value["blocked-codex"], afterRecovery + 1) + + // Backoff is process-local; reopening does not inherit a stale deadline. + try await active!.stop(); active = nil + let beforeRestart = starts.value["blocked-codex", default: 0] + active = try XCTUnwrap(Runtime.open(settingsURL: f.settings, secretLoader: f.secret, + diagnostics: { _ in }, eventStreamFactory: factory, retryClock: { clock.value })) + _ = try await active!.runOnce(now: 100) + XCTAssertEqual(starts.value["blocked-codex"], beforeRestart + 1) + try await active!.stop(); active = nil + await replicas.stop() + } catch { + try? await active?.stop() + await replicas.stop() + throw error + } + } + + func testBlockedBootstrapResidentWaitDoesNotSpinAndRetriesBeforeLongPoll_repro() async throws { + try XCTSkipIf(geteuid() == 0, "root bypasses the fixture's DAC permission denial") + let f = try RuntimeFixture(); defer { f.remove() } + let replicas = try await RuntimeReplicas.start(parent: f.base) + let locked = f.sources.appendingPathComponent("locked") + try FileManager.default.createDirectory(at: locked, withIntermediateDirectories: false) + defer { _ = chmod(locked.path, 0o700) } + XCTAssertEqual(chmod(locked.path, 0), 0) + let clock = RuntimeLocked(ContinuousClock.now) + let turns = RuntimeLocked(0) + let starts = RuntimeLocked(0) + var active: Runtime? + do { + try f.writeSettings(f.document(replicas: replicas, pollIntervalMilliseconds: 10_000)) + active = try XCTUnwrap(Runtime.open(settingsURL: f.settings, secretLoader: f.secret, + diagnostics: { _ in }, eventStreamFactory: { _, _ in + starts.update { $0 += 1 } + return RuntimeSignalStream() + }, retryClock: { clock.value })) + await active!.setTestCaptureTurnBarrier { turns.update { $0 += 1 } } + try await active!.start() + let failureSQL = "SELECT count(*) FROM collector_roots WHERE last_scan_failure = 'enumerationUnavailable'" + let deadline = ContinuousClock.now.advanced(by: .seconds(3)) + while try f.integer(failureSQL) == 0, ContinuousClock.now < deadline { + try await Task.sleep(for: .milliseconds(10)) + } + XCTAssertEqual(try f.integer(failureSQL), 1) + let blockedTurns = turns.value + try await Task.sleep(for: .milliseconds(1_200)) + XCTAssertEqual(turns.value, blockedTurns, "pending recovery must not force repeated capture turns during backoff") + XCTAssertEqual(starts.value, 1) + XCTAssertEqual(chmod(locked.path, 0o700), 0) + clock.update { $0 = $0.advanced(by: .seconds(1)) } + let convergedSQL = "SELECT count(*) FROM collector_roots WHERE completed_revision = requested_revision AND last_scan_failure IS NULL" + let retryDeadline = ContinuousClock.now.advanced(by: .seconds(3)) + while try f.integer(convergedSQL) == 0, ContinuousClock.now < retryDeadline { + try await Task.sleep(for: .milliseconds(20)) + } + XCTAssertEqual(try f.integer(convergedSQL), 1, "retry uses cheap wakes, not the ten-second periodic deadline") + XCTAssertEqual(starts.value, 2) + try await active!.stop(); active = nil + await replicas.stop() + } catch { + try? await active?.stop() + await replicas.stop() + throw error + } + } + // Repro for the 2026-10-02 Daily restart loop (exit 70 about every 12 s): a stream // loss admitted while the coordinator was still starting left it recoveryRequired, and // the runtime turned that into reconciliationRequired. A loss must only force the @@ -4304,6 +4472,21 @@ final class CollectorRuntimeTests: XCTestCase { } } +private final class RuntimeSignalStream: CollectorEventStream { + private let callback = RuntimeLocked<((CollectorEventStreamSignal) -> CollectorEventAdmission)?>(nil) + + func start(deliver: @escaping (CollectorEventStreamSignal) -> CollectorEventAdmission) throws { + callback.update { $0 = deliver } + _ = deliver(.historyDone) + } + + func emit(_ signal: CollectorEventStreamSignal) -> CollectorEventAdmission { + callback.value?(signal) ?? .rejectedClosed + } + + func stop() throws { callback.update { $0 = nil } } +} + private struct RuntimeStreamStart { let rootID: String let epoch: String @@ -4457,8 +4640,14 @@ final class RuntimeFixture: @unchecked Sendable { } func writeSettings(_ document: [String: Any]) throws { - try JSONSerialization.data(withJSONObject: document, options: [.sortedKeys]).write(to: settings) - guard chmod(settings.path, 0o600) == 0 else { throw Failure.unsafeFixture } + let staged = settings.deletingLastPathComponent() + .appendingPathComponent(".settings-\(UUID().uuidString).json") + defer { try? FileManager.default.removeItem(at: staged) } + try JSONSerialization.data(withJSONObject: document, options: [.sortedKeys]) + .write(to: staged, options: .withoutOverwriting) + // Readers must see a complete owner-only document, including during reload. + guard chmod(staged.path, 0o600) == 0, + rename(staged.path, settings.path) == 0 else { throw Failure.unsafeFixture } } func writeTranscript(_ text: String, name: String = "rollout-one.jsonl", diff --git a/macos/EngramServiceCoreTests/CollectorWebDemoTests.swift b/macos/EngramServiceCoreTests/CollectorWebDemoTests.swift index ce09cec53..c994536f4 100644 --- a/macos/EngramServiceCoreTests/CollectorWebDemoTests.swift +++ b/macos/EngramServiceCoreTests/CollectorWebDemoTests.swift @@ -484,6 +484,13 @@ final class CollectorWebDemoTests: XCTestCase { try Int.fetchOne(db, sql: "SELECT COUNT(*) FROM project_aliases WHERE alias = ?", arguments: [rawAlias]) } XCTAssertEqual(remaining, 0) + let aliasAudit = try await database.read { db in + try Row.fetchAll(db, sql: "SELECT actor, action, alias, canonical FROM web_alias_audit_log ORDER BY id") + } + XCTAssertEqual(aliasAudit.map { $0["actor"] as String }, ["web-editor", "web-editor"]) + XCTAssertEqual(aliasAudit.map { $0["action"] as String }, ["add", "remove"]) + XCTAssertEqual(aliasAudit.map { $0["alias"] as String }, [result.alias, result.alias]) + XCTAssertEqual(aliasAudit.map { $0["canonical"] as String }, [canonical, canonical]) let note = "ContinuitySaved " + String(repeating: "🙂中", count: 6_000) let deniedInsight = try await request("/web/api/insights", method: "POST", cookie: viewerCookie, body: ["content": note, "sourceSessionId": "alias-session"]) diff --git a/macos/EngramServiceCoreTests/EngramServiceIPCTests.swift b/macos/EngramServiceCoreTests/EngramServiceIPCTests.swift index 6f9aa9b7f..1b76e1a7e 100644 --- a/macos/EngramServiceCoreTests/EngramServiceIPCTests.swift +++ b/macos/EngramServiceCoreTests/EngramServiceIPCTests.swift @@ -3059,6 +3059,90 @@ final class EngramServiceIPCTests: XCTestCase { ]) } + func testSearchFiltersValueBandThroughRealDispatcherBeforeLimit() async throws { + let paths = try makeServiceIPCPaths() + defer { try? FileManager.default.removeItem(at: paths.runtime.deletingLastPathComponent()) } + try seedSearchFixture(at: paths.database.path) + try await DatabaseQueue(path: paths.database.path).write { db in + try db.execute(sql: "UPDATE sessions SET quality_score = CASE id WHEN 's1' THEN 60 ELSE 35 END") + try db.execute(sql: "UPDATE sessions_fts SET content = 'bandneedle'") + } + let gate = try ServiceWriterGate(databasePath: paths.database.path, runtimeDirectory: paths.runtime) + let handler = EngramServiceCommandHandler( + writerGate: gate, + readProvider: try SQLiteEngramServiceReadProvider(databasePath: paths.database.path) + ) + let server = UnixSocketServiceServer(socketPath: paths.socket.path) { await handler.handle($0) } + try server.start() + defer { server.stop() } + let client = EngramServiceClient(transport: UnixSocketEngramServiceTransport(socketPath: paths.socket.path)) + for (band, expected) in [("high", ["s1"]), ("low", ["s2"]), + ("medium", []), ("unknown", []), ("invalid", [])] { + let data = try JSONSerialization.data(withJSONObject: [ + "query": "bandneedle", "mode": "keyword", "limit": 1, "valueBand": band, + ]) + let request = try JSONDecoder().decode(EngramServiceSearchRequest.self, from: data) + let response = try await client.search(request) + XCTAssertEqual(response.items.map(\.id), expected, "band=\(band)") + } + let legacy = try await client.search(.init(query: "bandneedle", mode: "keyword", limit: 1)) + XCTAssertEqual(legacy.items.map(\.id), ["s2"], "omission preserves existing ranking") + } + + func testShortSearchValueFilterPrecedesCandidateCapInBothLayouts() async throws { + let fixture = try MetadataSQLFixture() + defer { fixture.remove() } + try fixture.migrate() + try fixture.seedRegistry() + for index in 0..<270 { + try fixture.seedBoundSession(id: "low-\(index)", start: "2026-09-03 12:00:00", + nativeID: "low-\(index)", title: "AI filler", indexReady: true) + } + try fixture.seedBoundSession(id: "high-old", start: "2020-01-01 12:00:00", + nativeID: "high-old", title: "AI target", indexReady: true) + try fixture.write { db in + try db.execute(sql: "UPDATE sessions SET quality_score = CASE id WHEN 'high-old' THEN 60 ELSE 20 END") + try db.execute(sql: "DELETE FROM fts_map") + try db.execute(sql: """ + INSERT INTO fts_map(session_id, msg_seq, fts_rowid, content_hash) + SELECT session_id, 0, rowid, '' FROM sessions_fts + """) + } + let provider = try SQLiteEngramServiceReadProvider(databasePath: fixture.path) + for mapped in [true, false] { + if !mapped { + try fixture.write { db in try db.execute(sql: "DROP TABLE fts_map") } + } + let response = try await provider.search(.init( + query: "AI", mode: "keyword", limit: 1, valueBand: "high" + )) + XCTAssertEqual(response.items.map(\.id), ["high-old"], "mapped=\(mapped)") + } + } + + func testSemanticAndHybridValueFilterPrecedesCandidateRanking() async throws { + let paths = try makeServiceIPCPaths() + defer { try? FileManager.default.removeItem(at: paths.runtime.deletingLastPathComponent()) } + try seedOriginFilteredSemanticSearchFixture(at: paths.database.path, targetOrigin: "hq", + interferenceOrigin: "local", interferenceCount: 40) + try await DatabaseQueue(path: paths.database.path).write { db in + try db.execute(sql: "UPDATE sessions SET quality_score = CASE id WHEN 's2' THEN 60 ELSE 20 END") + } + let provider = try SQLiteEngramServiceReadProvider( + databasePath: paths.database.path, + embeddingEnvironment: ["ENGRAM_EMBEDDING_API_KEY": "test", "ENGRAM_EMBEDDING_MODEL": "probe", + "ENGRAM_EMBEDDING_DIM": "3"], + embeddingProviderFactory: { _ in StaticEmbeddingProvider { _ in [1, 0, 0] } } + ) + for mode in ["semantic", "hybrid"] { + let response = try await provider.search(.init( + query: "originneedle", mode: mode, limit: 1, valueBand: "high" + )) + XCTAssertEqual(response.items.map(\.id), ["s2"], mode) + XCTAssertNil(response.warning) + } + } + func testSQLiteReadProviderSourcesExposeArchiveHealthFacts() async throws { let paths = try makeServiceIPCPaths() try seedSearchFixture(at: paths.database.path) @@ -4238,6 +4322,10 @@ final class EngramServiceIPCTests: XCTestCase { EngramServiceSearchRequest(query: "需要修复 测试", mode: "keyword", limit: 10) ) XCTAssertEqual(mixed.items.map(\.id), ["s1", "s2"]) + let shortFirst = try await provider.search( + EngramServiceSearchRequest(query: "测试 需要修复", mode: "keyword", limit: 1) + ) + XCTAssertEqual(shortFirst.items.map(\.id), ["s1"], "short terms must not erase later MATCH ranks") } func testKeywordSearchSubTrigramHitsAreRecencyBounded_repro() async throws { @@ -4310,7 +4398,7 @@ final class EngramServiceIPCTests: XCTestCase { ('lite', 'usage monitor go'); """) } - let queries = ["monitor", "usage monitor", "monitor go", "go", "parser fix"] + let queries = ["monitor", "usage monitor", "monitor usage", "monitor go", "go monitor", "go", "parser fix"] func run() async throws -> [String: [(String, String?)]] { let provider = try SQLiteEngramServiceReadProvider(databasePath: paths.database.path) @@ -4344,7 +4432,9 @@ final class EngramServiceIPCTests: XCTestCase { // Ranking, filters and conjunction semantics on the indexed layout. XCTAssertEqual(indexed["monitor"]?.map(\.0), ["s3", "s1", "s2"]) XCTAssertEqual(indexed["usage monitor"]?.map(\.0), ["s3", "s1"]) + XCTAssertEqual(indexed["monitor usage"]?.map(\.0), ["s3", "s1"]) XCTAssertEqual(indexed["monitor go"]?.map(\.0), ["s3", "s2"]) + XCTAssertEqual(indexed["go monitor"]?.map(\.0), ["s3", "s2"]) XCTAssertEqual(Set(indexed["go"]?.map(\.0) ?? []), ["s2", "s3"]) XCTAssertEqual(indexed["parser fix"]?.map(\.0), ["s1"]) let parserSnippet = try XCTUnwrap(indexed["parser fix"]?.first?.1) diff --git a/macos/EngramServiceCoreTests/WebAiSettingsWriteTests.swift b/macos/EngramServiceCoreTests/WebAiSettingsWriteTests.swift index f54f25c46..60da455bb 100644 --- a/macos/EngramServiceCoreTests/WebAiSettingsWriteTests.swift +++ b/macos/EngramServiceCoreTests/WebAiSettingsWriteTests.swift @@ -1,5 +1,8 @@ import Darwin import Foundation +import EngramCoreWrite +import LocalAuthentication +import Security import XCTest @testable import EngramServiceCore @@ -213,6 +216,189 @@ final class WebAiSettingsWriteTests: XCTestCase { XCTAssertEqual(try currentObject(url: target)["aiModel"] as? String, "keep-me") } + func testEndpointChangeCannotRedirectStoredCredential_repro() throws { + try writeSettings([ + "aiBaseURL": "https://trusted.example/v1", + "aiApiKey": "fixture-only-key", + "aiModel": "keep-me", + ]) + let original = try Data(contentsOf: settingsURL) + XCTAssertThrowsError(try EngramServiceCommandHandler.webPatchAiSettings( + EngramServiceWebPatchAiSettingsRequest( + aiBaseURL: "https://other.example/v1", aiModel: "must-not-save" + ), settingsURL: settingsURL + )) { error in + guard case .commandFailed(let name, _, let retry, _) = error as? EngramServiceError else { + return XCTFail("Expected a structured endpoint rejection") + } + XCTAssertEqual(name, "WebAiEndpointRequiresLocalUpdate") + XCTAssertEqual(retry, "never") + XCTAssertFalse(String(describing: error).contains("fixture-only-key")) + } + XCTAssertEqual(try Data(contentsOf: settingsURL), original, "Reject the entire patch atomically") + let config = EngramServiceCommandHandler.ServiceAISettings.read( + settingsPath: settingsURL, environment: [:], keychainReader: { _ in nil } + ) + XCTAssertEqual(config.summaryConfig?.baseURL, "https://trusted.example/v1") + XCTAssertEqual(config.summaryConfig?.apiKey, "fixture-only-key") + } + + func testProtectsSharedFallbackLegacyDefaultsAndDormantCredentials() throws { + let cases: [([String: Any], EngramServiceWebPatchAiSettingsRequest)] = try [ + (["aiProtocol": "disabled", "aiApiKey": "@keychain"], + .init(aiBaseURL: "https://other.example/v1")), + (["embeddingApiKey": "fixture-embedding"], + .init(aiBaseURL: "https://other.example/v1")), + (["aiApiKey": "fixture-shared"], + .init(embeddingBaseURL: "https://other.example/v1")), + (["aiApiKey": "fixture-shared", "embeddingBaseURL": "https://trusted.example/v1"], + .init(embeddingBaseURL: "")), + (["titleApiKey": "@keychain", "titleBaseURL": "https://legacy.example/v1"], + .init(titleBaseUrl: "https://other.example/v1")), + (["titleApiKey": "fixture-title", "titleProvider": "ollama"], + .init(titleBaseUrl: "https://other.example/v1")), + (["titleApiKey": "fixture-title", "titleProvider": "ollama", + "titleBaseUrl": "https://same.example/v1"], .init(titleProvider: "custom")), + (["titleApiKey": "fixture-title", "titleProvider": "custom"], .init(titleProvider: "ollama")), + (["aiApiKey": "@keychain", "aiBaseURL": "https://trusted.example/old"], + .init(aiBaseURL: "https://trusted.example/new")), + ] + for (object, patch) in cases { + try writeSettings(object) + let original = try Data(contentsOf: settingsURL) + XCTAssertThrowsError(try isolatedPatch(patch)) + XCTAssertEqual(try Data(contentsOf: settingsURL), original) + } + // A successful disable is not permission to retarget the dormant key. + try writeSettings(["aiApiKey": "fixture-key"]) + _ = try isolatedPatch(.init(aiProtocol: "disabled")) + XCTAssertThrowsError(try isolatedPatch(.init(aiBaseURL: "https://other.example"))) + XCTAssertEqual(try currentObject()["aiProtocol"] as? String, "disabled") + } + + func testAllowsCredentialFreeAndUnrelatedEditsWithoutSecretReads() throws { + try writeSettings([:]) + let changed = try isolatedPatch(.init( + aiBaseURL: "https://new.example/v1", titleProvider: "custom", + titleBaseUrl: "https://title.example/v1", embeddingBaseURL: "https://embed.example/v1" + )) + XCTAssertEqual(changed.settings.aiBaseURL, "https://new.example/v1") + try writeSettings([ + "aiApiKey": "@keychain", "titleApiKey": "@keychain", "embeddingApiKey": "@keychain", + "titleBaseURL": "http://localhost:11434", + ]) + let edited = try EngramServiceCommandHandler.webPatchAiSettings( + .init(aiBaseURL: "https://api.openai.com", aiModel: "new-model", + titleBaseUrl: "http://localhost:11434", embeddingBaseURL: "https://api.openai.com/v1"), + settingsURL: settingsURL, environment: [:], keychainMayContain: { _ in + XCTFail("Unchanged effective endpoints must not probe credentials") + return true + } + ) + XCTAssertEqual(edited.settings.aiModel, "new-model") + XCTAssertNil(try currentObject()["titleBaseURL"]) + // An independent embedding key must not lock a credential-free summary endpoint. + try writeSettings(["embeddingApiKey": "@keychain", "embeddingBaseURL": "https://embed.example/v1"]) + XCTAssertNoThrow(try isolatedPatch(.init(aiBaseURL: "https://summary.example/v1"))) + } + + func testProtectsEnvironmentSidecarAndMarkerlessKeychainCredentials() throws { + let patch = try EngramServiceWebPatchAiSettingsRequest(embeddingBaseURL: "https://other.example/v1") + try writeSettings([:]) + XCTAssertThrowsError(try isolatedPatch(patch, environment: [ + "ENGRAM_EMBEDDING_API_KEY": "fixture-env", + "ENGRAM_EMBEDDING_BASE_URL": "https://env.example/v1", + ])) + for account in ["aiApiKey", "embeddingApiKey", "titleApiKey"] { + let request = try EngramServiceWebPatchAiSettingsRequest( + aiBaseURL: "https://other.example/v1", titleBaseUrl: "https://other.example/v1", + embeddingBaseURL: "https://other.example/v1" + ) + XCTAssertThrowsError(try EngramServiceCommandHandler.webPatchAiSettings( + request, settingsURL: settingsURL, environment: [:], keychainMayContain: { $0 == account } + )) + for explicit in [false, true] { + let path = root.appendingPathComponent(explicit ? "custom-secrets.json" : "run/ai-secrets.json") + try FileManager.default.createDirectory(at: path.deletingLastPathComponent(), withIntermediateDirectories: true) + try writeSettings([account: "fixture-sidecar"], url: path) + let environment = explicit ? ["ENGRAM_RUNTIME_AI_SECRETS_PATH": path.path] : [:] + XCTAssertThrowsError(try isolatedPatch(request, environment: environment)) + try FileManager.default.removeItem(at: path) + } + } + XCTAssertTrue(try currentObject().isEmpty) + } + + func testUnreadableRuntimeSecretStateFailsClosedAndDoesNotFollowSymlink() throws { + try writeSettings([:]) + let outside = root.appendingPathComponent("outside.json") + try writeSettings(["aiApiKey": "fixture-outside"], url: outside) + let sidecar = root.appendingPathComponent("secrets.json") + let environment = ["ENGRAM_RUNTIME_AI_SECRETS_PATH": sidecar.path] + let patch = try EngramServiceWebPatchAiSettingsRequest(aiBaseURL: "https://other.example/v1") + try FileManager.default.createSymbolicLink(at: sidecar, withDestinationURL: outside) + XCTAssertThrowsError(try isolatedPatch(patch, environment: environment)) + XCTAssertEqual(try currentObject(url: outside)["aiApiKey"] as? String, "fixture-outside") + try FileManager.default.removeItem(at: sidecar) + try Data("invalid json".utf8).write(to: sidecar) + XCTAssertThrowsError(try isolatedPatch(patch, environment: environment)) + try writeSettings([:], url: sidecar) + XCTAssertNoThrow(try isolatedPatch(patch, environment: environment)) + } + + func testKeychainPresenceProbeNeverRequestsSecretDataOrInteraction() { + for status in [errSecSuccess, errSecItemNotFound, errSecInteractionNotAllowed, errSecAuthFailed, errSecNotAvailable] { + let possible = EngramServiceCommandHandler.webAiKeychainMayContain("aiApiKey") { raw in + let query = raw as NSDictionary + XCTAssertEqual(query[kSecAttrService] as? String, "com.engram.app") + XCTAssertEqual(query[kSecAttrAccount] as? String, "aiApiKey") + XCTAssertEqual(query[kSecReturnAttributes] as? Bool, true) + XCTAssertNil(query[kSecReturnData]) + XCTAssertEqual((query[kSecUseAuthenticationContext] as? LAContext)?.interactionNotAllowed, true) + return status + } + XCTAssertEqual(possible, status != errSecItemNotFound) + } + } + + func testEndpointRejectionSurvivesRealServiceIPCAndLeavesSettingsIntact() async throws { + let previous = getenv("ENGRAM_SETTINGS_PATH").map { String(cString: $0) } + setenv("ENGRAM_SETTINGS_PATH", settingsURL.path, 1) + defer { + if let previous { setenv("ENGRAM_SETTINGS_PATH", previous, 1) } + else { unsetenv("ENGRAM_SETTINGS_PATH") } + } + try writeSettings(["aiApiKey": "fixture-ipc", "aiModel": "keep-me"]) + let original = try Data(contentsOf: settingsURL) + let database = root.appendingPathComponent("index.sqlite") + try EngramDatabaseWriter(path: database.path).migrate() + let gate = try ServiceWriterGate(databasePath: database.path, runtimeDirectory: root) + let handler = EngramServiceCommandHandler(writerGate: gate) + let socket = root.appendingPathComponent("service.sock").path + let server = UnixSocketServiceServer(socketPath: socket) { request in await handler.handle(request) } + try server.start() + defer { server.stop() } + let client = try EngramServiceWebWriteClient(socketPath: socket) + do { + _ = try await client.patchAiSettings(.init(aiBaseURL: "https://other.example/v1", aiModel: "must-not-save")) + XCTFail("Endpoint changes with saved credentials must be rejected through real IPC") + } catch { + XCTAssertEqual(error as? EngramServiceWebWriteClientError, .aiEndpointRequiresLocalUpdate) + } + XCTAssertEqual(try Data(contentsOf: settingsURL), original) + let accepted = try await client.patchAiSettings(.init(aiModel: "safe-edit")) + XCTAssertEqual(accepted.settings.aiModel, "safe-edit") + XCTAssertEqual(try currentObject()["aiApiKey"] as? String, "fixture-ipc") + } + + private func isolatedPatch( + _ patch: EngramServiceWebPatchAiSettingsRequest, environment: [String: String] = [:] + ) throws -> EngramServiceWebAiSettingsResponse { + try EngramServiceCommandHandler.webPatchAiSettings( + patch, settingsURL: settingsURL, environment: environment, keychainMayContain: { _ in false } + ) + } + private func writeSettings(_ object: [String: Any], mode: Int = 0o600, url: URL? = nil) throws { let destination = url ?? settingsURL! try JSONSerialization.data(withJSONObject: object).write(to: destination) diff --git a/macos/EngramServiceCoreTests/WebAliasWriteTests.swift b/macos/EngramServiceCoreTests/WebAliasWriteTests.swift index 798d4c840..f5feb7025 100644 --- a/macos/EngramServiceCoreTests/WebAliasWriteTests.swift +++ b/macos/EngramServiceCoreTests/WebAliasWriteTests.swift @@ -10,6 +10,160 @@ final class WebAliasWriteTests: XCTestCase { private let siblingAlias = "/keep/path" private let siblingCanonical = "/Users/fixture/other" + func testAliasActorsReachAuditThroughCapabilitySocketAndDispatcher_repro() async throws { + let env = try prepared(project: rawProject) + defer { env.tearDown() } + let runtime = URL(fileURLWithPath: "/tmp/eg-alias-audit-\(UUID().uuidString.prefix(8))") + try FileManager.default.createDirectory(at: runtime, withIntermediateDirectories: false, + attributes: [.posixPermissions: 0o700]) + defer { try? FileManager.default.removeItem(at: runtime) } + let gate = try ServiceWriterGate(databasePath: env.fixture.path, runtimeDirectory: runtime) + let handler = EngramServiceCommandHandler(writerGate: gate, webMetadataProducer: env.producer) + let socket = runtime.appendingPathComponent("service.sock").path + let server = UnixSocketServiceServer(socketPath: socket) { await handler.handle($0) } + try server.start() + defer { server.stop() } + let client = try EngramServiceWebWriteClient(socketPath: socket) + let canonical = try XCTUnwrap(EngramServiceWebWriteValidation.publishedProjectKey(rawProject)) + let alias = try XCTUnwrap(EngramServiceWebWriteValidation.publishedProjectKey(rawAlias)) + let added = try await client.addAlias(decode([ + "alias": rawAlias, "canonical": canonical, "actor": "editor@example.com", + ])) + XCTAssertEqual(added.changed, 1) + let removed = try await client.removeAlias(decode([ + "alias": alias, "canonical": canonical, "actor": "remover@example.com", + ])) + XCTAssertEqual(removed.changed, 1) + _ = try await client.addAlias(EngramServiceWebAddAliasRequest(canonical: canonical, alias: rawAlias)) + XCTAssertEqual(try auditRows(env.writer).map { $0["actor"] as String }, + ["editor@example.com", "remover@example.com", "web-editor"]) + try env.writer.write { db in + try db.execute(sql: """ + CREATE TRIGGER fail_alias_audit BEFORE INSERT ON web_alias_audit_log + BEGIN SELECT RAISE(ABORT, 'private database detail'); END; + """) + } + do { + _ = try await client.removeAlias(EngramServiceWebRemoveAliasRequest(alias: alias, canonical: canonical)) + XCTFail("Audit failure must not become successful removal") + } catch { + XCTAssertEqual(error as? EngramServiceWebWriteClientError, .unavailable) + } + XCTAssertEqual(try pairKeys(env.writer), [rawAlias + "\u{1E}" + rawProject]) + XCTAssertEqual(try auditRows(env.writer).count, 3) + } + + func testAliasAuditRecordsOnlyTransitionsAndPublishedIdentities_repro() throws { + let env = try prepared(project: rawProject) + defer { env.tearDown() } + try insertSibling(env.writer) + let canonical = try XCTUnwrap(EngramServiceWebWriteValidation.publishedProjectKey(rawProject)) + let alias = try XCTUnwrap(EngramServiceWebWriteValidation.publishedProjectKey(rawAlias)) + let add: EngramServiceWebAddAliasRequest = try decode([ + "alias": rawAlias, "canonical": canonical, "actor": "editor@example.com", + ]) + XCTAssertEqual(try env.producer.addProjectAlias(add, writer: env.writer).changed, 1) + XCTAssertEqual(try env.producer.addProjectAlias(add, writer: env.writer).changed, 0) + let remove: EngramServiceWebRemoveAliasRequest = try decode([ + "alias": alias, "canonical": canonical, "actor": "remover@example.com", + ]) + XCTAssertEqual(try env.producer.removeProjectAlias(remove, writer: env.writer).changed, 1) + XCTAssertEqual(try env.producer.removeProjectAlias(remove, writer: env.writer).changed, 0) + _ = try env.producer.addProjectAlias( + EngramServiceWebAddAliasRequest(canonical: canonical, alias: rawAlias), writer: env.writer + ) + let rows = try auditRows(env.writer) + XCTAssertEqual(rows.map { $0["actor"] as String }, ["editor@example.com", "remover@example.com", "web-editor"]) + XCTAssertEqual(rows.map { $0["action"] as String }, ["add", "remove", "add"]) + XCTAssertEqual(rows.map { $0["alias"] as String }, [alias, alias, alias]) + XCTAssertEqual(rows.map { $0["canonical"] as String }, [canonical, canonical, canonical]) + for row in rows { + XCTAssertFalse((row["created_at"] as String).isEmpty) + } + XCTAssertEqual(try pairKeys(env.writer), [ + siblingAlias + "\u{1E}" + siblingCanonical, rawAlias + "\u{1E}" + rawProject, + ]) + let rejected: EngramServiceWebAddAliasRequest = try decode(["alias": rawAlias, "canonical": "missing"]) + XCTAssertThrowsError(try env.producer.addProjectAlias(rejected, writer: env.writer)) + XCTAssertEqual(try auditRows(env.writer).count, 3) + } + + func testAliasAuditRejectsInvalidActorDuringDecoding_repro() throws { + for actor in ["", " editor", "editor\n", "a b", "\"", "\\", "中", String(repeating: "a", count: 255)] { + let data = try JSONSerialization.data(withJSONObject: [ + "alias": "old", "canonical": "new", "actor": actor, + ]) + XCTAssertThrowsError(try JSONDecoder().decode(EngramServiceWebAddAliasRequest.self, from: data), actor) + XCTAssertThrowsError(try JSONDecoder().decode(EngramServiceWebRemoveAliasRequest.self, from: data), actor) + } + } + + func testAliasAuditInsertFailureRollsBackAddAndRemove_repro() throws { + let env = try prepared(project: rawProject) + defer { env.tearDown() } + let canonical = try XCTUnwrap(EngramServiceWebWriteValidation.publishedProjectKey(rawProject)) + let alias = try XCTUnwrap(EngramServiceWebWriteValidation.publishedProjectKey(rawAlias)) + let add = try EngramServiceWebAddAliasRequest(canonical: canonical, alias: rawAlias) + try env.writer.write { db in + try db.execute(sql: """ + CREATE TRIGGER fail_alias_audit BEFORE INSERT ON web_alias_audit_log + BEGIN SELECT RAISE(ABORT, 'injected audit failure'); END; + """) + } + XCTAssertThrowsError(try env.producer.addProjectAlias(add, writer: env.writer)) + XCTAssertEqual(try pairKeys(env.writer), []) + XCTAssertEqual(try auditRows(env.writer).count, 0) + try env.writer.write { try $0.execute(sql: "DROP TRIGGER fail_alias_audit") } + _ = try env.producer.addProjectAlias(add, writer: env.writer) + try env.writer.write { db in + try db.execute(sql: """ + CREATE TRIGGER fail_alias_audit BEFORE INSERT ON web_alias_audit_log + BEGIN SELECT RAISE(ABORT, 'injected audit failure'); END; + """) + } + XCTAssertThrowsError(try env.producer.removeProjectAlias( + EngramServiceWebRemoveAliasRequest(alias: alias, canonical: canonical), writer: env.writer + )) + XCTAssertEqual(try pairKeys(env.writer), [rawAlias + "\u{1E}" + rawProject]) + XCTAssertEqual(try auditRows(env.writer).count, 1) + } + + func testAliasAuditRetentionAndPruningFailureAreTransactional_repro() throws { + let env = try prepared(project: rawProject) + defer { env.tearDown() } + let canonical = try XCTUnwrap(EngramServiceWebWriteValidation.publishedProjectKey(rawProject)) + let add = try EngramServiceWebAddAliasRequest(canonical: canonical, alias: rawAlias) + try env.writer.write { db in + try db.execute(sql: """ + WITH RECURSIVE n(x) AS (VALUES(1) UNION ALL SELECT x + 1 FROM n WHERE x < 10000) + INSERT INTO web_alias_audit_log (action, actor, alias, canonical, created_at) + SELECT 'add', 'seed', 'old', 'new', + CASE WHEN x = 1 THEN '2099-01-01T00:00:00Z' ELSE '2000-01-01T00:00:00Z' END FROM n; + CREATE TRIGGER fail_alias_prune BEFORE DELETE ON web_alias_audit_log + BEGIN SELECT RAISE(ABORT, 'injected prune failure'); END; + """) + } + XCTAssertThrowsError(try env.producer.addProjectAlias(add, writer: env.writer)) + XCTAssertEqual(try pairKeys(env.writer), []) + XCTAssertEqual(try auditRows(env.writer).count, 10_000) + try env.writer.write { try $0.execute(sql: "DROP TRIGGER fail_alias_prune") } + _ = try env.producer.addProjectAlias(add, writer: env.writer) + let rows = try auditRows(env.writer) + XCTAssertEqual(rows.count, 10_000) + XCTAssertEqual(rows.first?["id"] as Int?, 2, "Retention uses insertion order despite clock rollback") + XCTAssertEqual(rows.last?["actor"] as String?, "web-editor") + XCTAssertEqual(try env.producer.addProjectAlias(add, writer: env.writer).changed, 0) + XCTAssertEqual(try auditRows(env.writer).map { $0["id"] as Int }, rows.map { $0["id"] as Int }) + } + + private func decode(_ object: [String: String]) throws -> T { + try JSONDecoder().decode(T.self, from: JSONSerialization.data(withJSONObject: object)) + } + + private func auditRows(_ writer: EngramDatabaseWriter) throws -> [Row] { + try writer.read { try Row.fetchAll($0, sql: "SELECT * FROM web_alias_audit_log ORDER BY id") } + } + func testAddStoresPathShapedAliasAgainstAuthorizedRawCanonical() throws { let env = try prepared(project: rawProject) defer { env.tearDown() } diff --git a/macos/EngramServiceCoreTests/WebProjectMigrationWriteTests.swift b/macos/EngramServiceCoreTests/WebProjectMigrationWriteTests.swift index 366045d8d..c0e80a52e 100644 --- a/macos/EngramServiceCoreTests/WebProjectMigrationWriteTests.swift +++ b/macos/EngramServiceCoreTests/WebProjectMigrationWriteTests.swift @@ -1,5 +1,6 @@ import Darwin import Foundation +import GRDB import XCTest import EngramCoreWrite @testable import EngramServiceCore @@ -141,6 +142,86 @@ final class WebProjectMigrationWriteTests: XCTestCase { } } + func testWebProjectActorsReachAuditThroughRealIPCAndScopeReplay_repro() async throws { + try skipIfGitMissing() + try await withIsolatedFixedHome { home in + let paths = try makePaths() + defer { try? FileManager.default.removeItem(at: paths.database.deletingLastPathComponent()) } + try migrate(paths.database) + let gate = try ServiceWriterGate(databasePath: paths.database.path, runtimeDirectory: paths.runtime) + let handler = EngramServiceCommandHandler(writerGate: gate) + let socket = paths.runtime.appendingPathComponent("service.sock").path + let server = UnixSocketServiceServer(socketPath: socket) { request in await handler.handle(request) } + try server.start() + defer { server.stop() } + let client = try EngramServiceWebWriteClient(socketPath: socket) + let (src, dst) = try makeProjectPair(home: home, name: "audit-move") + let operationId = UUID().uuidString + var moveBody: [String: Any] = ["src": src, "dst": dst, "dry_run": false, + "operation_id": operationId, "actor": "editor@example.com"] + let moved = try await client.projectMove( + JSONDecoder().decode(EngramServiceWebProjectMoveRequest.self, + from: JSONSerialization.data(withJSONObject: moveBody))) + XCTAssertEqual(moved.result.state, "committed") + moveBody["actor"] = "other@example.com" + do { + _ = try await client.projectMove( + JSONDecoder().decode(EngramServiceWebProjectMoveRequest.self, + from: JSONSerialization.data(withJSONObject: moveBody))) + XCTFail("a different actor must not receive the cached operation result") + } catch let error as EngramServiceWebWriteClientError { + guard case .project(let failure) = error else { + return XCTFail("expected a Web operation conflict, got \(error)") + } + XCTAssertEqual(failure.category, "conflict") + } + + let undoBody: [String: Any] = ["migration_id": moved.result.migrationId, + "operation_id": UUID().uuidString, "actor": "undo@example.com"] + let undone = try await client.projectUndo( + JSONDecoder().decode(EngramServiceWebProjectUndoRequest.self, + from: JSONSerialization.data(withJSONObject: undoBody))) + XCTAssertEqual(undone.result.state, "committed") + + let (archiveSrc, _) = try makeProjectPair(home: home, name: "audit-archive") + let archiveBody: [String: Any] = ["src": archiveSrc, "dry_run": false, + "operation_id": UUID().uuidString, "actor": "archive@example.com"] + let archived = try await client.projectArchive( + JSONDecoder().decode(EngramServiceWebProjectArchiveRequest.self, + from: JSONSerialization.data(withJSONObject: archiveBody))) + XCTAssertEqual(archived.result.state, "committed") + + let (batchSrc, batchDst) = try makeProjectPair(home: home, name: "audit-batch") + let yaml = String(decoding: try JSONSerialization.data(withJSONObject: ["version": 1, + "operations": [["src": batchSrc, "dst": batchDst]]]), as: UTF8.self) + let batchBody: [String: Any] = ["yaml": yaml, "dry_run": false, + "operation_id": UUID().uuidString, "actor": "batch@example.com"] + _ = try await client.projectMoveBatch( + JSONDecoder().decode(EngramServiceWebProjectMoveBatchRequest.self, + from: JSONSerialization.data(withJSONObject: batchBody))) + + let (legacySrc, legacyDst) = try makeProjectPair(home: home, name: "audit-legacy") + _ = try await client.projectMove( + EngramServiceWebProjectMoveRequest(src: legacySrc, dst: legacyDst, + dryRun: false, operationId: UUID().uuidString)) + let writer = try EngramDatabaseWriter(path: paths.database.path) + let actors = try writer.read { try String.fetchAll($0, sql: "SELECT actor FROM migration_log ORDER BY actor") } + XCTAssertEqual(actors, ["archive@example.com", "batch@example.com", "editor@example.com", "undo@example.com", "web-editor"]) + } + } + + func testWebProjectActorDecodingRejectsInvalidAttribution_repro() throws { + for actor in ["", "line\nbreak", "with space", "quote\"", "back\\slash", String(repeating: "x", count: 255)] { + let payload: [String: Any] = ["src": "/tmp/a", "dst": "/tmp/b", "dry_run": true, + "operation_id": UUID().uuidString, "actor": actor, "migration_id": "fixture", "yaml": "{}"] + let data = try JSONSerialization.data(withJSONObject: payload) + XCTAssertThrowsError(try JSONDecoder().decode(EngramServiceWebProjectMoveRequest.self, from: data), actor) + XCTAssertThrowsError(try JSONDecoder().decode(EngramServiceWebProjectArchiveRequest.self, from: data), actor) + XCTAssertThrowsError(try JSONDecoder().decode(EngramServiceWebProjectUndoRequest.self, from: data), actor) + XCTAssertThrowsError(try JSONDecoder().decode(EngramServiceWebProjectMoveBatchRequest.self, from: data), actor) + } + } + private func withIsolatedFixedHome(_ body: (URL) async throws -> Void) async throws { let home = URL(fileURLWithPath: "/tmp", isDirectory: true) .appendingPathComponent("engram-d15-\(UUID().uuidString.prefix(8))", isDirectory: true) diff --git a/macos/EngramTests/DatabaseManagerTests.swift b/macos/EngramTests/DatabaseManagerTests.swift index 1c6481d50..3ac3d1a8e 100644 --- a/macos/EngramTests/DatabaseManagerTests.swift +++ b/macos/EngramTests/DatabaseManagerTests.swift @@ -1310,6 +1310,134 @@ final class DatabaseManagerTests: XCTestCase { XCTAssertEqual(results.map(\.id), ["s1"]) } + @MainActor + func testSearchValueBandsMatchBadgesAndApplyBeforeLimitAcrossReaders() async throws { + let scores: [(String, Int?)] = [ + ("top", 100), ("high", 60), ("mid-high", 59), ("mid-low", 36), + ("low", 35), ("zero", 0), ("unknown", nil), + ] + let queue = try DatabaseQueue(path: dbPath) + for (index, entry) in scores.enumerated() { + try insertTestSession(at: dbPath, id: entry.0, startTime: "2026-05-0\(index + 1)T00:00:00Z") + try insertFTSContent(at: dbPath, sessionId: entry.0, content: "bandneedle AI") + try await queue.write { db in + try db.execute(sql: "UPDATE sessions SET quality_score = ? WHERE id = ?", + arguments: [entry.1, entry.0]) + } + } + let provider = try SQLiteEngramServiceReadProvider(databasePath: dbPath) + let cases: [(String?, [String])] = [ + (nil, scores.reversed().map(\.0)), ("high", ["high", "top"]), + ("medium", ["mid-low", "mid-high"]), ("low", ["zero", "low"]), + ("unknown", ["unknown"]), ("invalid", []), + ] + for (band, expected) in cases { + for query in ["bandneedle", "AI"] { + for limit in [1, 10] { + let ids = Array(expected.prefix(limit)) + let local = try db.search(query: query, limit: limit, valueBand: band) + XCTAssertEqual(local.map(\.id), ids, "plain \(query), \(String(describing: band))") + if let band, band != "invalid" { + XCTAssertTrue(local.allSatisfy { $0.valueBand.rawValue == band }) + } + let snippets = try db.searchWithSnippets(query: query, limit: limit, valueBand: band) + XCTAssertEqual(snippets.map(\.session.id), ids) + let response = try await provider.search(.init( + query: query, mode: "keyword", limit: limit, valueBand: band + )) + XCTAssertEqual(response.items.map(\.id), ids, "service \(query), \(String(describing: band))") + } + } + } + let combined = try db.search(query: "bandneedle", sources: ["codex"], valueBand: "high") + XCTAssertTrue(combined.isEmpty, "value filter must intersect existing filters") + let browse = try db.search(query: " ", valueBand: "high") + XCTAssertEqual(browse.map(\.id), ["high", "top"]) + } + + @MainActor + func testKeywordRankingUsesEveryTermBeforeLimitAcrossReaders_repro() async throws { + try insertTestSession(at: dbPath, id: "relevant", startTime: "2026-05-01T00:00:00Z") + try insertTestSession(at: dbPath, id: "recent", startTime: "2026-05-02T00:00:00Z") + try insertTestSession(at: dbPath, id: "partial", startTime: "2026-05-03T00:00:00Z") + for id in ["relevant", "recent", "partial"] { + try insertFTSContent(at: dbPath, sessionId: id, content: "alpha planning") + try insertFTSContent(at: dbPath, sessionId: id, content: "gamma verification") + } + try insertFTSContent(at: dbPath, sessionId: "relevant", content: "beta beta beta beta beta") + try insertFTSContent(at: dbPath, sessionId: "recent", + content: "beta " + String(repeating: "unrelated material ", count: 50)) + + // Alpha and gamma have identical rank in both complete sessions. Beta + // must contribute even when it is not first, across separate messages. + for query in ["alpha beta gamma", "alpha gamma beta", "beta alpha gamma", + "beta gamma alpha", "gamma alpha beta", "gamma beta alpha"] { + try await assertHumanKeywordRanking(query, expected: ["relevant", "recent"]) + try await assertHumanKeywordRanking(query, limit: 1, expected: ["relevant"]) + } + } + + @MainActor + func testKeywordRankingShortFirstKeepsLongTermRelevanceAcrossReaders_repro() async throws { + try insertTestSession(at: dbPath, id: "relevant", startTime: "2026-05-01T00:00:00Z") + try insertTestSession(at: dbPath, id: "recent", startTime: "2026-05-02T00:00:00Z") + let queue = try DatabaseQueue(path: dbPath) + for (short, long) in [("AI", "needle"), ("测试", "需要修复"), + ("AI", "데이터베이스"), ("AI", "データベース")] { + try await queue.write { sqlDB in try sqlDB.execute(sql: "DELETE FROM sessions_fts") } + try insertFTSContent(at: dbPath, sessionId: "relevant", + content: String(repeating: "\(long) ", count: 5) + short) + try insertFTSContent(at: dbPath, sessionId: "recent", + content: "\(short) \(long) " + String(repeating: "unrelated material ", count: 50)) + for query in [long, "\(short) \(long)", "\(long) \(short)"] { + try await assertHumanKeywordRanking(query, expected: ["relevant", "recent"]) + try await assertHumanKeywordRanking(query, limit: 1, expected: ["relevant"]) + } + // A short-only search retains the recency fallback. + try await assertHumanKeywordRanking(short, expected: ["recent", "relevant"]) + } + } + + @MainActor + func testKeywordRankingBreaksTiesByRecencyThenIdentityAcrossReaders() async throws { + for id in ["z-tie", "a-tie", "newest"] { + try insertTestSession(at: dbPath, id: id, + startTime: id == "newest" ? "2026-05-02T00:00:00Z" : "2026-05-01T00:00:00Z") + try insertFTSContent(at: dbPath, sessionId: id, content: "alpha beta AI") + } + for query in ["alpha beta", "beta alpha", "AI"] { + try await assertHumanKeywordRanking(query, expected: ["newest", "a-tie", "z-tie"]) + try await assertHumanKeywordRanking(query, limit: 2, expected: ["newest", "a-tie"]) + } + } + + @MainActor + private func assertHumanKeywordRanking( + _ query: String, + limit: Int = 10, + expected: [String], + file: StaticString = #filePath, + line: UInt = #line + ) async throws { + XCTAssertEqual(try db.search(query: query, limit: limit).map(\.id), expected, + "App query=\(query), limit=\(limit)", file: file, line: line) + let snippets = try db.searchWithSnippets(query: query, limit: limit) + XCTAssertEqual(snippets.map(\.session.id), expected, + "App snippets query=\(query), limit=\(limit)", file: file, line: line) + let provider = try SQLiteEngramServiceReadProvider(databasePath: dbPath) + let response = try await provider.search( + EngramServiceSearchRequest(query: query, mode: "keyword", limit: limit) + ) + XCTAssertEqual(response.items.map(\.id), expected, + "Service query=\(query), limit=\(limit)", file: file, line: line) + for snippet in snippets.map(\.snippet) + response.items.compactMap(\.snippet) { + for token in query.split(separator: " ") { + XCTAssertTrue(snippet.contains(token), "missing \(token): \(snippet)", file: file, line: line) + } + XCTAssertTrue(snippet.contains(""), snippet, file: file, line: line) + } + } + // quality_score (already computed at index time) must decode into the GUI // read model. Session uses an explicit CodingKeys enum, so qualityScore must // be a listed key or it silently stays nil. @@ -1575,7 +1703,7 @@ final class DatabaseManagerTests: XCTestCase { try insertTestSession(at: dbPath, id: "s-cjk", source: "claude-code") try insertFTSContent(at: dbPath, sessionId: "s-cjk", content: "重构了数据库连接池逻辑") - // CJK path requires query.count >= 2 and uses LIKE fallback + // Three-scalar CJK uses the same trigram MATCH route as Service. let results = try db.search(query: "数据库") XCTAssertEqual(results.count, 1) XCTAssertEqual(results.first?.id, "s-cjk") @@ -1595,9 +1723,7 @@ final class DatabaseManagerTests: XCTestCase { XCTAssertNoThrow(try db.searchWithSnippets(query: "handleRequest(payload)", limit: 5)) } - // Hangul must route through the CJK LIKE fallback (trigram MATCH is broken for - // Korean). Before the containsCJK fix, Hangul Syllables (>= U+AC00) were not - // detected, so this took the MATCH path and returned nothing. + // Long Hangul terms must remain searchable through the Unicode trigram path. @MainActor func testSearchWithKoreanContent() throws { try insertTestSession(at: dbPath, id: "s-ko", source: "claude-code") @@ -1892,7 +2018,7 @@ final class DatabaseManagerTests: XCTestCase { let local = try db.search(query: "alpha beta", limit: 10).map(\.id) // (a) Parity with the service-side CTE query shape (drive from FTS, - // inner-join per term, order by first term's rank then start_time). + // inner-join per term, order by combined rank then start_time and id). let terms = CJKText.ftsMatchTerms("alpha beta") let reference = try DatabaseQueue(path: dbPath).read { db -> [String] in try String.fetchAll(db, sql: """ @@ -1905,7 +2031,7 @@ final class DatabaseManagerTests: XCTestCase { JOIN m1 ON m1.session_id = m0.session_id JOIN sessions s ON s.id = m0.session_id WHERE s.hidden_at IS NULL AND (s.tier IS NULL OR s.tier NOT IN ('skip', 'lite')) - ORDER BY m0.rank, s.start_time DESC + ORDER BY m0.rank + m1.rank, s.start_time DESC, s.id LIMIT 10 """, arguments: StatementArguments(terms)) } diff --git a/macos/EngramTests/EngramServiceLauncherTests.swift b/macos/EngramTests/EngramServiceLauncherTests.swift index 686d55864..84f6a4c81 100644 --- a/macos/EngramTests/EngramServiceLauncherTests.swift +++ b/macos/EngramTests/EngramServiceLauncherTests.swift @@ -1071,7 +1071,13 @@ final class EngramServiceLauncherTests: XCTestCase { // Thread.sleep on the main actor, blocking the run loop for up to the // full 2s timeout. The bounded wait now suspends instead of blocking, // so the call must return promptly even while the child is alive. - let executable = try makeSleeperExecutable() + let marker = FileManager.default.temporaryDirectory + .appendingPathComponent("engram-stop-ready-\(UUID().uuidString)") + let executable = try makeDelayedTerminationExecutable(marker: marker) + defer { + try? FileManager.default.removeItem(at: marker) + try? FileManager.default.removeItem(at: executable.deletingLastPathComponent()) + } let launcher = EngramServiceLauncher() let config = EngramServiceLaunchConfiguration( executablePath: executable.path, @@ -1081,6 +1087,10 @@ final class EngramServiceLauncherTests: XCTestCase { ) try launcher.start(configuration: config) + // Wait until SIGTERM is ignored; an ordinary sleeper may exit before + // the assertion below even when stopIfOwned correctly returns at once. + let ready = await waitForMarkerLineCount(marker, count: 1, timeoutNanoseconds: 5_000_000_000) + XCTAssertTrue(ready, "helper must install its termination handler before stop") XCTAssertTrue(launcher.isRunning) let start = Date() @@ -1183,7 +1193,7 @@ final class EngramServiceLauncherTests: XCTestCase { let initialSpawned = await waitForMarkerLineCount( marker, count: 1, - timeoutNanoseconds: 500_000_000 + timeoutNanoseconds: 5_000_000_000 ) XCTAssertTrue(initialSpawned) @@ -1197,7 +1207,7 @@ final class EngramServiceLauncherTests: XCTestCase { let spawnedReplacement = await waitForMarkerLineCount( marker, count: 2, - timeoutNanoseconds: 250_000_000 + timeoutNanoseconds: 5_000_000_000 ) // docs/invariants.md #1: a replacement must not race the still-live @@ -1234,7 +1244,7 @@ final class EngramServiceLauncherTests: XCTestCase { let initialStarted = await waitForMarkerLineCount( marker, count: 1, - timeoutNanoseconds: 500_000_000 + timeoutNanoseconds: 5_000_000_000 ) XCTAssertTrue(initialStarted) await launcher.restart( @@ -1245,7 +1255,7 @@ final class EngramServiceLauncherTests: XCTestCase { let replacementStarted = await waitForMarkerLineCount( marker, count: 2, - timeoutNanoseconds: 500_000_000 + timeoutNanoseconds: 5_000_000_000 ) launcher.stopIfOwned() diff --git a/macos/EngramTests/HomePopoverActionsTests.swift b/macos/EngramTests/HomePopoverActionsTests.swift index c6f7dd026..210310d4e 100644 --- a/macos/EngramTests/HomePopoverActionsTests.swift +++ b/macos/EngramTests/HomePopoverActionsTests.swift @@ -8,6 +8,55 @@ import XCTest /// embedding bindings, the indexing vs empty states, and the popover Live /// section that must reuse the badge's exact active-session predicate. final class HomePopoverActionsTests: XCTestCase { + @MainActor + func testPopoverDateGroupsKeepDistinctIdentityForResumedSessions_repro() throws { + // Recent activity order can interleave sessions started on the same day. + let sessions = try [ + timelineSession(id: "recent", day: "2020-01-02"), + timelineSession(id: "resumed", day: "2020-01-01"), + timelineSession(id: "earlier", day: "2020-01-02"), + ] + let groups = PopoverView.groupedByDate(sessions) + + XCTAssertEqual(groups.map(\.key), ["2020-01-02", "2020-01-01", "2020-01-02"]) + XCTAssertEqual(groups.flatMap(\.sessions).map(\.id), sessions.map(\.id)) + XCTAssertEqual(Set(groups.map(\.id)).count, groups.count, + "Repeated date labels must not produce duplicate SwiftUI identities") + } + + @MainActor + func testPopoverDateGroupsKeepAdjacentSessionsAndStableIdentity() throws { + let sessions = try [ + timelineSession(id: "first", day: "2020-01-02"), + timelineSession(id: "second", day: "2020-01-02"), + ] + let groups = PopoverView.groupedByDate(sessions) + XCTAssertEqual(groups.count, 1) + XCTAssertEqual(groups.first?.sessions.map(\.id), ["first", "second"]) + XCTAssertEqual(PopoverView.groupedByDate(sessions).map(\.id), groups.map(\.id)) + + let newest = try timelineSession(id: "newest", day: "2020-01-03") + let refreshed = PopoverView.groupedByDate([newest] + sessions) + XCTAssertEqual(refreshed.dropFirst().map(\.id), groups.map(\.id), + "Inserting another date group must not renumber existing groups") + } + + @MainActor + func testPopoverDateGroupsAcceptEmptyTimeline() { + XCTAssertTrue(PopoverView.groupedByDate([]).isEmpty) + } + + private func timelineSession(id: String, day: String) throws -> Session { + let fields: [String: Any] = [ + "id": id, "source": "claude-code", "start_time": "\(day)T12:00:00Z", + "cwd": "/tmp", "message_count": 2, "user_message_count": 1, + "assistant_message_count": 1, "system_message_count": 0, + "tool_message_count": 0, "file_path": "/tmp/\(id).jsonl", + "size_bytes": 0, "indexed_at": "\(day)T12:00:00Z", + ] + return try JSONDecoder().decode(Session.self, from: JSONSerialization.data(withJSONObject: fields)) + } + func testOpenCodeLiveProbeUsesShortBusyTimeout_repro() throws { let provider = try source("macos/EngramService/Core/EngramServiceReadProvider.swift") XCTAssertTrue(provider.contains("configuration.busyMode = .timeout(0.1)")) diff --git a/macos/EngramTests/SearchOutcomeTests.swift b/macos/EngramTests/SearchOutcomeTests.swift index 2f0ea6b95..dc7d1ec0d 100644 --- a/macos/EngramTests/SearchOutcomeTests.swift +++ b/macos/EngramTests/SearchOutcomeTests.swift @@ -1,10 +1,43 @@ import XCTest +import SwiftUI @testable import Engram /// Locks the SearchPageView result-state classifier so a backend double-fault /// stays distinguishable from a genuine no-match (search-2/3, xc-states-4, /// command-palette-5). final class SearchOutcomeTests: XCTestCase { + @MainActor + func testValueFilterPickerRendersAllChoices() throws { + for locale in ["en", "zh-Hans"] { + let content = VStack(alignment: .leading, spacing: 8) { + ForEach(SearchValueFilter.allCases, id: \.self) { filter in + SearchValueFilterPicker(selection: .constant(filter)) + } + } + .padding(16) + .background(Color.white) + .environment(\.colorScheme, .light) + .environment(\.locale, Locale(identifier: locale)) + // Native menu pickers cannot be flattened by ImageRenderer. + let hosting = NSHostingView(rootView: content.frame(width: 320, height: 220)) + let window = NSWindow(contentRect: NSRect(x: -2000, y: -2000, width: 320, height: 220), + styleMask: [.borderless], backing: .buffered, defer: false) + window.isReleasedWhenClosed = false + window.contentView = hosting + window.orderBack(nil) + defer { window.contentView = nil; window.close() } + hosting.layoutSubtreeIfNeeded() + let bitmap = try XCTUnwrap(hosting.bitmapImageRepForCachingDisplay(in: hosting.bounds)) + hosting.cacheDisplay(in: hosting.bounds, to: bitmap) + let image = NSImage(size: hosting.bounds.size) + image.addRepresentation(bitmap) + let attachment = XCTAttachment(image: image) + attachment.name = "value-filter-\(locale)" + attachment.lifetime = .keepAlways + add(attachment) + } + } + // The selected origin is forwarded to both service and offline SQL search // before their result limits. This helper characterizes the corresponding // stored-origin semantics: HQ is exact, nil/non-HQ is local, and all passes. diff --git a/macos/EngramTests/SessionModelTests.swift b/macos/EngramTests/SessionModelTests.swift index 7d1ec3247..5fd6886d8 100644 --- a/macos/EngramTests/SessionModelTests.swift +++ b/macos/EngramTests/SessionModelTests.swift @@ -1,6 +1,7 @@ // macos/EngramTests/SessionModelTests.swift import XCTest import GRDB +import SwiftUI @testable import Engram final class SessionModelTests: XCTestCase { @@ -82,6 +83,51 @@ final class SessionModelTests: XCTestCase { // MARK: - displayTitle tests + func testValueBandBoundariesPreserveUnknownScores() { + for (score, expected): (Int?, Session.ValueBand) in [ + (nil, .unknown), (0, .low), (35, .low), (36, .medium), + (59, .medium), (60, .high), (100, .high), + ] { + var session = makeSession() + session.qualityScore = score + XCTAssertEqual(session.valueBand, expected, "score=\(String(describing: score))") + } + } + + @MainActor + func testValueBandsRenderInSessionCardsAndTableCells() throws { + let sessions = [nil, 20, 45, 65].enumerated().map { index, score in + var session = makeSession(id: "value-\(index)", summary: "Session with \(score.map(String.init) ?? "no") score") + session.qualityScore = score + return session + } + for locale in ["en", "zh-Hans"] { + for scheme in [ColorScheme.light, .dark] { + let content = VStack(alignment: .leading, spacing: 12) { + ForEach(Array(sessions.enumerated()), id: \.element.id) { index, session in + SessionCard(session: session) + SessionTableCell(row: SessionTableRow( + id: session.id, session: session, kind: .parent, depth: 0, index: index + )) + .frame(width: 420) + } + } + .padding(16) + .frame(width: 800) + .background(Theme.background) + .environment(\.locale, Locale(identifier: locale)) + .environment(\.colorScheme, scheme) + let renderer = ImageRenderer(content: content) + renderer.scale = 2 + let image = try XCTUnwrap(renderer.nsImage) + let attachment = XCTAttachment(image: image) + attachment.name = "value-bands-\(locale)-\(scheme)" + attachment.lifetime = .keepAlways + add(attachment) + } + } + } + /// 1. displayTitle returns customName when set func testDisplayTitleCustomName() throws { let session = makeSession(summary: "Summary", customName: "My Custom Name", generatedTitle: "Generated") diff --git a/macos/EngramTests/TranscriptSessionSwitchTests.swift b/macos/EngramTests/TranscriptSessionSwitchTests.swift new file mode 100644 index 000000000..6dcb9e191 --- /dev/null +++ b/macos/EngramTests/TranscriptSessionSwitchTests.swift @@ -0,0 +1,267 @@ +import AppKit +import SwiftUI +import XCTest +@testable import Engram + +/// Mount the actual detail view, keep its SwiftUI identity while switching +/// sessions, and finish non-cooperative work in an explicitly chosen order. +@MainActor +final class TranscriptSessionSwitchTests: XCTestCase { + func testOldPageCompletionCannotClearNewSessionsPagingState_repro() async throws { + for action in [SessionDetailView.TranscriptTestAction.page, .all, .copy] { + for fail in [false, true] { + try await assertOldPageCompletion(action: action, fail: fail, returnToFirstSession: false) + } + } + try await assertOldPageCompletion(action: .page, fail: false, returnToFirstSession: true) + } + + private func assertOldPageCompletion( + action: SessionDetailView.TranscriptTestAction, fail: Bool, returnToFirstSession: Bool + ) async throws { + let fixture = Fixture() + fixture.label = "action=\(action) fail=\(fail) return=\(returnToFirstSession)" + defer { fixture.close() } + try await fixture.waitForRequest("A", offset: 0) + fixture.finish("A", offset: 0, text: "A initial", count: 500) + try await fixture.waitForState { $0.sessionID == "A" && !$0.loading && $0.indexed == ["A initial"] } + fixture.send?(action) + try await fixture.waitForRequest("A", offset: 500) + + fixture.model.session = Self.session("B") + try await fixture.waitForRequest("B", offset: 0) + fixture.finish("B", offset: 0, text: "B initial", count: 500) + try await fixture.waitForState { $0.sessionID == "B" && !$0.loading && $0.indexed == ["B initial"] } + let current = returnToFirstSession ? "A" : "B" + if returnToFirstSession { + fixture.model.session = Self.session("A") + try await fixture.waitForRequest("A", offset: 0) + fixture.finish("A", offset: 0, text: "A current", count: 500) + try await fixture.waitForState { $0.sessionID == "A" && !$0.loading && $0.indexed == ["A current"] } + } + let initial = returnToFirstSession ? "A current" : "B initial" + fixture.send?(.page) + try await fixture.waitForRequest(current, offset: 500, count: returnToFirstSession ? 2 : 1) + try await fixture.waitForState("current page busy before old completion") { $0.paging } + + // The old cancelled task can resume its defer while the current page is + // waiting, including A -> B -> A. A session ID alone is insufficient. + fixture.finish("A", offset: 500, text: "A stale page", count: 1, fail: fail) + try await fixture.settle() + let state = try XCTUnwrap(fixture.state) + XCTAssertEqual(state.sessionID, current) + XCTAssertEqual(state.messages, [initial]) + XCTAssertEqual(state.indexed, [initial]) + XCTAssertFalse(state.failed) + XCTAssertTrue(fixture.copies.isEmpty, "Cancelled copy work must not write a stale transcript") + XCTAssertTrue(state.paging, "Old-page cleanup must not unlock a new session's in-flight page") + let requests = fixture.requestCount + fixture.send?(.page) + try await fixture.settle() + XCTAssertEqual(fixture.requestCount, requests, "A duplicate click must not start or cancel the current page") + guard fixture.requestCount == requests else { return } + fixture.finish(current, offset: 500, text: "current next", count: 1) + try await fixture.waitForState("current page committed") { !$0.paging && $0.indexed == [initial, "current next"] } + } + + func testCurrentPageAllAndCopyCompleteAndReleaseBusyState() async throws { + for action in [SessionDetailView.TranscriptTestAction.page, .all, .copy] { + let fixture = Fixture() + defer { fixture.close() } + try await fixture.waitForRequest("A", offset: 0) + fixture.finish("A", offset: 0, text: "current initial", count: 500) + try await fixture.waitForState { !$0.loading && $0.indexed == ["current initial"] } + fixture.send?(action) + try await fixture.waitForRequest("A", offset: 500) + fixture.finish("A", offset: 500, text: "current next", count: 1) + try await fixture.waitForState { !$0.paging && !$0.hasMore && $0.indexed == ["current initial", "current next"] } + if case .copy = action { + XCTAssertEqual(fixture.copies.count, 1) + XCTAssertTrue(fixture.copies.first?.contains("current initial") == true) + XCTAssertTrue(fixture.copies.first?.contains("current next") == true) + } else { XCTAssertTrue(fixture.copies.isEmpty) } + } + } + + func testImmediateDuplicatePagingActionsStartOnlyOneRequest_repro() async throws { + for action in [SessionDetailView.TranscriptTestAction.page, .all, .copy] { + let fixture = Fixture() + defer { fixture.close() } + try await fixture.waitForRequest("A", offset: 0) + fixture.finish("A", offset: 0, text: "A initial", count: 500) + try await fixture.waitForState { !$0.loading && $0.indexed == ["A initial"] } + fixture.send?(action) + fixture.send?(action) + try await fixture.waitForRequest("A", offset: 500) + try await fixture.settle() + XCTAssertEqual(fixture.requestCount, 2, "Mark paging busy before scheduling its task") + } + } + + func testCancelledIndexBuildCannotReplaceNewSession() async throws { + let fixture = Fixture(holdFirstIndex: true) + defer { fixture.close() } + try await fixture.waitForRequest("A", offset: 0) + fixture.finish("A", offset: 0, text: "A old", count: 1) + try await fixture.waitForIndex() + fixture.model.session = Self.session("B") + try await fixture.waitForRequest("B", offset: 0) + fixture.finish("B", offset: 0, text: "B current", count: 1) + try await fixture.waitForState { $0.sessionID == "B" && !$0.loading && $0.indexed == ["B current"] } + fixture.finishIndex() + try await fixture.settle() + XCTAssertEqual(fixture.state?.messages, ["B current"]) + XCTAssertEqual(fixture.state?.indexed, ["B current"]) + } + + func testOldInitialSuccessAndFailureCannotReplaceNewTranscript() async throws { + for fail in [false, true] { + let fixture = Fixture() + defer { fixture.close() } + try await fixture.waitForRequest("A", offset: 0) + fixture.model.session = Self.session("B") + try await fixture.waitForRequest("B", offset: 0) + fixture.finish("B", offset: 0, text: "B current", count: 1) + try await fixture.waitForState { $0.sessionID == "B" && !$0.loading && $0.indexed == ["B current"] } + fixture.finish("A", offset: 0, text: "A stale", count: 1, fail: fail) + try await fixture.settle() + let state = try XCTUnwrap(fixture.state) + XCTAssertEqual(state.messages, ["B current"]) + XCTAssertEqual(state.indexed, ["B current"]) + XCTAssertFalse(state.failed) + XCTAssertFalse(state.hasMore) + } + } + + fileprivate static func session(_ id: String) -> Session { + Session(id: id, source: "codex", startTime: "2026-10-23T00:00:00Z", endTime: nil, + cwd: "/tmp/engram-transcript-switch", project: nil, model: nil, + messageCount: 1_000, userMessageCount: 500, assistantMessageCount: 500, + systemMessageCount: 0, summary: nil, filePath: "/tmp/fixture-unused.jsonl", + sourceLocator: nil, sizeBytes: 0, indexedAt: "2026-10-23T00:00:00Z", + agentRole: nil, hiddenAt: nil, customName: id, tier: nil, toolMessageCount: 0, + generatedTitle: nil, parentSessionId: nil, suggestedParentId: nil, linkSource: nil) + } + + @MainActor private final class Model: ObservableObject { + @Published var session = TranscriptSessionSwitchTests.session("A") + } + + private struct Host: View { + @ObservedObject var model: Model + let hooks: SessionDetailView.TranscriptTestHooks + let database: DatabaseManager + var body: some View { + SessionDetailView(session: model.session, transcriptTestHooks: hooks) + .environment(database) + .environment(\.engramServiceClient, MockEngramServiceClient()) + } + } + + @MainActor private final class Fixture { + typealias Window = (messages: [ChatMessage], producedCount: Int, truncated: Bool, parseFailed: Bool) + let model = Model() + private var readState: (() -> SessionDetailView.TranscriptTestState)? + private var actionSessionID: String? + var state: SessionDetailView.TranscriptTestState? { readState?() } + var send: ((SessionDetailView.TranscriptTestAction) -> Void)? + var requestCount = 0 + var label = "" + var requests: [String] = [] + var copies: [String] = [] + private var pending: [String: [CheckedContinuation]] = [:] + private var window: NSWindow? + + private var pendingIndex: CheckedContinuation? + + init(holdFirstIndex: Bool = false) { + var holdIndex = holdFirstIndex + let hooks = SessionDetailView.TranscriptTestHooks(parse: { [weak self] id, offset, _ in + guard let self else { throw CancellationError() } + return try await withCheckedThrowingContinuation { continuation in + self.requestCount += 1 + self.requests.append("\(id):\(offset)") + self.pending["\(id):\(offset)", default: []].append(continuation) + } + }, attach: { [weak self] sessionID, readState, send in + self?.actionSessionID = sessionID + self?.readState = readState + self?.send = send + }, beforeIndexedApply: { [weak self] in + guard holdIndex, let self else { return } + holdIndex = false + await withCheckedContinuation { self.pendingIndex = $0 } + }, copy: { [weak self] in self?.copies.append($0) }) + let database = DatabaseManager(path: "/tmp/engram-switch-\(UUID().uuidString)/absent.sqlite") + let hosting = NSHostingView(rootView: Host(model: model, hooks: hooks, database: database)) + let window = NSWindow(contentRect: NSRect(x: -2000, y: -2000, width: 900, height: 700), + styleMask: [.borderless], backing: .buffered, defer: false) + window.isReleasedWhenClosed = false + window.contentView = hosting + window.orderBack(nil) + self.window = window + } + + func close() { + window?.contentView = nil + window?.close() + window = nil + let remaining = pending.values.flatMap { $0 } + pending.removeAll() + for continuation in remaining { continuation.resume(throwing: CancellationError()) } + finishIndex() + send = nil + readState = nil + } + + func finish(_ id: String, offset: Int, text: String, count: Int, fail: Bool = false) { + let key = "\(id):\(offset)" + guard var items = pending[key], !items.isEmpty else { + XCTFail("No pending parser request for \(key)"); return + } + let continuation = items.removeFirst() + pending[key] = items + if fail { continuation.resume(throwing: CocoaError(.fileReadCorruptFile)) } + else { + continuation.resume(returning: ([ChatMessage(role: "user", content: text, systemCategory: .none)], + count, false, false)) + } + } + + func waitForRequest(_ id: String, offset: Int, count: Int = 1) async throws { + try await wait("request \(id):\(offset) count \(count)") { (self.pending["\(id):\(offset)"]?.count ?? 0) >= count } + } + + func waitForIndex() async throws { + try await wait { self.pendingIndex != nil } + } + + func finishIndex() { + let continuation = pendingIndex + pendingIndex = nil + continuation?.resume() + } + + func waitForState(_ detail: String = "state", _ predicate: (SessionDetailView.TranscriptTestState) -> Bool) async throws { + try await wait(detail) { + guard let state = self.state, state.sessionID == self.actionSessionID else { return false } + return predicate(state) + } + } + + private func wait(_ detail: String = "state", _ condition: () -> Bool) async throws { + let deadline = ContinuousClock.now + .seconds(5) + while !condition(), ContinuousClock.now < deadline { + try await Task.sleep(for: .milliseconds(10)) + } + guard condition() else { + XCTFail("Mounted transcript \(label) waiting for \(detail), requests \(requests), state \(String(describing: state))") + throw CocoaError(.coderInvalidValue) + } + } + + func settle() async throws { + try await Task.sleep(for: .milliseconds(150)) + } + } +} diff --git a/macos/Shared/EngramCore/ArchiveV2/ArchiveHash.swift b/macos/Shared/EngramCore/ArchiveV2/ArchiveHash.swift index eff7573c6..261ff951c 100644 --- a/macos/Shared/EngramCore/ArchiveV2/ArchiveHash.swift +++ b/macos/Shared/EngramCore/ArchiveV2/ArchiveHash.swift @@ -2,10 +2,16 @@ import CryptoKit import Foundation public enum ArchiveV2Hash { + private static let hexDigits = Array("0123456789abcdef".utf8) + public static func sha256(_ data: Data) -> String { - SHA256.hash(data: data) - .map { String(format: "%02x", $0) } - .joined() + var bytes = [UInt8]() + bytes.reserveCapacity(64) + for byte in SHA256.hash(data: data) { + bytes.append(hexDigits[Int(byte >> 4)]) + bytes.append(hexDigits[Int(byte & 0x0f)]) + } + return String(decoding: bytes, as: UTF8.self) } public static func isValidSHA256(_ value: String) -> Bool { diff --git a/macos/Shared/EngramCore/Indexing/SearchFilterPredicates.swift b/macos/Shared/EngramCore/Indexing/SearchFilterPredicates.swift index 426937ecd..dad54e7e6 100644 --- a/macos/Shared/EngramCore/Indexing/SearchFilterPredicates.swift +++ b/macos/Shared/EngramCore/Indexing/SearchFilterPredicates.swift @@ -6,6 +6,9 @@ import Foundation /// `projects` therefore accepts canonical and caller-expanded alias values, but /// always matches those values exactly; this helper never introduces `LIKE`. public enum SearchFilterPredicates { + public static let valueBandLowMax = 35 + public static let valueBandHighMin = 60 + public struct Clause: Equatable, Sendable { public let sql: String public let bindings: [String] @@ -19,6 +22,7 @@ public enum SearchFilterPredicates { projects: [String] = [], origin: String? = nil, since: String? = nil, + valueBand: String? = nil, alias: String = "s" ) -> [Clause] { var clauses: [Clause] = [] @@ -55,6 +59,18 @@ public enum SearchFilterPredicates { ) } + if let valueBand = normalized(valueBand) { + let score = column("quality_score", alias: alias) + let sql: String + switch valueBand { + case "high": sql = "\(score) >= \(valueBandHighMin)" + case "medium": sql = "\(score) > \(valueBandLowMax) AND \(score) < \(valueBandHighMin)" + case "low": sql = "\(score) <= \(valueBandLowMax)" + case "unknown": sql = "\(score) IS NULL" + default: sql = "0 = 1" // Unsupported filters must not broaden results. + } + clauses.append(Clause(sql: sql, bindings: [])) + } return clauses } diff --git a/macos/Shared/Service/EngramServiceModels.swift b/macos/Shared/Service/EngramServiceModels.swift index 9d4ce81ce..d88013805 100644 --- a/macos/Shared/Service/EngramServiceModels.swift +++ b/macos/Shared/Service/EngramServiceModels.swift @@ -182,6 +182,7 @@ struct EngramServiceSearchRequest: Codable, Equatable, Sendable { let source: String? let since: String? let origin: String? + let valueBand: String? init( query: String, @@ -190,7 +191,8 @@ struct EngramServiceSearchRequest: Codable, Equatable, Sendable { project: String? = nil, source: String? = nil, since: String? = nil, - origin: String? = nil + origin: String? = nil, + valueBand: String? = nil ) { self.query = query self.mode = mode @@ -199,6 +201,7 @@ struct EngramServiceSearchRequest: Codable, Equatable, Sendable { self.source = source self.since = since self.origin = origin + self.valueBand = valueBand } } diff --git a/macos/Shared/Service/EngramServiceWebProjectModels.swift b/macos/Shared/Service/EngramServiceWebProjectModels.swift index 514564b1f..101326495 100644 --- a/macos/Shared/Service/EngramServiceWebProjectModels.swift +++ b/macos/Shared/Service/EngramServiceWebProjectModels.swift @@ -11,6 +11,14 @@ enum EngramServiceWebProjectValidation { "fs_pending", "fs_done", "committed", "failed", "rolled_back", "dry-run", "cancelled", ] + static func auditActor(_ value: String) throws -> String { + try EngramServiceWebMetadataValidation.require( + !value.isEmpty && value.utf8.count <= 254 + && value.utf8.allSatisfy { (33...126).contains($0) && $0 != 34 && $0 != 92 } + ) + return value + } + static func publishedOperationId(_ value: String) throws -> String { let trimmed = value.trimmingCharacters(in: .whitespacesAndNewlines) try EngramServiceWebMetadataValidation.require(!trimmed.hasPrefix(operationPrefix)) @@ -225,6 +233,7 @@ struct EngramServiceWebProjectMoveRequest: Codable, Equatable, Sendable { let force: Bool let auditNote: String? let operationId: String + let actor: String? init( src: String, @@ -232,7 +241,8 @@ struct EngramServiceWebProjectMoveRequest: Codable, Equatable, Sendable { dryRun: Bool, force: Bool = false, auditNote: String? = nil, - operationId: String + operationId: String, + actor: String? = nil ) throws { self.src = try EngramServiceWebProjectValidation.confinedPath(src, label: "source") self.dst = try EngramServiceWebProjectValidation.confinedPath(dst, label: "destination") @@ -240,6 +250,7 @@ struct EngramServiceWebProjectMoveRequest: Codable, Equatable, Sendable { self.force = force self.auditNote = try auditNote.map(Self.note) self.operationId = try EngramServiceWebProjectValidation.publishedOperationId(operationId) + self.actor = try actor.map(EngramServiceWebProjectValidation.auditActor) } init(from decoder: Decoder) throws { @@ -251,7 +262,8 @@ struct EngramServiceWebProjectMoveRequest: Codable, Equatable, Sendable { dryRun: try container.decode(Bool.self, forKey: .dryRun), force: try container.decodeIfPresent(Bool.self, forKey: .force) ?? false, auditNote: try container.decodeIfPresent(String.self, forKey: .auditNote), - operationId: try container.decode(String.self, forKey: .operationId) + operationId: try container.decode(String.self, forKey: .operationId), + actor: try container.decodeIfPresent(String.self, forKey: .actor) ) } catch { throw DecodingError.dataCorruptedError( @@ -266,6 +278,7 @@ struct EngramServiceWebProjectMoveRequest: Codable, Equatable, Sendable { } enum CodingKeys: String, CodingKey { + case actor case src, dst, force case dryRun = "dry_run" case auditNote = "audit_note" @@ -280,6 +293,7 @@ struct EngramServiceWebProjectArchiveRequest: Codable, Equatable, Sendable { let force: Bool let auditNote: String? let operationId: String + let actor: String? init( src: String, @@ -287,7 +301,8 @@ struct EngramServiceWebProjectArchiveRequest: Codable, Equatable, Sendable { dryRun: Bool, force: Bool = false, auditNote: String? = nil, - operationId: String + operationId: String, + actor: String? = nil ) throws { self.src = try EngramServiceWebProjectValidation.confinedPath(src, label: "source") self.archiveTo = try archiveTo.map { @@ -301,6 +316,7 @@ struct EngramServiceWebProjectArchiveRequest: Codable, Equatable, Sendable { return $0 } self.operationId = try EngramServiceWebProjectValidation.publishedOperationId(operationId) + self.actor = try actor.map(EngramServiceWebProjectValidation.auditActor) } init(from decoder: Decoder) throws { @@ -312,7 +328,8 @@ struct EngramServiceWebProjectArchiveRequest: Codable, Equatable, Sendable { dryRun: try container.decode(Bool.self, forKey: .dryRun), force: try container.decodeIfPresent(Bool.self, forKey: .force) ?? false, auditNote: try container.decodeIfPresent(String.self, forKey: .auditNote), - operationId: try container.decode(String.self, forKey: .operationId) + operationId: try container.decode(String.self, forKey: .operationId), + actor: try container.decodeIfPresent(String.self, forKey: .actor) ) } catch { throw DecodingError.dataCorruptedError( @@ -322,6 +339,7 @@ struct EngramServiceWebProjectArchiveRequest: Codable, Equatable, Sendable { } enum CodingKeys: String, CodingKey { + case actor case src, force case archiveTo = "archive_to" case dryRun = "dry_run" @@ -334,12 +352,14 @@ struct EngramServiceWebProjectUndoRequest: Codable, Equatable, Sendable { let migrationId: String let force: Bool let operationId: String + let actor: String? - init(migrationId: String, force: Bool = false, operationId: String) throws { + init(migrationId: String, force: Bool = false, operationId: String, actor: String? = nil) throws { try EngramServiceWebMetadataValidation.trimmed(migrationId, maximumBytes: 128) self.migrationId = migrationId self.force = force self.operationId = try EngramServiceWebProjectValidation.publishedOperationId(operationId) + self.actor = try actor.map(EngramServiceWebProjectValidation.auditActor) } init(from decoder: Decoder) throws { @@ -348,7 +368,8 @@ struct EngramServiceWebProjectUndoRequest: Codable, Equatable, Sendable { self = try Self( migrationId: try container.decode(String.self, forKey: .migrationId), force: try container.decodeIfPresent(Bool.self, forKey: .force) ?? false, - operationId: try container.decode(String.self, forKey: .operationId) + operationId: try container.decode(String.self, forKey: .operationId), + actor: try container.decodeIfPresent(String.self, forKey: .actor) ) } catch { throw DecodingError.dataCorruptedError( @@ -358,6 +379,7 @@ struct EngramServiceWebProjectUndoRequest: Codable, Equatable, Sendable { } enum CodingKeys: String, CodingKey { + case actor case force case migrationId = "migration_id" case operationId = "operation_id" @@ -369,8 +391,9 @@ struct EngramServiceWebProjectMoveBatchRequest: Codable, Equatable, Sendable { let dryRun: Bool let force: Bool let operationId: String + let actor: String? - init(yaml: String, dryRun: Bool, force: Bool = false, operationId: String) throws { + init(yaml: String, dryRun: Bool, force: Bool = false, operationId: String, actor: String? = nil) throws { try EngramServiceWebMetadataValidation.require( !yaml.isEmpty && yaml.utf8.count <= EngramServiceWebProjectValidation.maximumBatchBytes @@ -380,6 +403,7 @@ struct EngramServiceWebProjectMoveBatchRequest: Codable, Equatable, Sendable { self.dryRun = dryRun self.force = force self.operationId = try EngramServiceWebProjectValidation.publishedOperationId(operationId) + self.actor = try actor.map(EngramServiceWebProjectValidation.auditActor) } init(from decoder: Decoder) throws { @@ -389,7 +413,8 @@ struct EngramServiceWebProjectMoveBatchRequest: Codable, Equatable, Sendable { yaml: try container.decode(String.self, forKey: .yaml), dryRun: try container.decode(Bool.self, forKey: .dryRun), force: try container.decodeIfPresent(Bool.self, forKey: .force) ?? false, - operationId: try container.decode(String.self, forKey: .operationId) + operationId: try container.decode(String.self, forKey: .operationId), + actor: try container.decodeIfPresent(String.self, forKey: .actor) ) } catch { throw DecodingError.dataCorruptedError( @@ -399,6 +424,7 @@ struct EngramServiceWebProjectMoveBatchRequest: Codable, Equatable, Sendable { } enum CodingKeys: String, CodingKey { + case actor case yaml, force case dryRun = "dry_run" case operationId = "operation_id" diff --git a/macos/Shared/Service/EngramServiceWebWriteClient.swift b/macos/Shared/Service/EngramServiceWebWriteClient.swift index f8029b79e..5ca727278 100644 --- a/macos/Shared/Service/EngramServiceWebWriteClient.swift +++ b/macos/Shared/Service/EngramServiceWebWriteClient.swift @@ -282,6 +282,7 @@ struct EngramServiceWebWriteClient: Sendable { private static func failure(_ name: String) -> EngramServiceWebWriteClientError { switch name { + case "WebAiEndpointRequiresLocalUpdate": return .aiEndpointRequiresLocalUpdate case "InvalidRequest", "invalidRequest": return .invalid case "StaleCursor", "staleCursor": return .stale case "UnsupportedCommand", "unsupportedCommand": return .unsupported diff --git a/macos/Shared/Service/EngramServiceWebWriteModels.swift b/macos/Shared/Service/EngramServiceWebWriteModels.swift index 802d0ae40..5b1fa8644 100644 --- a/macos/Shared/Service/EngramServiceWebWriteModels.swift +++ b/macos/Shared/Service/EngramServiceWebWriteModels.swift @@ -8,6 +8,7 @@ enum EngramServiceWebWriteClientError: Error, Equatable, LocalizedError, Sendabl case unavailable case notFound case malformed + case aiEndpointRequiresLocalUpdate case project(EngramServiceWebProjectFailure) var errorDescription: String? { @@ -18,6 +19,7 @@ enum EngramServiceWebWriteClientError: Error, Equatable, LocalizedError, Sendabl case .unavailable: return "Web alias service is unavailable." case .notFound: return "Web write target was not found." case .malformed: return "Web alias response is invalid." + case .aiEndpointRequiresLocalUpdate: return "Change AI endpoints and credentials in the server's local settings." case .project(let failure): return failure.message } } @@ -27,14 +29,16 @@ enum EngramServiceWebWriteClientError: Error, Equatable, LocalizedError, Sendabl struct EngramServiceWebAddAliasRequest: Codable, Equatable, Sendable { let canonical: String let alias: String + let actor: String? - init(canonical: String, alias: String) throws { + init(canonical: String, alias: String, actor: String? = nil) throws { try EngramServiceWebMetadataValidation.projectIdentity(canonical) let alias = try EngramServiceWebWriteValidation.aliasText(alias) let published = try EngramServiceWebWriteValidation.requirePublished(alias) try EngramServiceWebMetadataValidation.require(!published.utf8.elementsEqual(canonical.utf8)) self.canonical = canonical self.alias = alias + self.actor = try actor.map(EngramServiceWebProjectValidation.auditActor) } init(from decoder: Decoder) throws { @@ -42,7 +46,8 @@ struct EngramServiceWebAddAliasRequest: Codable, Equatable, Sendable { do { self = try Self( canonical: try container.decode(String.self, forKey: .canonical), - alias: try container.decode(String.self, forKey: .alias) + alias: try container.decode(String.self, forKey: .alias), + actor: try container.decodeIfPresent(String.self, forKey: .actor) ) } catch { throw DecodingError.dataCorruptedError(forKey: .alias, in: container, debugDescription: "invalid") @@ -54,13 +59,15 @@ struct EngramServiceWebAddAliasRequest: Codable, Equatable, Sendable { struct EngramServiceWebRemoveAliasRequest: Codable, Equatable, Sendable { let alias: String let canonical: String + let actor: String? - init(alias: String, canonical: String) throws { + init(alias: String, canonical: String, actor: String? = nil) throws { try EngramServiceWebMetadataValidation.projectIdentity(alias) try EngramServiceWebMetadataValidation.projectIdentity(canonical) try EngramServiceWebMetadataValidation.require(!alias.utf8.elementsEqual(canonical.utf8)) self.alias = alias self.canonical = canonical + self.actor = try actor.map(EngramServiceWebProjectValidation.auditActor) } init(from decoder: Decoder) throws { @@ -68,7 +75,8 @@ struct EngramServiceWebRemoveAliasRequest: Codable, Equatable, Sendable { do { self = try Self( alias: try container.decode(String.self, forKey: .alias), - canonical: try container.decode(String.self, forKey: .canonical) + canonical: try container.decode(String.self, forKey: .canonical), + actor: try container.decodeIfPresent(String.self, forKey: .actor) ) } catch { throw DecodingError.dataCorruptedError(forKey: .alias, in: container, debugDescription: "invalid") diff --git a/scripts/check-archive-v2-safety.sh b/scripts/check-archive-v2-safety.sh index 57ab51e22..218b328e6 100755 --- a/scripts/check-archive-v2-safety.sh +++ b/scripts/check-archive-v2-safety.sh @@ -301,8 +301,16 @@ const expectedWildcardGuardBody = [ ].join(' '); const expectedWebLogoutBody = 'request, _ in await logout(request, boundary: boundary, sessions: sessions)'; -const expectedWebAliasDeleteBody = - 'request, _ in try await mutate(request, keys: removeKeys) { data in try await surface.removeAlias(try JSONDecoder().decode(EngramServiceWebRemoveAliasRequest.self, from: data)) }'; +const expectedWebAliasDeleteBody = [ + 'request, _ in', + 'guard let token = boundary.sessionToken(in: request),', + 'let actor = await sessions.writeActor(sessionToken: token) else {', + 'return Response(status: .unauthorized) }', + 'return try await mutate(request, keys: removeKeys) { data in', + 'try await surface.removeAlias(', + 'try JSONDecoder().decode(EngramServiceWebRemoveAliasRequest.self, from: attributed(data, actor: actor))', + ') }', +].join(' '); const expectedWebUnlinkDeleteBody = 'request, context in try await mutate(request, keys: unlinkKeys) { _ in try await surface.unlink(try EngramServiceWebUnlinkRequest( sessionId: try pathSessionID(context.parameters.get("id")) )) }'; const expectedWebSuggestionDeleteBody = diff --git a/scripts/db/measure-database.py b/scripts/db/measure-database.py new file mode 100644 index 000000000..a59669e1d --- /dev/null +++ b/scripts/db/measure-database.py @@ -0,0 +1,145 @@ +#!/usr/bin/env python3 +"""Read-only lifecycle measurements; emits aggregate JSON, never transcript text.""" + +import argparse +from contextlib import closing +import json +import math +from pathlib import Path +import sqlite3 +import sys +import time +from datetime import datetime, timezone + + +def measure(database, timeout): + started = time.monotonic() + # Do not use immutable=1: live WAL commits are part of the measurement. + uri = Path(database).resolve().as_uri() + "?mode=ro" + with closing(sqlite3.connect(uri, uri=True, timeout=min(timeout, 5))) as connection: + connection.execute("PRAGMA query_only = ON") + connection.set_progress_handler( + lambda: time.monotonic() - started >= timeout, 1000 + ) + connection.execute("BEGIN") + objects = dict(connection.execute("SELECT name, type FROM sqlite_schema")) + owners = dict(connection.execute("SELECT name, tbl_name FROM sqlite_schema")) + if objects.get("sessions") != "table": + raise ValueError("required sessions table is missing") + page_size = connection.execute("PRAGMA page_size").fetchone()[0] + page_count = connection.execute("PRAGMA page_count").fetchone()[0] + free_pages = connection.execute("PRAGMA freelist_count").fetchone()[0] + storage = {"status": "available", "objects": []} + try: + storage["objects"] = [ + {"name": name, "table": owners.get(name, name), + "type": objects.get(name, "internal"), "bytes": size} + for name, size in connection.execute( + "SELECT name, SUM(pgsize) FROM dbstat GROUP BY name ORDER BY name" + ) + ] + storage["fts_bytes"] = sum( + item["bytes"] for item in storage["objects"] + if item["table"] == "sessions_fts" + or item["table"].startswith("sessions_fts_") + ) + storage["fts_page_share"] = storage["fts_bytes"] / (page_size * page_count) + except sqlite3.OperationalError as error: + if str(error) != "no such table: dbstat": + raise + storage = {"status": "unavailable", "reason": "SQLite has no dbstat module"} + + tiers = [ + {"tier": tier, "sessions": count} + for tier, count in connection.execute( + "SELECT tier, COUNT(*) FROM sessions GROUP BY tier ORDER BY tier" + ) + ] + fts = {"status": "unavailable", "reason": "sessions_fts is missing"} + if objects.get("sessions_fts") == "table": + counts = list(connection.execute(""" + SELECT s.tier, s.id IS NULL AS orphan, COUNT(*) + FROM sessions_fts f LEFT JOIN sessions s ON s.id = f.session_id + GROUP BY s.tier, s.id IS NULL + """)) + total = sum(row[2] for row in counts) + fts = {"status": "available", "rows": total, + "orphan_rows": sum(count for _, orphan, count in counts if orphan)} + by_tier = {tier: count for tier, orphan, count in counts if not orphan} + for item in tiers: + item["fts_rows"] = by_tier.get(item["tier"], 0) + item["fts_row_share"] = item["fts_rows"] / total if total else 0 + + histogram = {} + missing = invalid = 0 + for score, count in connection.execute(""" + SELECT quality_score, COUNT(*) FROM sessions + WHERE tier IS NULL OR tier != 'skip' GROUP BY quality_score + """): + if score is None: + missing += count + elif isinstance(score, int) and 0 <= score <= 100: + histogram[score] = count + else: + invalid += count + scored = sum(histogram.values()) + + def percentile(percent): + rank = math.ceil(scored * percent / 100) + cumulative = 0 + for score in sorted(histogram): + cumulative += histogram[score] + if cumulative >= rank: + return score + return None + + formula_version = None + if objects.get("metadata") == "table": + row = connection.execute( + "SELECT value FROM metadata WHERE key = 'quality_score_formula_version'" + ).fetchone() + formula_version = row[0] if row else None + result = { + "schema_version": 1, + "captured_at": datetime.now(timezone.utc).isoformat(), + "sqlite_version": sqlite3.sqlite_version, + "snapshot": "single read transaction, including committed WAL", + "database": {"page_size": page_size, "page_count": page_count, + "logical_bytes": page_size * page_count, "free_pages": free_pages}, + "storage": storage, + "sessions": {"total": sum(item["sessions"] for item in tiers), "tiers": tiers}, + "fts": fts, + "quality_score": { + "scope": "all non-skip sessions, including hidden and NULL tier", + "formula_version": formula_version, + "scored": scored, "missing": missing, "invalid": invalid, + "min": min(histogram, default=None), "max": max(histogram, default=None), + "mean": sum(score * count for score, count in histogram.items()) / scored if scored else None, + "percentile_method": "nearest rank over valid integer scores 0..100", + "percentiles": {f"p{p}": percentile(p) for p in [10, 33, 50, 67, 90]}, + "histogram": [{"score": score, "sessions": histogram[score]} for score in sorted(histogram)], + }, + } + connection.rollback() + result["elapsed_seconds"] = round(time.monotonic() - started, 3) + return result + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--database", required=True, help="Explicit SQLite path; no default user database") + parser.add_argument("--timeout", type=float, default=30, help="Read budget in seconds (default: 30)") + args = parser.parse_args() + if not math.isfinite(args.timeout) or args.timeout <= 0: + parser.error("--timeout must be a positive finite number") + try: + result = measure(args.database, args.timeout) + except (OSError, ValueError, sqlite3.Error) as error: + print(f"measurement failed: {error}", file=sys.stderr) + return 1 + print(json.dumps(result, indent=2, allow_nan=False)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/scripts/archive-v2-safety-gate.test.ts b/tests/scripts/archive-v2-safety-gate.test.ts index 52fcdf79c..c78a3d399 100644 --- a/tests/scripts/archive-v2-safety-gate.test.ts +++ b/tests/scripts/archive-v2-safety-gate.test.ts @@ -24,8 +24,14 @@ const webLogoutRoute = [ ].join('\n'); const webAliasDeleteRoute = [ 'router.delete("/web/api/settings/aliases") { request, _ in', - ' try await mutate(request, keys: removeKeys) { data in', - ' try await surface.removeAlias(try JSONDecoder().decode(EngramServiceWebRemoveAliasRequest.self, from: data))', + ' guard let token = boundary.sessionToken(in: request),', + ' let actor = await sessions.writeActor(sessionToken: token) else {', + ' return Response(status: .unauthorized)', + ' }', + ' return try await mutate(request, keys: removeKeys) { data in', + ' try await surface.removeAlias(', + ' try JSONDecoder().decode(EngramServiceWebRemoveAliasRequest.self, from: attributed(data, actor: actor))', + ' )', ' }', '}', ].join('\n'); @@ -519,6 +525,36 @@ describe('archive v2 release safety gate', () => { expect(runGate(root).status).toBe(0); }); + it.each([ + [ + 'an unverified actor', + 'await sessions.writeActor(sessionToken: token)', + '"web-editor"', + ], + [ + 'unattributed request data', + 'from: attributed(data, actor: actor)', + 'from: data', + ], + [ + 'an extra archive deletion', + 'return try await mutate(request, keys: removeKeys)', + 'await deleteArchive()\n return try await mutate(request, keys: removeKeys)', + ], + ])('rejects Web alias DELETE with %s', (_name, original, replacement) => { + const root = makeSafeFixture(); + write( + root, + webWritePath, + webAliasDeleteRoute.replace(original, replacement), + ); + const result = runGate(root); + expect(result.status).not.toBe(0); + expect(result.stderr).toContain( + 'Web alias DELETE must be one exact settings wrapper', + ); + }); + it('rejects a Web alias DELETE at the wrong file', () => { const root = makeSafeFixture(); write(root, webAuthPath, webAliasDeleteRoute); diff --git a/tests/scripts/collector-web-ui.test.ts b/tests/scripts/collector-web-ui.test.ts index aa5ed83fb..67ccae494 100644 --- a/tests/scripts/collector-web-ui.test.ts +++ b/tests/scripts/collector-web-ui.test.ts @@ -4928,6 +4928,28 @@ describe('native AI settings editing', () => { expect(ui.node('ai-config-status').textContent).toContain('Reload'); expect(ui.node('ai-config-save').disabled).toBe(true); }); + it('explains an atomic endpoint rejection and permits correcting the draft', async () => { + const ui = await ready(); + ui.node('ai-config-aiBaseURL').value = 'https://other.example/v1'; + const pending = ui.call('saveAISettings()'); + ui.requests[1].resolve(undefined, 409); + await pending; + expect(ui.node('ai-config-status').textContent).toContain('not saved'); + expect(ui.node('ai-config-status').textContent).toContain('server'); + expect(ui.node('ai-config-save').disabled).toBe(false); + expect(ui.node('ai-config-aiBaseURL').value).toBe( + 'https://other.example/v1', + ); + ui.node('ai-config-aiBaseURL').value = settings.aiBaseURL; + ui.node('ai-config-aiModel').value = 'new-model'; + const corrected = ui.call('saveAISettings()'); + expect(JSON.parse(ui.requests[2].body || '{}')).toEqual({ + aiModel: 'new-model', + }); + ui.requests[2].resolve({ settings: { ...settings, aiModel: 'new-model' } }); + await corrected; + expect(ui.node('ai-config-status').textContent).toContain('Saved'); + }); it('clears private prompt text on logout and ignores late responses', async () => { const ui = await ready(); ui.node('ai-config-summaryPrompt').value = 'Private drafting instruction'; diff --git a/tests/scripts/measure-database.test.ts b/tests/scripts/measure-database.test.ts new file mode 100644 index 000000000..2064b45b7 --- /dev/null +++ b/tests/scripts/measure-database.test.ts @@ -0,0 +1,211 @@ +import { execFileSync, spawnSync } from 'node:child_process'; +import { mkdtempSync, readdirSync, readFileSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join, resolve } from 'node:path'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; + +const script = resolve('scripts/db/measure-database.py'); +let root: string; +let database: string; + +function seed(sql: string) { + execFileSync( + 'python3', + [ + '-c', + ` +import sqlite3, sys +with sqlite3.connect(sys.argv[1]) as db: + db.executescript(sys.stdin.read()) +`, + database, + ], + { input: sql }, + ); +} + +function report() { + return JSON.parse( + execFileSync('python3', [script, '--database', database], { + encoding: 'utf8', + }), + ); +} + +beforeEach(() => { + root = mkdtempSync(join(tmpdir(), 'engram-measure-')); + database = join(root, 'fixture ?# 数据.sqlite'); +}); +afterEach(() => rmSync(root, { recursive: true, force: true })); + +describe('read-only database measurements', () => { + it('reports exact distributions, FTS shares and storage without changing the database', () => { + seed(` + CREATE TABLE sessions(id TEXT PRIMARY KEY, tier TEXT, quality_score INTEGER); + CREATE TABLE metadata(key TEXT PRIMARY KEY, value TEXT); + INSERT INTO metadata VALUES('quality_score_formula_version', '1'); + INSERT INTO sessions VALUES + ('a', 'normal', 0), ('b', 'normal', 35), ('c', 'normal', 36), + ('d', 'premium', 60), ('e', NULL, 100), ('missing', 'lite', NULL), + ('bad', 'normal', 'not-a-score'), ('skip', 'skip', 99); + CREATE VIRTUAL TABLE sessions_fts USING fts5(session_id UNINDEXED, content); + CREATE INDEX custom_identity_index ON sessions_fts_content(c0); + INSERT INTO sessions_fts VALUES ('a', 'private transcript marker'), ('a', 'second'), + ('d', 'third'), ('orphan', 'orphan body'); + `); + const before = readFileSync(database); + const files = readdirSync(root); + const result = report(); + expect(result.sessions.total).toBe(8); + expect(result.quality_score).toMatchObject({ + formula_version: '1', + scored: 5, + missing: 1, + invalid: 1, + min: 0, + max: 100, + mean: 46.2, + percentiles: { p10: 0, p33: 35, p50: 36, p67: 60, p90: 100 }, + }); + expect(result.fts).toEqual({ + status: 'available', + rows: 4, + orphan_rows: 1, + }); + expect( + result.sessions.tiers.find( + (item: { tier: string }) => item.tier === 'normal', + ), + ).toMatchObject({ sessions: 4, fts_rows: 2, fts_row_share: 0.5 }); + expect(result.storage.status).toBe('available'); + expect(result.storage.fts_bytes).toBeGreaterThan(0); + const customIndex = result.storage.objects.find( + (item: { name: string }) => item.name === 'custom_identity_index', + ); + expect(customIndex.table).toBe('sessions_fts_content'); + expect(result.storage.fts_bytes).toBe( + result.storage.objects + .filter((item: { table: string }) => + item.table.startsWith('sessions_fts'), + ) + .reduce((sum: number, item: { bytes: number }) => sum + item.bytes, 0), + ); + expect(result.database.logical_bytes).toBe( + result.database.page_count * result.database.page_size, + ); + expect(JSON.stringify(result)).not.toContain('private transcript marker'); + expect(readFileSync(database)).toEqual(before); + expect(readdirSync(root)).toEqual(files); + }); + + it('reports empty scores and a missing FTS table explicitly', () => { + seed( + 'CREATE TABLE sessions(id TEXT PRIMARY KEY, tier TEXT, quality_score INTEGER);', + ); + const result = report(); + expect(result.sessions.total).toBe(0); + expect(result.quality_score).toMatchObject({ + scored: 0, + min: null, + max: null, + mean: null, + }); + expect(Object.values(result.quality_score.percentiles)).toEqual([ + null, + null, + null, + null, + null, + ]); + expect(result.fts.status).toBe('unavailable'); + }); + + it('reads committed WAL content while a writer keeps the WAL open', () => { + const result = JSON.parse( + execFileSync( + 'python3', + [ + '-c', + ` +import sqlite3, subprocess, sys +db = sqlite3.connect(sys.argv[1]) +db.execute('PRAGMA journal_mode=WAL') +db.execute('PRAGMA wal_autocheckpoint=0') +db.execute('CREATE TABLE sessions(id TEXT PRIMARY KEY, tier TEXT, quality_score INTEGER)') +db.execute("INSERT INTO sessions VALUES('wal-only', 'normal', 77)") +db.commit() +print(subprocess.check_output([sys.executable, sys.argv[2], '--database', sys.argv[1]], text=True)) +db.close() +`, + database, + script, + ], + { encoding: 'utf8' }, + ), + ); + expect(result.sessions.total).toBe(1); + expect(result.quality_score.min).toBe(77); + }); + + it('fails without creating missing databases or printing partial JSON', () => { + const missing = spawnSync('python3', [script, '--database', database], { + encoding: 'utf8', + }); + expect(missing.status).not.toBe(0); + expect(missing.stdout).toBe(''); + expect(readdirSync(root)).toEqual([]); + seed('CREATE TABLE unrelated(id INTEGER);'); + const invalid = spawnSync('python3', [script, '--database', database], { + encoding: 'utf8', + }); + expect(invalid.status).not.toBe(0); + expect(invalid.stdout).toBe(''); + expect(invalid.stderr).toContain('required sessions table is missing'); + }); + + it('requires an explicit database and positive bounded timeout', () => { + expect( + spawnSync('python3', [script], { encoding: 'utf8' }).status, + ).not.toBe(0); + for (const timeout of ['0', '-1', 'nan', 'inf']) { + expect( + spawnSync('python3', [ + script, + '--database', + database, + '--timeout', + timeout, + ]).status, + ).not.toBe(0); + } + }); + + it('returns failure without partial JSON when a database is locked', () => { + seed( + 'CREATE TABLE sessions(id TEXT PRIMARY KEY, tier TEXT, quality_score INTEGER);', + ); + const result = JSON.parse( + execFileSync( + 'python3', + [ + '-c', + ` +import json, sqlite3, subprocess, sys +db = sqlite3.connect(sys.argv[1]) +db.execute('BEGIN EXCLUSIVE') +r = subprocess.run([sys.executable, sys.argv[2], '--database', sys.argv[1], '--timeout', '0.05'], capture_output=True, text=True) +print(json.dumps({'code': r.returncode, 'stdout': r.stdout, 'stderr': r.stderr})) +db.rollback() +db.close() +`, + database, + script, + ], + { encoding: 'utf8' }, + ), + ); + expect(result.code).not.toBe(0); + expect(result.stdout).toBe(''); + expect(result.stderr).toContain('database is locked'); + }); +});