Skip to content

Improve search relevance and harden collector and Web workflows - #496

Draft
bbingz wants to merge 24 commits into
mainfrom
codex/leader-loop-20261023
Draft

bbingz wants to merge 24 commits into
mainfrom
codex/leader-loop-20261023

Conversation

@bbingz

@bbingz bbingz commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

Collector retries, transcript paging, Web writes and human search had gaps in scheduling, ownership, authorization/auditing and ranking. This branch closes the reproduced gaps and exposes existing session value scores through badges and filters.

Changes:

  • Back off blocked collector roots; align MCP discovery with runtime roles; fix collector test fixture races/cleanup.
  • Bind transcript paging cleanup to its task and prevent duplicate scheduling.
  • Give popover date groups unique stable identities when resumed sessions interleave the same date.
  • Preserve Web project actors through IPC, audit alias changes transactionally, and reject endpoint edits that would reuse saved AI keys.
  • Rank all keyword terms consistently across App and Service, align CJK routing, and add existing value-band badges/filters.
  • Reduce measured SHA-256 formatting cost and add an explicit-path, read-only database measurement CLI.
  • Reconcile canonical roadmap/TODO/followups against current source. Product decisions and host acceptance remain open.

Validation (local, per-batch receipts; not a cumulative full-suite run):

  • Behavioral failing regressions followed by focused passing App, Service, collector, MCP and HTTP tests; exact commands and counts are in CHANGELOG.md.
  • Most recent search/value receipts: 193 App tests, 66 Service tests, 20 final render tests; bilingual component/menu renders inspected.
  • Database measurement CLI: 6 tests, including committed WAL and lock timeout; copied repository fixture readback.
  • Relevant invariant/module/convention gates, lint and XcodeGen drift checks passed in the affected batches; baseline lint warnings remain documented.
  • Final diff whitespace check passed. Deployment evidence and its limits are recorded below.
  • Archive safety gate: updated the exact alias DELETE wrapper to require session-derived attribution; all 65 gate tests pass, including rejection of an unverified actor, raw unattributed data and extra archive deletion.

Hosted validation of deployed App source a9e59d50f3ef8563de956836ea9fe608ac0c2776 (2026-10-25):

  • Tests / CI Gate: SUCCESS, including Node quality/tests, macOS script/fixture gates, Swift unit tests, remote-server tests and UI smoke.
  • CodeQL / CodeQL Gate: SUCCESS for TypeScript, Swift product and Swift remote server.
  • Dependency Review: SUCCESS.
  • All 14 executed checks succeeded; two jobs were conditionally skipped. PR was CLEAN before installation.
  • UI smoke: 14 tests and screenshot comparison passed. Popover regression: old code failed, then all 46 local HomePopoverActionsTests passed; a real isolated A/B/A fixture emitted the duplicate-ID warning on build 1779 and none during 92 seconds on the signed build 1782.

Compatibility / limits:

  • Upgrade Service before RemoteServer for the Web changes, and upgrade App/Service together for value-band filtering (old Service ignores the additive field).
  • Includes an additive Web alias audit migration; existing migration and audit rollback tests passed.
  • Foreground UI fault injection was blocked by LocalAuthentication before test execution. Full all-target tests, production latency/score distribution, collector active-write/startup performance and per-source cutover acceptance remain unverified. The real-host quiet-source CPU/RSS window passed as recorded below.
  • The digest optimization is a synthetic-path measurement, not proof of the whole-process CPU target.
  • Cosine insight supersession, score-formula changes, manual overrides and other product decisions remain in docs/roadmap.md.

Review routing: docs/reviews/2026-10-23-development-reconciliation.md.

Deployment status (2026-10-25)

Installed and launched App 1.0.5 build 1782 from exact green a9e59d50 after both CI and CodeQL gates passed. All 80 installed bundle entries matched the signed candidate; App PID 80289, local IPC status and both HQ health endpoints passed readback. The old build 1779 is preserved for App-only rollback. This closes the duplicate popover DateGroup ID residual.

HQ index/Web, Daily Collector and the paired local Service remain on previously deployed fb9538a. The local Service stayed at PID 21750 during this App-only update. Earlier value-band/served-Web-byte probes passed; Daily converged 16/16 roots and continued uploads. The collector quiet-source CPU/RSS gate now passed; active-write/startup performance, full local UI/business-flow acceptance and an App rollback drill remain unverified. Local screenshot capture lacked Screen Recording permission; hosted UI smoke and its screenshot comparisons passed.

The release bundle was signed with the existing Developer ID identity on Daily through a one-shot GUI job, then the job was unloaded. Full Developer ID/deep signature/timestamp checks passed, and normalized comparison proves signing preserved all eight executable/framework code files. No key export or security-policy change.

Backup and detailed receipts: ~/.engram-shadow-core-20260911/hq/state/deploy-a9e59d50-20261025/, documented in CHANGELOG/MEMO/TODO/followups. Documentation-only closeout 148bfbb5 also passed all 14 executed checks. The deployed App source remains the already-green a9e59d50. PR is Draft and unmerged.

Daily resource acceptance (2026-10-25)

Documentation commit db601947 records a completed, read-only 30-minute Daily window on the existing fb9538a collector. Independent recomputation of 61 samples over 1800 seconds gives mean one-core CPU 1.921% (<=2%, PASS) and maximum sampled RSS 58.5 MiB (<=150 MiB, PASS). All 16 roots stayed converged; PID/start time, binary hash and log sizes stayed stable.

There were no root revisions or new publication ACKs, so this closes only the missing quiet-source window. Active-write and startup costs remain unverified; the CPU margin is small. All 603 pending publications per replica were privacy-withheld. No code, configuration, source data or service was changed. Evidence and residual details are in CHANGELOG, TODO, followups and the source-retirement checklist. That documentation head's Swift unit job failed two timing-sensitive assertions; the scoped test correction is recorded below.

Swift CI timing correction (2026-10-25)

Current head 1379562c242488cacc1fc8f37a1e7fdbeed35697 changes tests and records only; deployed product source is unchanged. The digest regression keeps its 26 MiB input, tail-marker assertions and 10-second limit, measuring process CPU time instead of wall time. The launcher stop test waits for the existing SIGTERM-ignoring helper to be ready; related restart fixtures have a 5-second readiness budget while retaining shutdown-order assertions.

All 70 related tests passed in an isolated home, including a final run after negative controls. Disabling the ASCII search fast path failed at 26.799 CPU seconds; inserting a blocking stop wait failed at 2.002 seconds. Runtime source was then restored byte for byte. Swift conventions, invariant ledger and whitespace checks passed. Original hosted failure: Tests run 40174238672.

Hosted validation for exact head 1379562c242488cacc1fc8f37a1e7fdbeed35697 is complete: all 14 executed checks succeeded, two jobs were conditionally skipped, and PR readback is CLEAN. Tests / CI Gate, CodeQL / CodeQL Gate and Dependency Review succeeded. The four affected timing tests all executed and passed in the hosted Swift job. Final 08:51 UTC readback confirmed App build 1782, unchanged Service PID 21750, local IPC and both HQ health endpoints. No further deployment was needed for this tests-and-records correction. PR remains Draft and unmerged. Full local UI/business-flow acceptance, active-write/startup collector performance and rollback drills remain unverified. Detailed receipts are in CHANGELOG and ~/.engram-shadow-core-20260911/hq/state/ci-timing-20261025/.

Copilot deferral diagnosis (2026-10-25)

Read-only Daily inspection at 08:56 UTC resolved the previously unverified 255 Copilot entries: all are identical 172-byte checkpoint indexes with no valid rows and no sibling events file. The existing 60-second retry is expected while no entrypoint is selectable. Four focused native tests passed for empty-index deferral, no false ACK, grouped retry and valid checkpoint publication to both fixture replicas. No runtime source, live settings, inventory records or processes changed; this does not establish historical source completeness or the CPU cost of retries. The aggregate receipt is at ~/.engram-shadow-core-20260911/hq/state/copilot-deferrals-20261025/classification.json. Local CHANGELOG/MEMO/followups were reconciled; the hosted head remains the verified 1379562c.

reviewer added 24 commits October 23, 2026 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant