Skip to content

docs: close out the 2026-10-18 open-item cleanup and the D7 apply #479

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bbingz merged 3 commits into from
Oct 18, 2026
Merged

docs: close out the 2026-10-18 open-item cleanup and the D7 apply #479

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5076803
docs: close out the 2026-10-18 open-item cleanup (W6.2 merge SHA pend…
Oct 18, 2026
120bc7f
docs: record the D7 purge apply and the HQ service-index move to 9ef7…
Oct 18, 2026
c19ac1c
docs: fill the #475 merge SHA and final hosted gate result
Oct 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
187 changes: 187 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,193 @@ All notable changes to this project will be documented in this file.
Format based on [Keep a Changelog](https://keepachangelog.com/).


## D7 legacy grok/pi purge applied on HQ; service-index moved to main `9ef7fb4f` (2026-10-18)

Owner authorization, in order: "批准" for the D7 dry-run report, then
"如果 dry-run 没问题，就批准部署和 apply" (deploy and apply if the dry run is
clean). The offline report and the live preview were both clean, so both steps
ran.

**Deploy.** Release/arm64 built from a detached worktree at `9ef7fb4f`
(`main` with #476-#478; no schema migration since `b95d99ad`), packaged with
`macos/scripts/package-service.sh` after `--dry-run`; `--verify-only` PASS in
scratch and again at the destination, `codesign --verify --deep --strict` ok.
`bin/EngramService` sha256
`2714e296484802ddda31f732a7e1ecade325239a2e63a7e21b5c2b3d86c114a2`,
BUILD-METADATA sourceRevision `9ef7fb4f01c60e40406bfcd864add09127a7170e`.
Package `hq/service-index-main-9ef7fb4f-20261018`; plist backup
`state/service-index/persistent/main-9ef7fb4f-20261018-job-before.plist`; only
`ProgramArguments[0]` changed; bootout 05:13:31Z, bootstrap 05:13:34Z, new
PID 625 logged the listening line, stderr stayed at 0 bytes, receiver
`127.0.0.1:18787` and tailnet Web `/v1/health` 200. Rollback: restore the
backed-up plist and bootstrap it; `service-index-main-b95d99ad-20261017` is
kept. The operator CLI is `EngramCLI` Release from the same build (sha256
`6670a54ee4b5892eaea8703dc9fe5511b33e36cb204b1cfed2c400e51b59d094`), run from
DerivedData, not installed anywhere.

**Pre-apply checks.** Live `EngramCLI maintenance legacy-local-purge preview`
through the new service (16 s) reproduced the approved values exactly:
`sessionCount` 6,191, `planToken`
`9fa54d7d1ab87778faf42d525076f8544b322207313ed06b4bb916f9d8890b8f`, every
blocking dependent 0. Rollback copy taken right before the apply with
`sqlite3 -readonly SRC "VACUUM INTO ..."` (05:14:31-05:16:16Z, 12.8 GB,
quick_check ok, 45,815 sessions, 6,191 candidates) at
`state/service-index/persistent/d7-rollback-20261018/index.sqlite`. Restore
path: bootout, replace the database file and remove its `-wal`/`-shm`,
bootstrap; anything ingested after 05:14Z would have to be re-delivered.

**Apply.** `apply --expect-sessions 6191 --expect-token 9fa54d7d…` ran
05:17:15-05:17:45Z (30 s, inside the token-protected 10-minute client
budget): `sessionsDeleted` 6,191; derived rows deleted sessionsFts 51,937,
ftsMap 51,937, session_index_jobs 11,578, session_tools 23,232,
session_costs 6,191, session_work_beats 531, everything else 0; audit row
`legacy_local_purge.v1.2026-10-18T05:17:15Z` in `metadata`.

**Verified independently.** Re-preview: `sessionCount` 0. Read-only SQL on
the live index: candidate predicate 0, `origin=local` rows of any source 0,
capture-bound grok 866 and pi 573 unchanged, no orphan `fts_map` or
`session_index_jobs` rows, 39,626 sessions, ledger pending 0, audit row
present. Service PID 625 still up, stderr 0 bytes, receiver and Web 200.
Freed pages stay in the file until an owner-scheduled vacuum; that is
outside D7. This closes the D7 TODO item.

Note: a pre-apply check suspected the CLI's IPC timeout (30 s default) was
too short for a 13 GB index; a throwaway repro test showed token-protected
commands already get the 10-minute `mutatingCommandTimeout`, so no change
was made.

## Open-item cleanup by Opus workflows: four TODO items shipped, Daily CPU measured, two items need owner decisions (2026-10-18)

Owner request: "继续组织opus清理开放项" (keep orchestrating Opus to clear the open
items). Two Workflow runs with Opus agents, orchestrated from this session:
a read-only scoping pass (7 agents, one per `docs/TODO.md` open item, 11 min)
and an implementation pass (15 agents: 4 implementers in isolated worktrees
off `4b84e3d5`, a spec/invariants reviewer plus an independent test executor
per item, one fix round for D7, 67 min). The orchestrator read every diff,
pushed the branches and opened the PRs; nothing was deployed and no host was
changed.

**Shipped (PRs opened from this session; merged in the order #477, #476, #478, #475, each after its required checks passed on a branch updated to the then-current `main`):**

- PR #475 `5b5b09af` — W6.2 real-binary chain gate.
`.github/workflows/real-binary.yml` (weekly, `workflow_dispatch`, and PRs
touching the gate; **not** a required check, `CI Gate` contract unchanged),
`scripts/ci/run-real-binary-gate.sh` (builds Debug EngramCollector /
EngramService / EngramRemoteServer, runs `CollectorBinaryShadowIntegrationTests`,
`CollectorCLIIntegrationTests` with the env vars set, writes a receipt) and
`scripts/ci/check-real-binary-results.py`, which fails unless the exact
expected test set executed (a green run with 50 skips is a fail; proven
locally: `result=fail expected=50 passed=0 skipped=50`). Local full run:
`result=pass expected=50 passed=49 skipped=1` (browser demo skipped by
design). Hosted runs on the PR: every run executed all 50 tests and only
the 114 MB large-history test ever failed. Run 1 hit its fixed 240 s scope
deadline, run 2 the fixture's 0.5 s replica request timeout, run 3
(39093667839) passed with that test at 429.6 s, run 4 (39097383289) hit the
3x deadline with HQ ingest still running after 12 min. The runner's speed
varies about 2x between runs, so the merged gate scales the scope deadlines
and the fixture's replica HTTP timeouts by `--timeout-scale 6` with a
60-minute job budget; assertions and the expected test set are unchanged.
Final run on the merged head (39098816343, 10f5ab21): success.
- PR #476 `7a48d19d` — App "Open Web Reader" action in `collector`/`replica`
roles, driven by a new optional `webReaderURL` key in `settings.json`
(https only, host required, no userinfo/query/fragment), disabled with a
caption when unset; the blocked-role branch still never opens the DB or
spawns a Service. 5 new `RuntimeRoleAppTests`. The per-host settings edit is
a later operator step.
- PR #477 `9f131f25` — W4.7 HQ half: `collectorEpochDryRun` (writer-gate
read, peer-UID gated, index role only, also lists observed-but-unapproved
epochs per instance) and `collectorApproveEpoch` (writer-gate write,
capability-token protected, exact expected epoch + authority generation,
role and enabled source re-checked inside the write), reached through
`EngramCLI collector-epoch dry-run|approve`; not on any Web allowlist, not on
the client protocol or mock. Design
`docs/superpowers/specs/2026-10-18-hq-collector-epoch-approval-design.md`,
ledger entry "Collector Epoch Approval Is A Local Operator Action", repro
`testUnapprovedEpochBecomesIngestibleAfterOperatorApproval_repro`.
Finding recorded as followup `collector-epoch-producer-1`: the collector
never rotates an epoch under an existing source instance
(`CollectorInventoryStore` mints fresh instance and epoch UUIDs per root
revision), so a spool reset arrives at HQ as a new instance on a registered
root and would hit `overlappingRoot` in provisioning; the new command covers
the design's same-instance case, the producer side is an owner decision.
- PR #478 `9ef7fb4f` — D7 option (c) tooling: `LegacyLocalSessionPurge`
(candidate predicate for legacy `origin=local` grok/pi rows that are not
capture-bound, blocking user-state counts, `planToken` over the sorted IDs,
one-transaction purge that refuses on plan change or any blocking dependent,
explicit FTS cleanup, audit metadata row), IPC `legacyLocalPurgePreview` /
`legacyLocalPurgeApply` (token protected, index role only), the offline
`EngramService --legacy-local-purge-report --database-path <copy>` for a
report from a backup copy, and `EngramCLI maintenance legacy-local-purge
preview|apply --expect-sessions --expect-token`. Design
`docs/superpowers/specs/2026-10-18-d7-legacy-local-origin-purge-design.md`
(Rollout holds the operator runbook), ledger entry "Legacy Local-Origin
Purge Is Dry-Run First And Dependent-Free". Dry-run report produced from
a read-only `VACUUM INTO` snapshot of the HQ index (02:24Z, 13.4 GB,
quick_check ok, 45,764 sessions), first with the branch's Debug binary and
again with one built from merged `main` `9ef7fb4f` (byte-identical JSON),
and cross-checked with plain SQL: 6,191 candidates (grok 5,959, newest
indexed_at 2026-10-07T11:20Z; pi 232, newest 2026-09-11), every blocking
dependent 0 (insights, outside parent/suggested links, local state,
favorites, tags, offloaded), `purgeAllowed` true, planToken
`9fa54d7d1ab87778faf42d525076f8544b322207313ed06b4bb916f9d8890b8f`
(identical to the SQL recomputation); derived rows ftsMap/sessionsFts
51,937, session_index_jobs 11,578, session_tools 23,232, session_costs
6,191, work beats 531; retained file_index_state 6,191; the 866 grok and
573 pi capture-bound rows are outside the predicate. The candidate count
has not grown since 2026-10-07, so the HQ role gate holds. The owner
approved this report on 2026-10-18 and then authorized the deploy and apply;
both are recorded in the entry above.

Verification per PR is in each PR description (implementer plus independent
re-run with the test.yml HOME isolation; verifiers also ran mutation checks,
for example removing the capture-binding exclusion or the index-role gate made
the named tests fail). All four PRs passed the required checks before merge.

**Daily collector CPU window (plan W6 item 3), read-only, not passed.** The
scoping agent found that the TODO's lead (the every-second Cursor peer
fingerprint scan) had already been mitigated on 2026-09-12 by the 30 s hint
cache and the 10 s gate, both present in the build that measured 13.178%, so
no code was changed; the done criterion is a host measurement. Window
2026-10-17T23:49:19Z to 2026-10-18T00:19:22Z on Daily (PID 96925, binary
`b83ada40…5c01` from the 2026-10-17 deployment, `pollIntervalMilliseconds`
10000, 16 roots, process uptime 10 h): 61 samples, wall 1803 s, **CPU 5.891%
of one core (target <= 2%: FAIL)**, max RSS 81.2 MiB (target <= 150 MiB:
PASS), PID stable, all 16 roots requested_revision == completed_revision
before and after (no re-walk). Per 30 s interval: 10 below 2%, 29 at 2-5%,
21 at or above 5%; quiet floor 0.45-1.0 s CPU per 30 s (1.5-3.5%), bursts of
3-9 s. Workload: 37 publications ACKed by both replicas (46,782 -> 46,819),
35 of them from the `daily-claude` root; the host was in active use (load
average 3.6 to 13, Spotlight `mds_stores` at 47% at the end). Discriminating
diagnostics all negative: `cursor_legacy_observer_error` 0 before and after
(FIX A not indicated), modern Cursor root revision unchanged (FIX B not
indicated), `~/.cursor` dirents 963 < 2048 (discoverModern budget fine);
`sample` (D4) not run because it attaches to the process and writes on Daily.
Reading: the cost is a polling floor near the target plus bursts tied to
Claude Code transcript appends; the next measured lead is the per-change
capture cost of large appended transcripts, unprofiled. Raw samples and the
receipt are in the session scratchpad; the checklist, TODO and followups carry
the summary. The item stays open.

**Needs owner decisions (scoped, not built):**

- W4.1 alias reconciliation: D7 chose the purge over the alias for the only
legacy rows in the central DB, D9 accepts the history gap, and HQ's legacy
Claude Code / Codex user state lives in the separate old local DB that an
alias inside the central DB cannot reach. Options: (a) close as superseded
by D7(c)/D9; (b) build as designed (about 800-1,100 lines, schema change,
design doc first) for a future local-role host that also ingests its own
collector; (c) re-scope as an old-DB user-state import. Also unverified:
whether the archive-v2 catalog is the accepted byte proof for any legacy row.
- W4.5 old-receipt bootstrap: D9 recorded it as "a later decision"; nothing
since reopens it. Building it needs W4.1 or an explicit duplicate rule and a
provenance change to `capture_ingest_generations` (publication sha and
sequence are NOT NULL), size XL. Options: keep D9 and mark won't-do, or
reopen with a design doc.

Not in scope of this cleanup and unchanged: `cutover-hq-local-gap-1` (needs
owner credential and root steps on HQ), `collector-epoch-change-scope-1`
(owner decision), the untracked `.grok/workflows/engram-session-status.rhai`.

## Collector reconcile command deployed to Daily; HQ service-index moved to main with P2 (2026-10-17)

Owner authorization: "同意，按顺序来" for the three-step plan (recover Daily,
Expand Down
11 changes: 11 additions & 0 deletions MEMO.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,17 @@

## Changelog Memo

### 2026-10-18

- [变更] 业主授权（「批准」报告，随后「如果 dry-run 没问题，就批准部署和 apply」）后：HQ service-index 从 `b95d99ad` 换到 main `9ef7fb4f`（PID 625，stderr 0 字节，receiver/Web 200，plist 备份与旧包保留可回滚）；live preview 复现 6,191 / `9fa54d7d…`，`VACUUM INTO` 做了 12.8 GB 回滚副本，然后 apply：删除 6,191 个旧 grok/pi 会话及派生行（FTS 51,937、index jobs 11,578、tools 23,232 等），审计键 `legacy_local_purge.v1.2026-10-18T05:17:15Z`。详见 `CHANGELOG.md`。
- [验证] re-preview 为 0；只读 SQL：候选 0、全库 `origin=local` 0、capture 绑定的 grok 866 / pi 573 未动、无孤儿派生行、账本 pending 0。D7 关闭。
- [变更] 业主要求继续用 Opus 清理开放项：两轮 Workflow（7 个只读 scoping agent、15 个实现/复核 agent），4 项落地为 PR 并按序合并：#475 `5b5b09af` W6.2 真实二进制门禁（非必需工作流 + 校验脚本，跳过即失败）；#476 `7a48d19d` App 在 collector/replica 角色下的「Open Web Reader」入口（新可选键 `webReaderURL`，仅 https）；#477 `9f131f25` W4.7 HQ 侧 `collectorEpochDryRun`/`collectorApproveEpoch` IPC + `EngramCLI collector-epoch`（令牌保护、不进 Web）；#478 `9ef7fb4f` D7 旧 `origin=local` grok/pi 清理的 dry-run/apply 工具与离线报告模式。均未部署，未改任何主机。详见 `CHANGELOG.md`。
- [验证] 每个 PR 由实现者与独立复核者各跑一遍测试（ServiceCore 用 CI 的 HOME 隔离），复核者另做了变异验证（去掉关键守卫后对应测试确实失败）；PR 的 required CI 全绿后才合并。#475 自身触发的托管 `Real-binary chain` 结果：第三次运行通过（run 39093667839，头 eb62c039）：`result=pass expected=50 passed=49 skipped=1 failed=0`，大历史测试 429.6 秒；四次运行都只在该测试上失败过（240 秒期限、0.5 秒副本请求超时、3 倍期限），托管机速度两次之间差约 2 倍，合并版用 `--timeout-scale 6` 与 60 分钟作业预算（只拉长期限，不改断言）。
- [排查] Daily collector 30 分钟只读测量（10-17 23:49Z 起，PID 96925，`b83ada40…`，poll 10 s，16 根全程收敛）：CPU 5.891%（目标 ≤ 2%，未达标）、最大 RSS 81.2 MiB（达标）；安静段地板 1.5% 到 3.5%，峰值与 35 条 Claude Code 转写发布对应；TODO 里的 Cursor peer scan 线索 9-12 已缓解，D1 到 D3 判别诊断均为阴性。新线索：大转写文件每次追加的捕获成本，未剖析（`sample` 需业主批准）。该项保持开放。
- [新增] followup `collector-epoch-producer-1`：collector 从不在同一 source instance 下轮换 epoch，spool 重置到 HQ 表现为新 instance，provisioning 会报 `overlappingRoot`；W4.7 新命令只覆盖设计假设的同 instance 场景，生产者侧待业主决定。
- [验证] D7 dry-run 报告已从 HQ 库只读快照（02:24Z，`VACUUM INTO`，13.4 GB，quick_check ok）生成（合并后 main `9ef7fb4f` 的二进制重跑，JSON 逐字节一致）并经独立 SQL 交叉核对：候选 6,191 行（grok 5,959、最新 indexed_at 10-07；pi 232），阻塞依赖全 0，`purgeAllowed=true`，planToken `9fa54d7d…90b8f`；capture 绑定的 grok 866 / pi 573 行正确排除；候选数自 10-07 起未增长。业主已于 10-18 批准该报告并授权部署与 apply（见本节首条）。
- [未验证] W4.1 别名对账、W4.5 旧回执引导需要业主决策（关闭 / 按设计建 / 重定范围），详见 `CHANGELOG.md`。

### 2026-10-17

- [变更] #473 合并为 `b95d99ad` 后，用同一提交的 Release/arm64 构建打包并部署：HQ 的 service-index 从 10-02 的 `8a6cdf6c` 换到 main（启动迁移建出 P2 索引，PID 20105，stderr 0 字节，账本无积压，`origin=local` 6,191 不变）；Daily 的 collector 换到带 `--reconcile-native-epoch` 的新包（PID 96925，一次启动无错误）。两处都只改 plist 的 `ProgramArguments[0]`，旧包与 plist 备份保留可回滚。详见 `CHANGELOG.md`。
Expand Down
Loading