Reclaim rebuild-created SQLite free pages (#5070)

* Fix rebuild free-page reclamation (#5057)

* Address adversarial review for Issue #5057

* Correct rebuild memory timeline for Issue #5057

Author: Widthdom

AGENT_GUIDE.md

@@ -138,6 +138,7 @@ CI watching must be bounded. Do not loop indefinitely.
 - `status --json` and related JSON/MCP payloads currently expose the trust fields documented in `README.md` and `DEVELOPER_GUIDE.md`, including `fold_ready`, `fold_ready_reason`, `graph_table_available`, `graph_data_current`, `index_complete`, `index_incomplete_reasons`, `issues_table_available`, `file_issues_data_current`, `migration_in_progress`, `sql_graph_contract_ready`, `sql_graph_contract_degraded_reason`, `hotspot_family_ready`, `hotspot_family_degraded_reason`, `language_readiness`, `csharp_symbol_name_ready`, `csharp_metadata_target_ready`, `csharp_metadata_target_degraded_reason`, `indexed_head_commit`, `workspace_verified_head_sha`, `worktree_head_changed`, `indexed_head_sha`, `indexed_head_branch`, `indexed_head_timestamp`, `commits_ahead_of_indexed_head`, `head_freshness`, `index_writer_version`, `index_newer_than_reader`, `index_newer_than_reader_reason`, `unknown_extension_file_count`, `unknown_extension_files`, `unknown_extension_files_truncated`, `unknown_extension_file_path_limit`, `unknown_extension_extension_counts`, `unknown_extension_category_counts`, `unknown_extension_groups`, `extractors`, `git_executable`, `path_case_sensitive`, `data_dir`, `data_dir_source`, `data_dir_mode`, `db_file_mode`, `database_permission_policy`, `database_permission_diagnostics`, `mac_profile`, `mac_profile_diagnostics`, `db_size_bytes`, `wal_size_bytes`, `db_pragma_settings` (`journal_mode`, `synchronous`, `wal_autocheckpoint`, `busy_timeout_ms`, `page_count`, `freelist_count`, `page_size`, `auto_vacuum`), `prepared_command_cache` (`count`, `capacity`, `hit_count`, `miss_count`, `eviction_count`), `maintenance_guidance`, WAL checkpoint diagnostics (`read_only_fallback`, `wal_checkpoint_attempted`, `wal_checkpoint_succeeded`, `wal_checkpoint_skipped_reason`, `wal_checkpoint_failure_reason`, `wal_checkpoint_busy`, `wal_checkpoint_log_page_count`, `wal_checkpoint_checkpointed_page_count`, `wal_checkpoint_remaining_page_count`, `read_only_immutable_fallback`, `wal_stale_snapshot_risk`, `wal_stale_snapshot_reason`), `symbol_kinds`, `symbols_by_language`, status kind cap metadata (`symbol_kind_limit`, `symbol_kind_name_limit`, `symbol_kind_total_count`, `symbol_kind_omitted_count`, `symbol_kind_names_truncated`, `symbols_by_language_kind_total_counts`, `symbols_by_language_kind_omitted_counts`, `symbols_by_language_kind_names_truncated`), `process`, `last_index_run`, `last_failed_or_partial_index_run`, `last_failed_or_partial_index_run.progress_persisted`, `last_failed_or_partial_index_run.recovery_hint`, `last_failed_or_partial_index_run.file_errors`, `last_workspace_freshened_at`, `hooks`, `hook_diagnostics`, `trust_overrides`, MCP-only `mcp_session`, `mcp.rate_limit.bucket_limit`, `mcp.rate_limit.bucket_limit_rejection_count`, and the `status --check`-only `stale_after_seconds` / `index_age_seconds` threshold audit fields and `repair_commands`.
 - `status --check` repair actions are structured by `name`, `action`, `args`, `mutation_class`, `safety_class`, and `safety_notes`. Preserve compatibility `reason` as the first trigger and ordered `reasons` as the complete trigger set. Deduplicate only exact structured identities; different targets, options, actions, mutation classes, or safety semantics must remain distinct. JSON and human output must use the same deduplicated order. Human output must preserve platform-aware shell quoting, visibly escape control characters to keep each repair action on one diagnostic line, and leave structured JSON `args` unchanged.
 - `maintenance_guidance.fts_optimization` is the shared, read-only recommendation contract for status, explain, optimize preview, and optimize execution. Keep `recommended`, `action`, `reason`, `threshold_writes`, `observed_writes`, and `state` synchronized; stale or unavailable snapshots must not recommend mutation.
+- A successful CLI or MCP `index --rebuild` evaluates the shared freelist warning threshold after the index transaction commits. Incremental-auto-vacuum databases may run bounded `PRAGMA incremental_vacuum`; legacy databases must skip automatic reclaim instead of running a full `VACUUM`. Preserve the immediate index-result and persisted `last_index_run.rebuild_reclaim` telemetry, stable states/reasons, before/after ratios and byte/page counts, and the rule that reclaim failure never reclassifies an already committed index run as failed.
 - A valid CLI `status --stale-after <duration>` implies the workspace check. Check-mode JSON includes `query_context.check_mode` (`explicit` or `implied_by_stale_after`) and `query_context.stale_after_seconds`; ordinary status JSON omits `query_context`.
 - Every bounded `workspace_check` path list (`changed_files`, `missing_files`, `outside_sparse_cone_files`, `unindexed_files`, `unverifiable_files`, and `scan_errors`) carries an authoritative count plus matching `*_truncated`, `*_path_limit`, and `*_omitted_count` fields. List-only `--fields` projections must retain those signals automatically; compact output retains the signals without path arrays; and `--max-json-bytes` may remove only trailing paths while updating both the per-list omitted count and the envelope byte-limit signal. Human check diagnostics must label each displayed list as a sample or complete.
 - `status --explain` derives accepted top-level keys from the same source-generated `StatusResult` serializer metadata as `status --json`, excludes ignored properties, and supports bounded dot-separated member paths without reading runtime values. Major readiness, trust, extension, maintenance, and cap-hit sections return structured meaning, source, dependencies, interpretation, and repair guidance; unknown input is sanitized and returns bounded valid candidates. Bounded status explain envelopes also omit database paths, timings, indexed HEADs, and stable-at timestamps.

DEVELOPER_GUIDE.md

@@ -1217,6 +1217,7 @@ Current stable codes and triggers:
 | Read-only opens and fallback | Query-only commands open with SQLite `Mode=ReadOnly` from the first attempt, retain WAL visibility, and never use writable setup or opportunistic migrations. A write-capable intent may still fall back to read-only when writable journal/WAL setup fails; an explicitly supplied `immutable=1` URI is the opt-in stale-snapshot escape hatch. If a WAL is present and must be observed from storage that cannot expose its sidecars, copy `.db`, `.db-wal`, and `.db-shm` together to a readable location or use a SQLite backup from an environment that can open the full WAL set. |
 | Status pragma diagnostics | `status --json` exposes the selected read-only connection under `sqlite_connection_policy` (`active_mode=read_only`, `open_mode=read_only`) and resolved connection values under `db_pragma_settings` (`journal_mode`, `synchronous`, `wal_autocheckpoint`, `busy_timeout_ms`, `page_count`, `freelist_count`, `page_size`, `auto_vacuum`). It also exposes prepared-command cache counters under `prepared_command_cache` (`count`, `capacity`, `hit_count`, `miss_count`, `eviction_count`) for automation and support diagnostics. `maintenance_guidance` derives `wal_state`, `freelist_ratio`, `freelist_state`, `estimated_*_reclaimable`, `auto_vacuum_mode(_name)`, `recommended_command`, and `post_maintenance_follow_up` from those raw metrics without changing the raw values. Its nested `fts_optimization` uses the same pure evaluator as optimize preview and execution, exposing `recommended`, `action`, `reason`, `threshold_writes`, `observed_writes`, and `state` without writing to the database. `status --check --json` adds structured `repair_commands[]` entries with `name`, `action`, `args`, `mutation_class`, `safety_class`, `safety_notes`, compatibility `reason`, and ordered `reasons`. Exact structured identities are deduplicated and aggregate reasons in check priority order; differences in target, options, action, mutation class, or safety semantics prevent merging. Human check output uses the same command set, preserves platform-aware shell quoting, visibly escapes control characters to keep each `[repair]` action on one diagnostic line, and leaves structured JSON `args` unchanged. `last_failed_or_partial_index_run` exposes bounded failed/partial index context (`status`, `mode`, timings, counts, stable error code, reason, `progress_persisted`, and bounded `recovery_hint`) and must not include raw exception text or file paths. |
 | Maintenance thresholds | WAL guidance flips to `checkpoint_recommended` at `CDIDX_MAINTENANCE_WAL_WARN_BYTES` (default 64 MiB). Freelist guidance flips to `vacuum_recommended` at `CDIDX_MAINTENANCE_FREELIST_WARN_RATIO` (default `0.20`). Invalid or out-of-range env values fall back to defaults. |
+| Post-rebuild reclaim | After a successful CLI or MCP rebuild commits, cdidx evaluates the shared freelist threshold and runs `PRAGMA incremental_vacuum` only when reclaim is recommended and `auto_vacuum=INCREMENTAL`. It never performs an automatic full `VACUUM`; legacy databases report `skipped/auto_vacuum_not_incremental` and retain the explicit `cdidx vacuum` conversion path. Reclaim runs after the index transaction, emits bounded progress/log phases, and persists `last_index_run.rebuild_reclaim` with stable state/reason, duration, before/after logical database sizes and physical main-file samples, page/free-page counts and ratios, and reclaimed page/byte counts. Physical samples can lag while WAL-backed pages await checkpoint; logical sizes and page metrics are the immediate attribution contract. Cancellation, busy/read-only I/O, or another reclaim failure cannot roll back or reclassify the already committed index generation. |
 | Maintenance command precedence | `maintenance_guidance.recommended_command` preserves the existing vacuum-then-checkpoint precedence. It returns `cdidx optimize --db <db>` only when WAL and freelist states are both exactly `ok` and the trusted FTS write snapshot reaches its threshold; an `unknown` higher-priority state or a stale/unavailable FTS snapshot never selects an optimize command. |
 | Page attribution | `status --json` reads SQLite page ownership without mutating the source. It prefers `dbstat` page bytes and otherwise traverses a bounded b-tree/WAL snapshot (at most 1,000,000 pages and 100,000 schema objects); when a live WAL connection is not already backed by a stable detached file set, the fallback first makes a cancellation-aware private backup of that connection's active read snapshot so a concurrent commit cannot mix generations. `allocated_object_bytes + freelist_bytes + unexplained_residual_bytes` equals `logical_database_bytes`; table/index and internal/leaf/overflow/other page subtotals each reconcile to `allocated_object_bytes`. Payload, unused space, and structural overhead form a second reconciliation. Physical main/WAL/SHM bytes are reported separately. Output is capped at 20 object names, each support-sanitized to at most 128 characters. A failed or inconsistent probe returns `available=false`, a stable `unavailable_reason`, and null/omitted attribution values rather than zeros. |
 | Vacuum | `cdidx vacuum` runs `PRAGMA incremental_vacuum` against writable incremental-auto-vacuum DBs, and performs a one-time `PRAGMA auto_vacuum=INCREMENTAL` plus full `VACUUM` conversion for legacy no-autovacuum DBs. `cdidx vacuum --dry-run --json` estimates reclaimable pages/bytes and returns the same maintenance guidance without executing vacuum pragmas. Real `cdidx vacuum --json` also reports before/after DB and WAL byte samples; `wal_checkpoint_timing_note` explains that `wal_size_bytes_after` is measured before connection cleanup, so later `status --json` output may show a smaller WAL after checkpoint/truncation. |
@@ -4851,6 +4852,7 @@ apply 時は `PRAGMA optimize` を実行します。
 | read-only open / fallback | query-only command は最初の試行から SQLite `Mode=ReadOnly` で開き、WAL の可視性を保ちながら writable setup と opportunistic migration を実行しません。write-capable intent は journal/WAL setup に失敗した場合に read-only へ fallback することがあります。明示的な `immutable=1` URI は stale snapshot を許容する opt-in escape hatch です。sidecar を公開できない storage 上の WAL を観測する必要がある場合は、`.db` / `.db-wal` / `.db-shm` をまとめて readable location に copy するか、full WAL set を open できる環境で SQLite backup を使います。 |
 | status pragma diagnostics | `status --json` は選択された read-only connection を `sqlite_connection_policy` (`active_mode=read_only`, `open_mode=read_only`) で、解決済みの接続値を `db_pragma_settings` (`journal_mode`, `synchronous`, `wal_autocheckpoint`, `busy_timeout_ms`, `page_count`, `freelist_count`, `page_size`, `auto_vacuum`) で公開します。また、prepared command cache counter を `prepared_command_cache` (`count`, `capacity`, `hit_count`, `miss_count`, `eviction_count`) で公開します。`maintenance_guidance` は raw 値を変えずに `wal_state`、`freelist_ratio`、`freelist_state`、`estimated_*_reclaimable`、`auto_vacuum_mode(_name)`、`recommended_command`、`post_maintenance_follow_up` を派生します。nested な `fts_optimization` は optimize preview / execution と同じ純粋 evaluator を使い、database に書き込まず `recommended`、`action`、`reason`、`threshold_writes`、`observed_writes`、`state` を公開します。`status --check --json` は `repair_commands[]` に `name`、`action`、`args`、`mutation_class`、`safety_class`、`safety_notes`、互換用の `reason`、順序付きの `reasons` を返します。完全に同一の構造化 identity は deduplicate して check の優先順に reason を集約し、target、option、action、mutation class、安全性 semantics が異なる場合は merge しません。human check output も同じ command set を使い、platform-aware な shell quote を維持し、control character を可視 escape して各 `[repair]` action を1行に保ちます。構造化 JSON の `args` は変更しません。`last_failed_or_partial_index_run` は bounded な failed / partial index context (`status`、`mode`、timing、count、stable error code、reason、`progress_persisted`、bounded な `recovery_hint`) のみを公開し、raw exception text や file path を含めてはいけません。 |
 | maintenance threshold | WAL guidance は `CDIDX_MAINTENANCE_WAL_WARN_BYTES` (既定 64 MiB) 以上で `checkpoint_recommended` になります。freelist guidance は `CDIDX_MAINTENANCE_FREELIST_WARN_RATIO` (既定 `0.20`) 以上で `vacuum_recommended` になります。不正・範囲外の環境変数値は既定値へ戻します。 |
+| rebuild 後の reclaim | CLI / MCP rebuild が正常に commit された後、cdidx は共通の freelist threshold を評価し、reclaim が推奨され、かつ `auto_vacuum=INCREMENTAL` の場合だけ `PRAGMA incremental_vacuum` を実行します。自動の full `VACUUM` は実行せず、legacy database は `skipped/auto_vacuum_not_incremental` を報告して明示的な `cdidx vacuum` conversion path を維持します。reclaim は index transaction の後に実行し、上限付き progress / log phase を出力して、stable な state / reason、duration、before / after の logical database size と物理 main-file sample、page / free-page count と ratio、回収 page / byte 数を `last_index_run.rebuild_reclaim` に保存します。WAL-backed page が checkpoint 待ちの間は物理 sample が遅れて変化する場合があり、即時 attribution の contract は logical size と page metrics です。cancellation、busy / read-only I/O、その他の reclaim failure は commit 済み index generation を rollback したり failure に再分類したりしません。 |
 | maintenance command の優先順位 | `maintenance_guidance.recommended_command` は既存の vacuum、checkpoint の順序を維持します。WAL と freelist の state が両方とも厳密に `ok` で、信頼できる FTS write snapshot が threshold に達した場合だけ `cdidx optimize --db <db>` を返します。上位 state が `unknown` の場合や FTS snapshot が stale / unavailable の場合は optimize command を選択しません。 |
 | page attribution | `status --json` は source を変更せずに SQLite page ownership を読み取ります。`dbstat` page byte を優先し、利用できない場合は件数上限付きの b-tree / WAL snapshot traversal（最大1,000,000 page、100,000 schema object）へ fallback します。live WAL connection が安定した detached file set に基づいていない場合、fallback は先にその connection の active read snapshot を cancellation 対応の private backup に固定し、並行 commit による世代混在を防ぎます。`allocated_object_bytes + freelist_bytes + unexplained_residual_bytes` は `logical_database_bytes` と一致し、table/index と internal/leaf/overflow/other page の小計はそれぞれ `allocated_object_bytes` と一致します。payload、unused space、structural overhead も別に再照合されます。物理 main/WAL/SHM byte は分離して報告します。出力する object 名は最大20件で、各名称は support-safe sanitizer により最大128文字になります。probe の失敗・不整合時は `available=false`、安定した `unavailable_reason`、null / 省略された attribution 値を返し、ゼロとして偽装しません。 |
 | vacuum | `cdidx vacuum` は incremental-auto-vacuum DB では `PRAGMA incremental_vacuum` を実行し、legacy no-autovacuum DB では初回のみ `PRAGMA auto_vacuum=INCREMENTAL` と full `VACUUM` で変換します。`cdidx vacuum --dry-run --json` は vacuum pragma を実行せず、回収可能 page/byte の推定と同じ maintenance guidance を返します。実行系 `cdidx vacuum --json` は DB / WAL byte の before / after sample も返します。`wal_checkpoint_timing_note` は `wal_size_bytes_after` が connection cleanup 前の計測であり、checkpoint / truncation 後の `status --json` では WAL が小さく見える場合があることを示します。 |