Merge pull request #2748 from Widthdom/fix-issue1718-1721

Author: Widthdom

COMPATIBILITY.md

@@ -0,0 +1,127 @@
+# CodeIndex DB Compatibility
+
+> **[日本語版はこちら / Japanese version](#codeindex-db-compatibility日本語)**
+
+This document defines the compatibility contract between `cdidx` binaries and
+the local SQLite database under `.cdidx/codeindex.db`.
+
+## Supported Readers
+
+The public compatibility boundary is the `cdidx` CLI and MCP server reading a
+database created by a released `cdidx` binary. The SQLite schema is an internal
+storage format, not a public API.
+
+Within a supported release line, newer binaries must read older databases and
+degrade optional features explicitly when stored readiness metadata is missing
+or stale. Older binaries may read newer databases only when the newer database
+does not advertise unknown readiness or contract stamps. If an older binary sees
+unknown persisted contract stamps, it must degrade loudly in `status` output and
+must refuse writes that could silently discard newer data.
+
+## Schema and Readiness Stamps
+
+`PRAGMA user_version` is a readiness bitmap, not a linear migration number:
+
+| Bit | Field | Meaning |
+|---|---|---|
+| `1` | `graph_table_available` / graph readiness | `symbol_references` has been fully populated for graph queries. |
+| `2` | `issues_table_available` / issue readiness | `file_issues` has been populated for validation results. |
+| `4` | `fold_ready` | Folded-name columns are current for Unicode-aware exact-name matching. |
+
+Additional per-feature contract versions live in `codeindex_meta`, including
+folded-key metadata, C# symbol-name and metadata-target versions, SQL graph
+contract stamps, hotspot-family readiness, index writer version, indexed HEAD
+metadata, unknown-extension counts, filesystem case-sensitivity, MAC profile,
+and DB/WAL/status diagnostics. These stamps let readers distinguish a feature
+that is absent, stale, or newer than the running binary.
+
+## Version Skew Behavior
+
+Use `cdidx status --json` or `cdidx status --check --json` before relying on a
+database across binary upgrades or downgrades.
+
+| Scenario | Expected behavior | Operator action |
+|---|---|---|
+| New binary reads an older DB | Queries continue where possible. Missing readiness fields report degraded status and include remediation strings. | Run the recommended maintenance command from `status`, usually `cdidx backfill-fold` or `cdidx index <projectPath> --rebuild`. |
+| Same binary reads its own DB | `status --check --json` reports `index_matches_workspace: true` when file content and HEAD metadata match. | No rebuild required. |
+| Older binary reads a newer DB | `index_newer_than_reader` becomes `true` when unknown readiness bits or contract stamps exceed the binary's maximum. Mutating commands refuse to write unsafe newer DBs. | Use the newer `cdidx` binary that wrote the DB, or rebuild the index with the older binary only after accepting loss of newer feature data. |
+| Read-only CI artifact | Query commands may use `--read-only` / `--immutable`. Mutating commands reject read-only DBs. | Pin the `cdidx` binary version with the DB artifact when possible. |
+
+## Rebuild Requirements
+
+Additive schema changes should be readable by newer binaries without requiring a
+full rebuild. Prefer in-place maintenance for derived data, such as
+`cdidx backfill-fold`, when a feature can be refreshed from existing rows.
+
+A rebuild is required when:
+
+- `status` recommends `cdidx index <projectPath> --rebuild`;
+- the workspace and DB are intentionally being reset to an older binary version;
+- the database is corrupt or fails `cdidx db --integrity-check`;
+- a release note explicitly calls out a breaking storage change.
+
+Breaking DB changes must be rare and must document the minimum binary version,
+the downgrade behavior, and the rebuild path in release notes.
+
+## CodeIndex DB Compatibility（日本語）
+
+この文書は、`cdidx` binary と `.cdidx/codeindex.db` のローカル SQLite database
+の互換性契約を定義します。
+
+## 対応する reader
+
+公開される互換性境界は、release 済み `cdidx` binary が作成した database を
+`cdidx` CLI / MCP server が読むことです。SQLite schema は内部 storage format
+であり、公開 API ではありません。
+
+対応 release line 内では、新しい binary は古い database を読み、保存済みの
+readiness metadata が不足または stale の場合は optional feature を明示的に
+degrade しなければなりません。古い binary が新しい database を読めるのは、
+その database が未知の readiness / contract stamp を示していない場合だけです。
+未知の永続 contract stamp を見た古い binary は `status` で明示的に degrade を
+報告し、新しい data を黙って破棄しうる write を拒否します。
+
+## Schema と readiness stamp
+
+`PRAGMA user_version` は線形 migration number ではなく readiness bitmap です。
+
+| Bit | Field | 意味 |
+|---|---|---|
+| `1` | `graph_table_available` / graph readiness | graph query 用の `symbol_references` が完全に作成済み。 |
+| `2` | `issues_table_available` / issue readiness | validation result 用の `file_issues` が作成済み。 |
+| `4` | `fold_ready` | Unicode-aware exact-name matching 用の folded-name column が最新。 |
+
+追加の feature contract version は `codeindex_meta` に保存されます。これには
+folded-key metadata、C# symbol-name / metadata-target version、SQL graph
+contract stamp、hotspot-family readiness、index writer version、indexed HEAD
+metadata、unknown-extension count、filesystem case-sensitivity、MAC profile、
+DB/WAL/status diagnostics が含まれます。reader はこれらの stamp により、feature
+が存在しないのか、stale なのか、実行中 binary より新しいのかを判別できます。
+
+## Version skew 時の動作
+
+binary upgrade / downgrade をまたいで database を使う前に、
+`cdidx status --json` または `cdidx status --check --json` を確認してください。
+
+| 状況 | 期待される動作 | 操作者の対応 |
+|---|---|---|
+| 新しい binary が古い DB を読む | 可能な query は継続します。不足した readiness field は degraded status と remediation を返します。 | `status` の推奨に従い、通常は `cdidx backfill-fold` または `cdidx index <projectPath> --rebuild` を実行します。 |
+| 同じ binary が自身の DB を読む | file content と HEAD metadata が一致すると `status --check --json` は `index_matches_workspace: true` を返します。 | rebuild は不要です。 |
+| 古い binary が新しい DB を読む | 未知の readiness bit または contract stamp が binary の最大値を超えると `index_newer_than_reader` が `true` になります。mutating command は unsafe な write を拒否します。 | その DB を書いた新しい `cdidx` binary を使うか、新しい feature data が失われることを受け入れて古い binary で index を作り直します。 |
+| read-only CI artifact | query command は `--read-only` / `--immutable` を利用できます。mutating command は read-only DB を拒否します。 | 可能なら DB artifact と `cdidx` binary version を一緒に pin します。 |
+
+## Rebuild が必要な場合
+
+Additive schema change は、full rebuild を要求せずに新しい binary で読めるべきです。
+既存 row から再生成できる derived data は、`cdidx backfill-fold` のような
+in-place maintenance を優先します。
+
+rebuild が必要なのは次の場合です。
+
+- `status` が `cdidx index <projectPath> --rebuild` を推奨している;
+- workspace と DB を意図的に古い binary version へ戻す;
+- database が壊れている、または `cdidx db --integrity-check` に失敗する;
+- release note が breaking storage change を明示している。
+
+Breaking DB change は稀であるべきで、minimum binary version、downgrade behavior、
+rebuild path を release note に記載しなければなりません。

DEVELOPER_GUIDE.md

@@ -61,6 +61,18 @@ cdidx search AuthService --db /artifacts/codeindex.db --immutable
 Mutating commands such as `index`, `backfill-fold`, `optimize`, and `vacuum`
 require writable storage and reject read-only database opens.
 
+Use `cdidx db checkpoint <name>` to take a filesystem snapshot of
+`codeindex.db` plus existing WAL/SHM sidecars before risky maintenance, and use
+`cdidx db restore <name>` to roll back. Checkpoints live next to the DB under
+`<db>.checkpoints/<name>/`; restore keeps the pre-restore files under
+`<db>.restore-backup-<timestamp>/`. `backfill-fold` creates an automatic
+checkpoint before it mutates rows unless `--no-checkpoint` is passed.
+
+Database compatibility across `cdidx` binary upgrades and downgrades is
+documented in [COMPATIBILITY.md](COMPATIBILITY.md). Keep that policy updated
+whenever readiness bits, `codeindex_meta` contract stamps, or rebuild
+requirements change.
+
 `backfill-fold --dry-run` previews the folded-key rows that would be rewritten
 without mutating the DB or stamping FoldReady. The MCP `backfill_fold` tool
 accepts the same preview as `dry_run: true`, and also accepts `force: true` to

README.md

@@ -72,6 +72,7 @@ After the first command, use these cues and follow-up commands:
 | Edits or branch switches | Refresh incrementally with `--files`, `--commits`, or `--changed-between <old-ref> <new-ref>` instead of rebuilding. See [Quick Start](USER_GUIDE.md#quick-start) and [Incremental update reliability](USER_GUIDE.md#incremental-update-reliability). |
 | Intentional rebuilds | Interactive terminals ask before deleting the DB. Scripts and CI must pass `--yes` or `--force`. |
 | Long-lived DB compaction | Run `cdidx optimize` or `cdidx index <projectPath> --optimize` to compact FTS5 segments immediately. Incremental refreshes also optimize opportunistically. |
+| Maintenance rollback | Run `cdidx db checkpoint <name>` before risky DB maintenance and `cdidx db restore <name>` to roll back. `backfill-fold` creates an automatic checkpoint unless `--no-checkpoint` is passed. |
 | Permission or I/O scan errors | `cdidx` records the scan error, continues other directories, and writes `.cdidx/scan-checkpoint.json` so same-HEAD retries can skip completed directories. |
 
 Output controls:
@@ -109,6 +110,8 @@ Install choice and network notes:
 
 See [DISTRIBUTION.md](DISTRIBUTION.md) for the full channel matrix and
 [isolated network install notes](USER_GUIDE.md#isolated-networks-and-proxies).
+For database compatibility across `cdidx` upgrades and downgrades, see
+[COMPATIBILITY.md](COMPATIBILITY.md).
 
 ### Validate
 
@@ -331,6 +334,7 @@ extractor fixture を確認できます。詳細は
 | 編集後やブランチ切り替え後 | 再構築ではなく `--files`、`--commits`、`--changed-between <old-ref> <new-ref>` で差分更新します。詳細は [クイックスタート](USER_GUIDE.md#クイックスタート) と [インクリメンタル更新の信頼性](USER_GUIDE.md#インクリメンタル更新の信頼性) を参照してください。 |
 | 意図的な再構築 | interactive terminal では既存 DB 削除前に確認を求めます。script / CI では `--yes` または `--force` が必要です。 |
 | 長期間使っている DB の compact | `cdidx optimize` または `cdidx index <projectPath> --optimize` で FTS5 segment をすぐに compact できます。差分更新中も必要に応じて自動 optimize します。 |
+| 保守作業の rollback | risky な DB 保守の前に `cdidx db checkpoint <name>`、戻す場合は `cdidx db restore <name>` を使います。`backfill-fold` は `--no-checkpoint` を渡さない限り自動 checkpoint を作成します。 |
 | 権限や I/O の scan error | `cdidx` は scan error を記録し、他のディレクトリの走査を続けます。同じ HEAD の再実行では `.cdidx/scan-checkpoint.json` により成功済みディレクトリを読み飛ばせます。 |
 
 出力を整える option:
@@ -345,6 +349,9 @@ extractor fixture を確認できます。詳細は
 | UTF-8 JSON pipeline | CLI の `--json` 出力は BOM なし UTF-8 で書き出され、human output 向けに色を強制していても ANSI escape sequence を含みません。 |
 | script 向け query pipeline の stderr を静かにする | `--quiet`、`-q`、`--silent`、`CDIDX_QUIET=1` で informational stderr を抑制し、error 行だけを残します。`--quiet` は `--verbose` より優先されます。 |
 
+`cdidx` の upgrade / downgrade をまたぐ database 互換性については
+[COMPATIBILITY.md](COMPATIBILITY.md) を参照してください。
+
 ターミナル、スクリプト、CI、AI ツールから同じリポジトリを繰り返し検索する
 場合は `cdidx` が向いています。1回限りのテキスト検索には `rg` が向いています。
 

changelog.d/unreleased/1718.docs.md

@@ -0,0 +1,17 @@
+---
+category: docs
+issues:
+  - 1718
+affected:
+  - COMPATIBILITY.md
+  - README.md
+  - DEVELOPER_GUIDE.md
+---
+
+## English
+
+- **Documented DB compatibility policy (#1718)** — Added a bilingual compatibility contract for schema readiness bits, `codeindex_meta` stamps, upgrade/downgrade behavior, and rebuild requirements.
+
+## 日本語
+
+- **DB 互換性ポリシーを文書化しました (#1718)** — schema readiness bit、`codeindex_meta` stamp、upgrade / downgrade 時の動作、rebuild が必要な条件を説明する bilingual compatibility contract を追加しました。

changelog.d/unreleased/1721.added.md

@@ -0,0 +1,23 @@
+---
+category: added
+issues:
+  - 1721
+affected:
+  - src/CodeIndex/Cli/DbCommandRunner.cs
+  - src/CodeIndex/Cli/IndexCommandRunner.Maintenance.cs
+  - src/CodeIndex/Cli/IndexCommandRunner.cs
+  - src/CodeIndex/Cli/ConsoleUi.cs
+  - src/CodeIndex/Cli/CliFlagSchema.cs
+  - src/CodeIndex/Cli/JsonOutputContracts.cs
+  - tests/CodeIndex.Tests/DbCommandRunnerTests.cs
+  - README.md
+  - DEVELOPER_GUIDE.md
+---
+
+## English
+
+- **Added DB checkpoints and restore (#1721)** — `cdidx db checkpoint`, `cdidx db checkpoints --list`, and `cdidx db restore` can snapshot and roll back `codeindex.db`, while `backfill-fold` now creates an automatic checkpoint before mutating rows unless `--no-checkpoint` is passed.
+
+## 日本語
+
+- **DB checkpoint / restore を追加しました (#1721)** — `cdidx db checkpoint`、`cdidx db checkpoints --list`、`cdidx db restore` で `codeindex.db` を snapshot / rollback でき、`backfill-fold` は `--no-checkpoint` を渡さない限り row を変更する前に自動 checkpoint を作成します。

src/CodeIndex/Cli/CliFlagSchema.cs

@@ -274,6 +274,7 @@ private static IReadOnlyList<CliFlag> BuildAll()
             new() { Name = "--rebuild", Description = "Delete existing DB and rebuild from scratch", Commands = Set("index") },
             new() { Name = "--optimize", Description = "Optimize the existing FTS5 table without scanning files", Commands = Set("index") },
             new() { Name = "--dry-run", Description = "Preview without writing", Commands = Set("index", "backfill-fold") },
+            new() { Name = "--no-checkpoint", Description = "Skip the automatic DB checkpoint before maintenance", Commands = Set("backfill-fold") },
             new() { Name = "--force", Description = "Bypass the per-database index lock", Commands = Set("index") },
             new() { Name = "--duration-format", ValuePlaceholder = "<auto|seconds|hms>", Description = "Index elapsed time display format", Commands = Set("index") },
             new() { Name = "--max-file-bytes", ValuePlaceholder = "<bytes>", Description = "Override the per-file indexing size limit", Commands = Set("index") },

src/CodeIndex/Cli/ConsoleUi.cs

@@ -70,7 +70,7 @@ private static readonly (string Command, string Usage)[] CommandUsageLines =
     [
         ("index", "cdidx index <projectPath> [--db <path>] [--rebuild] [--optimize] [--verbose] [--dry-run] [--force] [--quiet] [--json] [--memory-trace] [--duration-format <auto|seconds|hms>] [--notify <auto|bell|osc9|desktop|none>] [--max-file-bytes <bytes>] [--follow-symlinks <none|internal|all>] [--include-symbol-kind <kind>[,<kind>]] [--exclude-symbol-kind <kind>[,<kind>]] [--watch [--debounce <ms>]]"),
         ("hooks", "cdidx hooks <install|uninstall|status> [--project <path>] [--force] [--json]"),
-        ("backfill-fold", "cdidx backfill-fold [--db <path>] [--dry-run] [--json]"),
+        ("backfill-fold", "cdidx backfill-fold [--db <path>] [--dry-run] [--no-checkpoint] [--json]"),
         ("optimize", "cdidx optimize [--db <path>] [--json]"),
         ("vacuum", "cdidx vacuum [--db <path>] [--json]"),
         ("index-commits", "cdidx index <projectPath> --commits <id> [id ...] [--db <path>] [--verbose] [--dry-run] [--json] [--memory-trace] [--duration-format <auto|seconds|hms>] [--max-file-bytes <bytes>] [--include-symbol-kind <kind>[,<kind>]] [--exclude-symbol-kind <kind>[,<kind>]]"),
@@ -93,7 +93,7 @@ private static readonly (string Command, string Usage)[] CommandUsageLines =
         ("workspace", "cdidx workspace <list|status|use|current> [name] [--json]"),
         ("config", "cdidx config show [--json]"),
         ("validate-config", "cdidx validate-config"),
-        ("db", "cdidx db --integrity-check|schema|prune [--dry-run|--apply] [--db <path>] [--json]"),
+        ("db", "cdidx db --integrity-check|schema|prune [--dry-run|--apply] [--db <path>] [--json] | cdidx db checkpoint [name] [--db <path>] [--json] | cdidx db checkpoints --list [--db <path>] [--json] | cdidx db restore <name> [--db <path>] [--json]"),
         ("diff", "cdidx diff <db1> <db2> [--json] [--summary-only] [--detailed] [--limit <n>]"),
         ("report", "cdidx report --output <path> [--db <path>] [--json] [--log-lines <n>] [--no-log] [--include-args]"),
         ("validate", "cdidx validate [--db <path>] [--json] [--format <text|json|count|compact|csv|tsv|lsp|qf|sarif>] [--verbose] [--kind <kind>] [--path <glob>]"),