Skip to content

Commit 555e4be

Browse files
committed
Merge remote-tracking branch 'origin/main' into fix-issue4614
# Conflicts: # USER_GUIDE.md
2 parents 5d3d0bc + f0a1e76 commit 555e4be

73 files changed

Lines changed: 3205 additions & 233 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

AGENT_GUIDE.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -135,14 +135,15 @@ CI watching must be bounded. Do not loop indefinitely.
135135

136136
## Status Contract
137137

138-
- `status --json` and related JSON/MCP payloads currently expose the trust fields documented in `README.md` and `DEVELOPER_GUIDE.md`, including `fold_ready`, `fold_ready_reason`, `graph_table_available`, `issues_table_available`, `file_issues_data_current`, `migration_in_progress`, `sql_graph_contract_ready`, `sql_graph_contract_degraded_reason`, `hotspot_family_ready`, `hotspot_family_degraded_reason`, `language_readiness`, `csharp_symbol_name_ready`, `csharp_metadata_target_ready`, `csharp_metadata_target_degraded_reason`, `indexed_head_commit`, `worktree_head_changed`, `indexed_head_sha`, `indexed_head_branch`, `indexed_head_timestamp`, `commits_ahead_of_indexed_head`, `head_freshness`, `index_writer_version`, `index_newer_than_reader`, `index_newer_than_reader_reason`, `unknown_extension_file_count`, `unknown_extension_files`, `unknown_extension_files_truncated`, `unknown_extension_file_path_limit`, `unknown_extension_extension_counts`, `unknown_extension_category_counts`, `unknown_extension_groups`, `extractors`, `git_executable`, `path_case_sensitive`, `data_dir`, `data_dir_source`, `data_dir_mode`, `db_file_mode`, `database_permission_policy`, `database_permission_diagnostics`, `mac_profile`, `mac_profile_diagnostics`, `db_size_bytes`, `wal_size_bytes`, `db_pragma_settings` (`journal_mode`, `synchronous`, `wal_autocheckpoint`, `busy_timeout_ms`, `page_count`, `freelist_count`, `page_size`, `auto_vacuum`), `prepared_command_cache` (`count`, `capacity`, `hit_count`, `miss_count`, `eviction_count`), `maintenance_guidance`, WAL checkpoint diagnostics (`read_only_fallback`, `wal_checkpoint_attempted`, `wal_checkpoint_succeeded`, `wal_checkpoint_skipped_reason`, `wal_checkpoint_failure_reason`, `wal_checkpoint_busy`, `wal_checkpoint_log_page_count`, `wal_checkpoint_checkpointed_page_count`, `wal_checkpoint_remaining_page_count`, `read_only_immutable_fallback`, `wal_stale_snapshot_risk`, `wal_stale_snapshot_reason`), `symbol_kinds`, `symbols_by_language`, status kind cap metadata (`symbol_kind_limit`, `symbol_kind_name_limit`, `symbol_kind_total_count`, `symbol_kind_omitted_count`, `symbol_kind_names_truncated`, `symbols_by_language_kind_total_counts`, `symbols_by_language_kind_omitted_counts`, `symbols_by_language_kind_names_truncated`), `process`, `last_index_run`, `last_failed_or_partial_index_run`, `last_failed_or_partial_index_run.progress_persisted`, `last_failed_or_partial_index_run.recovery_hint`, `last_workspace_freshened_at`, `hooks`, `hook_diagnostics`, `trust_overrides`, MCP-only `mcp_session`, `mcp.rate_limit.bucket_limit`, `mcp.rate_limit.bucket_limit_rejection_count`, and the `status --check`-only `stale_after_seconds` / `index_age_seconds` threshold audit fields and `repair_commands`.
138+
- `status --json` and related JSON/MCP payloads currently expose the trust fields documented in `README.md` and `DEVELOPER_GUIDE.md`, including `fold_ready`, `fold_ready_reason`, `graph_table_available`, `graph_data_current`, `index_complete`, `index_incomplete_reasons`, `issues_table_available`, `file_issues_data_current`, `migration_in_progress`, `sql_graph_contract_ready`, `sql_graph_contract_degraded_reason`, `hotspot_family_ready`, `hotspot_family_degraded_reason`, `language_readiness`, `csharp_symbol_name_ready`, `csharp_metadata_target_ready`, `csharp_metadata_target_degraded_reason`, `indexed_head_commit`, `worktree_head_changed`, `indexed_head_sha`, `indexed_head_branch`, `indexed_head_timestamp`, `commits_ahead_of_indexed_head`, `head_freshness`, `index_writer_version`, `index_newer_than_reader`, `index_newer_than_reader_reason`, `unknown_extension_file_count`, `unknown_extension_files`, `unknown_extension_files_truncated`, `unknown_extension_file_path_limit`, `unknown_extension_extension_counts`, `unknown_extension_category_counts`, `unknown_extension_groups`, `extractors`, `git_executable`, `path_case_sensitive`, `data_dir`, `data_dir_source`, `data_dir_mode`, `db_file_mode`, `database_permission_policy`, `database_permission_diagnostics`, `mac_profile`, `mac_profile_diagnostics`, `db_size_bytes`, `wal_size_bytes`, `db_pragma_settings` (`journal_mode`, `synchronous`, `wal_autocheckpoint`, `busy_timeout_ms`, `page_count`, `freelist_count`, `page_size`, `auto_vacuum`), `prepared_command_cache` (`count`, `capacity`, `hit_count`, `miss_count`, `eviction_count`), `maintenance_guidance`, WAL checkpoint diagnostics (`read_only_fallback`, `wal_checkpoint_attempted`, `wal_checkpoint_succeeded`, `wal_checkpoint_skipped_reason`, `wal_checkpoint_failure_reason`, `wal_checkpoint_busy`, `wal_checkpoint_log_page_count`, `wal_checkpoint_checkpointed_page_count`, `wal_checkpoint_remaining_page_count`, `read_only_immutable_fallback`, `wal_stale_snapshot_risk`, `wal_stale_snapshot_reason`), `symbol_kinds`, `symbols_by_language`, status kind cap metadata (`symbol_kind_limit`, `symbol_kind_name_limit`, `symbol_kind_total_count`, `symbol_kind_omitted_count`, `symbol_kind_names_truncated`, `symbols_by_language_kind_total_counts`, `symbols_by_language_kind_omitted_counts`, `symbols_by_language_kind_names_truncated`), `process`, `last_index_run`, `last_failed_or_partial_index_run`, `last_failed_or_partial_index_run.progress_persisted`, `last_failed_or_partial_index_run.recovery_hint`, `last_failed_or_partial_index_run.file_errors`, `last_workspace_freshened_at`, `hooks`, `hook_diagnostics`, `trust_overrides`, MCP-only `mcp_session`, `mcp.rate_limit.bucket_limit`, `mcp.rate_limit.bucket_limit_rejection_count`, and the `status --check`-only `stale_after_seconds` / `index_age_seconds` threshold audit fields and `repair_commands`.
139139
- A valid CLI `status --stale-after <duration>` implies the workspace check. Check-mode JSON includes `query_context.check_mode` (`explicit` or `implied_by_stale_after`) and `query_context.stale_after_seconds`; ordinary status JSON omits `query_context`.
140140
- Explicit WAL truncate-checkpoint diagnostics must preserve SQLite's `(busy, log, checkpointed)` result, treat non-zero `busy` or positive remaining pages as unsuccessful with bounded machine reasons, accept `(0, -1, -1)` as the successful non-WAL no-op, and never expose raw exception text or paths.
141141
- When any readiness field is degraded, the CLI adds `degraded_root_cause`, `degraded_reason`, `recommended_action`, `alternative_action`, and `readiness_degradations[]`. `degraded_root_cause` is the primary stable machine code; `readiness_degradations[]` lists every degraded field with `root_cause`, human reason, and remediation strings.
142142
- `hotspot_family_degraded_reason` currently uses `hotspot_family_support_not_indexed`, `hotspot_family_metadata_stale`, `hotspot_family_disabled_at_index_time`, `partial_family_key_population`, and `hotspot_family_marker_fingerprint_incomplete`; the incomplete marker fingerprint code means marker traversal hit safety caps and should stay synchronized with README / developer-guide recovery notes.
143143
- `issues_table_available` reports physical `file_issues` table presence only. `file_issues_data_current` reports whether the table is also stamped current for the active index generation.
144+
- `graph_table_available` reports a queryable persisted reference generation, while `graph_data_current`, `reference_graph_complete`, and `index_complete` report current-generation coverage. Reference extraction is bounded at 50,000 lookup symbols, 20,000 lookup lines, 512 names per line, and 20,000 container candidates; `reference_extraction_limits`, `reference_graph_incomplete_reasons`, and `reference_extraction_cap_hits` publish cap state, and `last_index_run.reference_extraction_cap_hits` snapshots it per run. Cap hits persist per file and propagate degraded, non-authoritative absence semantics to callers, callees, deps, and impact. A per-file extraction failure keeps successful graph rows queryable, stamps completeness false with bounded `last_failed_or_partial_index_run.file_errors`, and returns exit `11` unless `index --allow-partial` explicitly opts into exit `0`. While such file failures remain unresolved, a later scoped update automatically uses the normal incremental full-scan path so unrelated targets cannot clear the failure and successful recovery can restore every workspace-wide readiness contract without `--rebuild`.
144145
- `index_writer_version` records the `cdidx` version that last wrote to the DB (stamped into `codeindex_meta` as `cdidx_writer_version` on every full scan, update, and MCP index). `index_newer_than_reader` flips to `true` whenever any persisted numeric contract stamp in `codeindex_meta` (or unknown `PRAGMA user_version` readiness bits) exceeds the current binary's compiled maximum, so an older CLI re-opening a DB written by a newer CLI degrades loudly with an audit trail instead of silently dropping back to text-search fallbacks. `index_newer_than_reader_reason` enumerates the specific newer-than-reader stamps.
145-
- `status` also surfaces indexed-HEAD freshness via `indexed_head_sha`, `indexed_head_branch`, `indexed_head_timestamp`, `commits_ahead_of_indexed_head`, and the compact `head_freshness` summary. They are stamped by `cdidx index` on every successful run (full scan AND partial update, distinct from `indexed_head_commit` which is full-scan only) on a best-effort basis (never blocks an otherwise-successful index) and omitted on non-git workspaces, detached HEAD (branch only), or legacy DBs created before this contract. `worktree_head_changed` compares runtime HEAD with this latest stamp when available and falls back to `indexed_head_commit` only for legacy DBs. `head_freshness.state=fresh` requires `status --check` to match the workspace, while `state=head_current` only means the runtime HEAD matches the `indexed_head` selected by `indexed_head_source`.
146+
- `status` also surfaces indexed-HEAD freshness via `indexed_head_sha`, `indexed_head_branch`, `indexed_head_timestamp`, `commits_ahead_of_indexed_head`, and the compact `head_freshness` summary. They are stamped by `cdidx index` on every successful run (full scan AND partial update, distinct from `indexed_head_commit` which is full-scan only) on a best-effort basis (never blocks an otherwise-successful index) and omitted on non-git workspaces, detached HEAD (branch only), or legacy DBs created before this contract. `worktree_head_changed` compares runtime HEAD with this latest stamp when available and falls back to `indexed_head_commit` only for legacy DBs. `head_freshness.state=fresh` requires `status --check` to match the workspace, `fresh_but_incomplete` keeps matching-workspace freshness distinct from incomplete extraction coverage, and `state=head_current` only means the runtime HEAD matches the `indexed_head` selected by `indexed_head_source`.
146147
- `status` also surfaces unknown-extension scan coverage via `unknown_extension_file_count`, stamped by successful full-repository index runs (`cdidx index <projectPath>` and MCP `index_project`) as the number of non-indexed files with non-empty extensions that do not map to a known language. Current scans also stamp `unknown_extension_files` as a path sample bounded by `unknown_extension_file_path_limit` items and the string-list decoded-character budget, `unknown_extension_files_truncated` when more paths existed than were emitted for either bound, and `unknown_extension_file_path_limit` as the item cap rather than a guarantee that that many paths are returned. Newer scans also expose `unknown_extension_extension_counts`, `unknown_extension_category_counts`, and `unknown_extension_groups`; groups classify common non-code buckets such as repository metadata, licenses, binary assets, configuration, structural metadata, and language-support candidates, and include `recommended_action` values of `ignore_configuration`, `first_class_structural_extraction`, or `language_support`. These fields are omitted on legacy DBs or before a current full scan has stamped them.
147148
- `status` also surfaces extractor plugin and pattern-config runtime diagnostics via `extractors`, including loaded counts, the zero parent `retained_load_context_count`, the isolated-worker `load_context_lifecycle`, skipped file counts, and a bounded diagnostics list for incompatible or malformed plugin/pattern files. `extractors.pattern_configs[]` reports accepted sidecars with sanitized `path`, `source` (`workspace` or `user`), normalized `language`, and `rule_count`; workspace discovery stops at its explicit root and path identity follows the live filesystem case policy. Pattern rules and their 128-rule budget live in immutable workspace snapshots; reindex replaces the owning snapshot, and timeout cooldowns/diagnostics remain workspace-scoped. `extractors.snapshot_scope` and highest-first `registration_precedence` expose snapshot selection and the fixed `built_in > user_plugin > user_pattern > workspace_plugin > workspace_pattern` resolution order; replacing one workspace snapshot never mutates another. Diagnostic paths, categories, and messages are sanitized before output; `diagnostics[].category` is the stable machine-readable failure code.
148149
- `status` surfaces worker-discovered post-extraction hook manifests and callback budgets through `hooks[]`, including stable assembly-qualified `hooks[].id`, `hooks[].callback_budget_ms`, and the worker-only `hooks[].load_context_lifecycle`. Assembly loading, module initialization, type inspection, and constructor validation occur only in deadline-, memory-, and output-bounded discovery workers, which are terminated after returning a manifest. `hook_diagnostics[]` reports sanitized discovery and callback diagnostics such as candidate-limit truncation, assembly load failure, constructor failure, callback failure, and timeout; `hook_diagnostics[].category` is the stable machine-readable failure code and `hook_diagnostics[].hook_id` links a concrete-hook failure to the corresponding manifest. Index runs enforce `CDIDX_HOOK_CALLBACK_BUDGET_MS` (default: 5000 ms) on scratch copies, discard timed-out mutations, and disable only the timed-out assembly-qualified hook ID for the remainder of the current run.

COMPATIBILITY.md

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ must refuse writes that could silently discard newer data.
2424

2525
| Bit | Field | Meaning |
2626
|---|---|---|
27-
| `1` | `graph_table_available` / graph readiness | `symbol_references` has been fully populated for graph queries. |
27+
| `1` | `graph_table_available` / graph presence | `symbol_references` contains a queryable committed generation. Use `graph_data_current` / `index_complete` / `reference_graph_complete` to decide whether current-workspace coverage is complete. |
2828
| `2` | `issues_table_available` / issue readiness | `file_issues` has been populated for validation results. |
2929
| `4` | `fold_ready` | Folded-name columns are current for Unicode-aware exact-name matching. |
3030
| `8` | hotspot reference aggregate storage contract | The database uses `hotspot_reference_counts`; this permanent downgrade guard is preserved while other readiness bits are cleared. |
@@ -44,6 +44,12 @@ metadata, unknown-extension counts, filesystem case-sensitivity, MAC profile,
4444
and DB/WAL/status diagnostics. These stamps let readers distinguish a feature
4545
that is absent, stale, or newer than the running binary.
4646

47+
Reference-extraction cap hits use existing per-file `file_issues` rows rather
48+
than a new schema bit. Current readers aggregate those rows into
49+
`reference_extraction_cap_hits` and set `reference_graph_complete=false`; a
50+
legacy database without inspectable issue state degrades rather than claiming
51+
complete graph coverage.
52+
4753
## Version Skew Behavior
4854

4955
Use `cdidx status --json` or `cdidx status --check --json` before relying on a
@@ -96,7 +102,7 @@ degrade しなければなりません。古い binary が新しい database を
96102

97103
| Bit | Field | 意味 |
98104
|---|---|---|
99-
| `1` | `graph_table_available` / graph readiness | graph query 用の `symbol_references` が完全に作成済み|
105+
| `1` | `graph_table_available` / graph presence | graph query 可能な commit 済み `symbol_references` generation が存在する。current workspace の coverage 完全性は `graph_data_current` / `index_complete` / `reference_graph_complete` で判定する|
100106
| `2` | `issues_table_available` / issue readiness | validation result 用の `file_issues` が作成済み。 |
101107
| `4` | `fold_ready` | Unicode-aware exact-name matching 用の folded-name column が最新。 |
102108
| `8` | hotspot reference aggregate storage contract | database が `hotspot_reference_counts` を使用することを示す永続 downgrade guard。他の readiness bit のクリア時にも保持される。 |
@@ -115,6 +121,11 @@ metadata、unknown-extension count、filesystem case-sensitivity、MAC profile
115121
DB/WAL/status diagnostics が含まれます。reader はこれらの stamp により、feature
116122
が存在しないのか、stale なのか、実行中 binary より新しいのかを判別できます。
117123

124+
reference-extraction cap hit は新しい schema bit ではなく、既存の file ごとの
125+
`file_issues` row を使います。current reader はそれを `reference_extraction_cap_hits`
126+
へ集約して `reference_graph_complete=false` にします。issue state を確認できない
127+
legacy database は complete graph coverage を主張せず degraded になります。
128+
118129
## Version skew 時の動作
119130

120131
binary upgrade / downgrade をまたいで database を使う前に、

0 commit comments

Comments
 (0)