Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
254 changes: 246 additions & 8 deletions DEVELOPER_GUIDE.md

Large diffs are not rendered by default.

37 changes: 28 additions & 9 deletions TESTING_GUIDE.md

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
---
category: changed
affected:
- DEVELOPER_GUIDE.md
- TESTING_GUIDE.md
- src/CodeIndex/Indexer/BoundedRegex.cs
- src/CodeIndex/Indexer/References/ReferenceExtractor.cs
- src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.AlterTargets.cs
- src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.Sources.cs
- src/CodeIndex/Indexer/References/Languages/SqlReferenceExtractor.StatementState.cs
- src/CodeIndex/Indexer/References/Support/LanguageReferenceExtractionSupport.Utilities.cs
- tests/CodeIndex.Tests/BoundedRegexTests.cs
- tests/CodeIndex.Tests/PerformanceTests.cs
---

## English

- **Lazy regex match enumeration no longer allocates iterator wrappers on indexing hot paths** — instance-regex scans now use concrete value enumerables and enumerators through reference extraction and SQL helpers. Direct scans and already-full reference caps avoid iterator and interface-boxing allocations while preserving timeout diagnostics, zero-length and `\G` progression, explicit and right-to-left start positions, early termination, and LINQ compatibility.

## 日本語

- **indexing の hot path で lazy regex match 列挙の iterator wrapper allocation を解消しました** — instance regex の走査は、reference extraction と SQL helper を通して concrete な value enumerable / enumerator を使います。direct scan と既に満杯の reference cap は iterator および interface boxing allocation を避けながら、timeout diagnostic、zero-length と `\G` の進行、明示的および right-to-left の開始位置、早期終了、LINQ 互換性を維持します。
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
---
category: fixed
affected:
- TESTING_GUIDE.md
- src/CodeIndex/Cli/PathCasing.cs
- tests/CodeIndex.Tests/PathCasingTests.cs
---

## English

- **Exact-case path comparisons no longer mutate directories during indexing** — path equality now resolves ordinal matches and impossible unequal-length matches, while parent-boundary checks accept exact-case matches and descendants, before either operation probes filesystem case sensitivity. Empty nested workspace members therefore keep their discovery snapshot stable instead of receiving a temporary case-probe entry after traversal.

## 日本語

- **大小文字まで一致する path 比較で index 中の directory を変更しないよう修正しました** — filesystem の case sensitivity を probe する前に、path equality は ordinal 一致と長さの異なる不一致を確定し、親境界の判定は大小文字まで一致する同一 path と descendant を受理します。これにより、空の nested workspace member は走査後に一時 case-probe entry を作られず、discovery snapshot を安定して維持します。
17 changes: 17 additions & 0 deletions changelog.d/unreleased/+csharp-reference-graph-facts.internal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
---
category: internal
affected:
- src/CodeIndex/Database/DbWriter.References.cs
- src/CodeIndex/Database/DbWriter.ReferenceGraphRefreshScope.cs
- tests/CodeIndex.Tests/DatabaseTests.cs
- DEVELOPER_GUIDE.md
- TESTING_GUIDE.md
---

## English

- **C# graph finalization reuses materialized scalar facts** — full, scoped, and retained reference-graph refreshes now compute C# reference and symbol arity, receiver, constructor, and value-type facts once per applicable row, avoiding repeated managed SQLite callbacks for every candidate in large graphs.

## 日本語

- **C# graph finalization が materialize 済み scalar fact を再利用します** — full / scoped / retained の reference-graph refresh は、C# reference と symbol の arity、receiver、constructor、value-type fact を対象 row ごとに1回だけ計算し、巨大 graph の candidate ごとに managed SQLite callback を反復する処理を避けるようになりました。
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
---
category: internal
affected:
- src/CodeIndex/Database/DbWriter.References.cs
- src/CodeIndex/Database/DbWriter.ReferenceGraphRefreshScope.cs
- tests/CodeIndex.Tests/DatabaseTests.cs
- DEVELOPER_GUIDE.md
- TESTING_GUIDE.md
---

## English

- **C# graph finalization reuses materialized type and constructor identities** — full, scoped, and retained reference-graph refreshes now build project/file-local type identities and ranked constructor-owner identities once per applicable symbol, replacing repeated candidate-side string construction and range scans with TEMP primary-key lookups.

## 日本語

- **C# graph finalization が materialize 済み type / constructor identity を再利用します** — full / scoped / retained の reference-graph refresh は project / file-local type identity と順位付き constructor-owner identity を対象 symbol ごとに1回だけ構築し、candidate 側で反復していた文字列生成と range scan を TEMP 主キー lookup に置き換えます。
16 changes: 16 additions & 0 deletions changelog.d/unreleased/+defer-reference-indexes.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
category: changed
affected:
- src/CodeIndex/Database/ReferenceSecondaryIndexBulkLoadGuard.cs
- src/CodeIndex/Database/ReferenceSecondaryIndexSql.cs
- src/CodeIndex/Cli/IndexCommandRunner.FullScan.cs
- src/CodeIndex/Mcp/McpToolHandlers.Indexing.Execution.cs
---

## English

- **Fresh and rebuilt indexes persist large reference sets without maintaining every query index row by row** — CLI indexing now defers reference query and graph secondary indexes inside its outer transaction and restores them once before graph finalization. Empty-database MCP indexing uses the same optimization with recoverable cleanup, while maintenance indexes, rollback safety, read repair, and the final database schema remain unchanged.

## 日本語

- **fresh / rebuild index が、大量の reference を query index ごとに逐次更新せず永続化するようになりました** — CLI indexing は外側 transaction 内で reference query / graph 用 secondary index を遅延し、graph finalization 前に1回だけ復元します。空 database から始める MCP indexing も recoverable cleanup 付きで同じ最適化を使い、保守用 index、rollback safety、read repair、最終 database schema は従来どおり維持します。
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
category: changed
affected:
- DEVELOPER_GUIDE.md
- TESTING_GUIDE.md
- src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallReferences.cs
- src/CodeIndex/Indexer/References/ReferenceExtractor.CoreTypeReferences.cs
- src/CodeIndex/Indexer/References/Languages/
- src/CodeIndex/Indexer/References/Support/
- tests/CodeIndex.Tests/ReferenceExtractorMarkerGateTests.cs
- tests/CodeIndex.Tests/ReferenceExtractorPerformanceBudgetTests.cs
---

## English

- **Large multi-language indexes avoid impossible per-line reference regex work** — reference extraction now checks cheap syntax markers before entering expensive call, type, framework, and markup patterns across C#, Java, Kotlin, JavaScript/TypeScript, Go, Dockerfile, GraphQL, HTML, Markdown, XAML/XML, Terraform, JSON, and GitHub Actions. Stateful multi-line parsing and syntax without the shared markers retain their existing references, while markerless source lines complete with substantially less regex work.

## 日本語

- **大規模な multi-language index で成立しない行単位 reference regex の実行を回避しました** — C#、Java、Kotlin、JavaScript/TypeScript、Go、Dockerfile、GraphQL、HTML、Markdown、XAML/XML、Terraform、JSON、GitHub Actions の call、type、framework、markup pattern を処理する前に、安価な構文 marker を確認します。stateful な複数行解析と共有 marker を持たない構文の既存 reference は維持しつつ、marker のない source 行に対する regex work を大幅に減らします。
17 changes: 17 additions & 0 deletions changelog.d/unreleased/+mixed-filesystem-marker-scope.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
---
category: fixed
affected:
- src/CodeIndex/Indexer/Scanning/FileIndexer.ProjectMarkers.cs
- src/CodeIndex/Indexer/Scanning/FileIndexer.ScanOrchestration.cs
- src/CodeIndex/Indexer/Scanning/FileIndexer.Types.cs
- tests/CodeIndex.Tests/FileIndexerTests.cs
- TESTING_GUIDE.md
---

## English

- **Project family scopes now honor case policy changes inside mixed filesystems** — the completed scan snapshot keeps case-only marker directories distinct under case-sensitive children while still resolving aliases under case-insensitive children, without returning to live marker enumeration or adding filesystem probes.

## 日本語

- **混在 filesystem 内で case policy が変わる場合も project family scope が正しくなりました** — 完了した scan snapshot は case-sensitive child 配下の大小文字だけが異なる marker directory を分離しつつ、case-insensitive child 配下の alias は同一 scope として解決し、live marker 列挙への後退や filesystem probe の追加も行いません。
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
---
category: changed
affected:
- src/CodeIndex/Cli/CliFlagSchema.cs
- src/CodeIndex/Cli/EnvironmentVariableInventory.cs
- src/CodeIndex/Cli/IndexCommandRunner.cs
- src/CodeIndex/Cli/IndexCommandRunner.FullScan.Interruption.cs
- src/CodeIndex/Cli/IndexCommandRunner.Update.FileLoop.cs
- src/CodeIndex/Cli/IndexCommandRunner.Update.FilePersistence.cs
- src/CodeIndex/Cli/IndexCommandRunner.Update.ParallelExtraction.cs
- src/CodeIndex/Cli/IndexCommandRunner.WorkItems.cs
- src/CodeIndex/Indexer/Hooks/PostExtractionHooks.cs
- tests/CodeIndex.Tests/IndexCommandRunnerUpdateTests.cs
- DEVELOPER_GUIDE.md
- TESTING_GUIDE.md
---

## English

- **Large authoritative C# scoped updates now extract files concurrently within bounded ordered windows** — eligible static-interface workspace updates reuse a fixed worker pool, cap each window at twice the worker count, and keep SQLite persistence, hooks, readiness, byte accounting, and progress on one target-ordered consumer. Three file-stat barriers, nullable language re-detection, serial probe/hook/filter fallbacks, phase-aware cancellation and stall handling, and source-contract candidate ordering preserve the existing update semantics while reducing extraction time on large C# workspaces.

## 日本語

- **大規模な authoritative C# scoped update が、上限付き ordered window 内で file extraction を並列実行するようになりました** — static-interface workspace の対象 update は固定 worker pool を再利用し、各 window を worker 数の2倍までに制限します。SQLite persistence、hook、readiness、byte accounting、progress は target 順の single consumer に維持します。3段階の file-stat barrier、nullable language re-detection、probe / hook / filter の serial fallback、phase-aware な cancellation / stall 処理、source-contract candidate の順序制御により既存の update semantics を保ちながら、大規模 C# workspace の extraction 時間を短縮します。
19 changes: 19 additions & 0 deletions changelog.d/unreleased/+prune-reference-indexes.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
---
category: changed
affected:
- src/CodeIndex/Database/ReferenceSecondaryIndexSql.cs
- src/CodeIndex/Database/ReferenceSecondaryIndexBulkLoadGuard.cs
- src/CodeIndex/Database/DbContext.cs
- src/CodeIndex/Database/DbContext.ReadMigrations.cs
- src/CodeIndex/Database/DbReader.cs
- src/CodeIndex/Database/DbWriter.ReferenceGraphRefreshScope.cs
- src/CodeIndex/Database/DbWriter.References.cs
---

## English

- **Large reference graphs rebuild fewer redundant index pages** — canonical schema creation, read migration, and bulk-load recovery now retire six single-column name/container indexes already covered by retained composite prefixes. The former all-row folded mutual-recursion index is replaced by a partial index containing only unresolved callable edges, while folded and legacy NOCASE exact queries continue to use retained composite indexes and older databases are pruned on open.

## 日本語

- **大規模なreference graphで冗長なindex pageの再構築を削減しました** — canonical schema作成、read migration、bulk-load recoveryは、保持済みcomposite prefixで代替できるname/container単一カラムindex 6本を退役させます。全rowを保持していた旧folded mutual-recursion indexは未解決のcallable edgeだけを含むpartial indexへ置き換え、folded/legacy NOCASEのexact queryは保持済みcomposite indexを引き続き利用し、旧databaseもopen時にpruneします。
17 changes: 17 additions & 0 deletions changelog.d/unreleased/+reuse-project-marker-scan.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
---
category: changed
affected:
- src/CodeIndex/Indexer/Scanning/FileIndexer.ProjectMarkers.cs
- src/CodeIndex/Indexer/Scanning/FileIndexer.DirectoryEnumeration.cs
- src/CodeIndex/Indexer/Scanning/FileIndexer.ScanOrchestration.cs
- src/CodeIndex/Cli/IndexCommandRunner.FullScan.cs
- src/CodeIndex/Cli/IndexCommandRunner.DryRun.cs
---

## English

- **Full indexing no longer walks large directory trees again for project-family metadata** — authoritative scans now collect C#, VB, F#, and MSBuild marker fingerprints during normal discovery and reuse a complete marker-directory snapshot for per-file family scopes. Fingerprint limits, ignore boundaries, incomplete-scan warnings, and live fallbacks remain fail-closed.

## 日本語

- **full indexing が project-family metadata のために巨大な directory tree を再走査しなくなりました** — authoritative scan は通常 discovery 中に C#、VB、F#、MSBuild の marker fingerprint を収集し、complete な marker-directory snapshot を file ごとの family scope に再利用します。fingerprint 上限、ignore 境界、不完全 scan warning、live fallback は引き続き fail-closed です。
18 changes: 18 additions & 0 deletions changelog.d/unreleased/+single-pass-content-facts.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
category: changed
affected:
- src/CodeIndex/Indexer/Scanning/FileContentLoader.cs
- src/CodeIndex/Indexer/Scanning/NormalizedContentFacts.cs
- src/CodeIndex/Indexer/Scanning/ChunkSplitter.cs
- src/CodeIndex/Indexer/Scanning/FileIndexer.ContentValidation.cs
- src/CodeIndex/Cli/IndexCommandRunner.WorkItems.cs
- src/CodeIndex/Mcp/McpToolHandlers.Indexing.Execution.cs
---

## English

- **Large multi-language files are normalized, analyzed, chunked, and validated with one shared content scan** — full scans, scoped updates, dry runs, and MCP indexing now carry normalized line, size, conflict-marker, replacement-character, FTS-token, and chunk-boundary facts through every language-independent path instead of rediscovering them in each consumer. Short files avoid unused boundary/token tracking, and high-ratio invalid UTF-8 input no longer retains one replacement-line entry per damaged line.

## 日本語

- **大規模な multi-language file の正規化・解析・chunk 分割・validation を1回の共有 content scan にまとめました** — full scan、scoped update、dry-run、MCP indexing は、正規化後の行数、size、conflict marker、replacement character、FTS token、chunk 境界の facts を全言語共通経路で引き回し、consumer ごとの再検出を省きます。短い file では不要な境界 / token 追跡を避け、高比率の invalid UTF-8 input では破損行ごとの replacement-line entry を保持しません。
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
---
category: internal
affected:
- src/CodeIndex/Database/ReferenceSecondaryIndexSql.cs
- src/CodeIndex/Database/ReferenceSecondaryIndexBulkLoadGuard.cs
- src/CodeIndex/Database/DbWriter.ReferenceGraphRefreshScope.cs
- src/CodeIndex/Database/DbWriter.References.cs
- src/CodeIndex/Cli/IndexCommandRunner.FullScan.cs
- src/CodeIndex/Cli/IndexCommandRunner.Update.cs
- src/CodeIndex/Mcp/McpToolHandlers.Indexing.Execution.cs
- tests/CodeIndex.Tests/ReferenceSecondaryIndexBulkLoadGuardTests.cs
- tests/CodeIndex.Tests/IndexCommandRunnerReferenceIndexBulkLoadTests.cs
- tests/CodeIndex.Tests/McpServerToolsCallTests.cs
- tests/CodeIndex.Tests/DatabaseTests.cs
- DEVELOPER_GUIDE.md
- TESTING_GUIDE.md
---

## English

- **Reference-index restoration is staged around graph finalization** — large CLI and MCP bulk loads now keep query-only reference indexes absent through identity resolution, restore only three reverse-edge indexes for mutual-recursion evaluation, and rebuild the remaining query indexes afterwards. Active scoped graph refreshes are promoted to full plans before deferral, while cancellation, rollback, recovery, heartbeat, and SQLite `changes()` contracts remain intact.

## 日本語

- **reference index の復元を graph finalization の前後に段階化しました** — 大規模な CLI / MCP bulk load は identity resolution 中も query-only reference index を外したままにし、mutual-recursion 評価には reverse-edge 用3本だけを復元して、残りの query index はその後に再構築します。active な scoped graph refresh は遅延前に full plan へ昇格し、cancellation、rollback、recovery、heartbeat、SQLite `changes()` の各契約を維持します。
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
category: changed
affected:
- src/CodeIndex/Cli/IndexCommandRunner.FullScan.cs
- src/CodeIndex/Cli/IndexCommandRunner.Update.cs
- src/CodeIndex/Database/DbWriter.ReferenceSql.cs
- src/CodeIndex/Database/DbWriter.References.cs
---

## English

- **Large multi-language reference batches perform less index maintenance, parameter binding, and tuple hashing** — existing-database full scans that cross the established FTS bulk-load threshold now defer reference-query indexes transactionally, while scoped updates do the same recoverably only after at least 64 targets reach 60% of indexed files. Reference inserts also write the normalized legacy context column as a SQL `NULL` literal and carry materialized reference-line IDs through compact ordinal arrays, avoiding one binding and a repeated file/line/context dictionary lookup per edge across fresh and replacement paths. Small updates retain every query index and avoid rebuild overhead.

## 日本語

- **大規模な multi-language reference batch の index maintenance、parameter binding、tuple hashing を削減しました** — 既存DBの full scan が既定の FTS bulk-load 閾値を超えた場合は reference query index を transaction 内で一時退避し、scoped update では64 target以上かつ indexed fileの60%以上の場合だけ recoverable に同じ処理を行います。reference insert は正規化済み legacy context column を SQL の `NULL` literal として書き込み、materialize 済み reference-line ID を compact な ordinal array で渡します。fresh / replacement の全経路で edge ごとの binding 1件と file / line / context dictionary の再 lookup を省きます。小規模 update は全 query index を維持し、再構築 overhead を避けます。
2 changes: 1 addition & 1 deletion src/CodeIndex/Cli/CliFlagSchema.cs
Original file line number Diff line number Diff line change
Expand Up @@ -670,7 +670,7 @@ private static IReadOnlyList<CliFlag> BuildAll()
new() { Name = "--max-file-bytes", ValuePlaceholder = "<bytes>", Description = "Override the per-file indexing size limit", PrimaryCommands = Set("index") },
new() { Name = "--max-symbols-per-file", ValuePlaceholder = "<n>", Description = "Skip file content, symbols, and references when one file emits too many symbols (max 50000)", PrimaryCommands = Set("index") },
new() { Name = "--max-references-per-file", ValuePlaceholder = "<n>", Description = "Skip references when one file emits too many references (max 1000000)", PrimaryCommands = Set("index") },
new() { Name = "--parallelism", ValuePlaceholder = "<n>", Description = "Full-scan extraction worker count (default: CPU count capped at 8; explicit max: 16; also honors CDIDX_INDEX_PARALLELISM)", PrimaryCommands = Set("index") },
new() { Name = "--parallelism", ValuePlaceholder = "<n>", Description = "Full-scan and authoritative C# update extraction worker count (default: CPU count capped at 8; explicit max: 16; also honors CDIDX_INDEX_PARALLELISM)", PrimaryCommands = Set("index") },
new() { Name = "--memory-trace", Description = "Include phase memory samples in index JSON output", PrimaryCommands = Set("index") },
new() { Name = "--commits", ValuePlaceholder = "<commit-ref>", Description = "Update files changed in given git commits", PrimaryCommands = Set("index") },
new() { Name = "--changed-between", ValuePlaceholder = "<old-ref> <new-ref>", Description = "Update files changed between two git refs", PrimaryCommands = Set("index") },
Expand Down
Loading
Loading