Merge pull request #5155 from Widthdom/perf-initial-full-index-20260823

Speed up initial full indexing

Author: Widthdom

DEVELOPER_GUIDE.md

@@ -0,0 +1,26 @@
+---
+category: changed
+affected:
+  - src/CodeIndex/Cli/IndexCommandRunner.FullScan.cs
+  - src/CodeIndex/Cli/IndexCommandRunner.FullScan.FilePersistence.cs
+  - src/CodeIndex/Database/DbWriter.AuthoritativeFreshBulkInsert.cs
+  - src/CodeIndex/Database/DbWriter.AuthoritativeFreshReturningInsert.cs
+  - src/CodeIndex/Database/DbWriter.ChunkSymbolBatches.cs
+  - src/CodeIndex/Database/DbWriter.Files.cs
+  - src/CodeIndex/Database/DbWriter.Issues.cs
+  - src/CodeIndex/Database/DbWriter.References.cs
+  - tests/CodeIndex.Tests/AuthoritativeFreshRawBulkInsertTests.cs
+  - tests/CodeIndex.Tests/IndexCommandRunnerFullScanTests.cs
+  - tests/CodeIndex.Tests/IndexCommandRunnerInitialFullIndexPerformanceTests.cs
+  - tests/CodeIndex.Tests/PerformanceTests.cs
+  - DEVELOPER_GUIDE.md
+  - TESTING_GUIDE.md
+---
+
+## English
+
+- **Authoritative empty CLI scans now bind their complete fresh-file persistence sequence through SQLitePCLRaw** — file and fresh reference-line `RETURNING` inserts join chunk, symbol, new-file issue, and atomic fresh-reference batches in a bounded native statement cache during extraction, removing row-count-proportional Microsoft.Data.Sqlite command, reader, and parameter allocation while retaining the existing 32-parameter statement/tail boundaries, transaction atomicity, row-skip replay, cancellation, hooks, date/Unicode/NULL semantics, and provider-owned connection lifetime. Returned IDs and input ordinals are fully buffered and validated before publication; malformed, incomplete, duplicate, failed, or cancelled result streams discard the statement and roll back through the per-file savepoint. Rebuild, incremental, replacement, symbols-only, race-fallback, MCP, and public-writer paths continue to use Microsoft.Data.Sqlite.
+
+## 日本語
+
+- **authoritative empty CLI scanのfresh-file永続化全体をSQLitePCLRawでbindするようになりました** — file / fresh reference-lineの`RETURNING` insertをchunk、symbol、new-file issue、atomic fresh-reference batchと同じbounded native statement cacheへ加え、既存の32 parameter statement / tail境界、transaction atomicity、row-skip replay、cancellation、hook、date / Unicode / NULL semantics、provider所有connection lifetimeを保ったまま、row数に比例するMicrosoft.Data.Sqlite command / reader / parameter allocationを除きます。返却IDとinput ordinalは公開前に全件buffer / validationし、不正、欠落、重複、失敗、cancelされたresult streamではstatementを破棄してfile単位SAVEPOINTからrollbackします。rebuild、incremental、replacement、symbols-only、race fallback、MCP、public writer経路は引き続きMicrosoft.Data.Sqliteを使います。

TESTING_GUIDE.md

@@ -0,0 +1,23 @@
+---
+category: changed
+affected:
+  - src/CodeIndex/Cli/IndexCommandRunner.FullScan.cs
+  - src/CodeIndex/Database/DbContext.ReadMigrations.cs
+  - src/CodeIndex/Database/DbContext.SchemaInitialization.cs
+  - src/CodeIndex/Database/ReferenceSecondaryIndexBulkLoadGuard.cs
+  - src/CodeIndex/Database/ReferenceSecondaryIndexSql.cs
+  - tests/CodeIndex.Tests/IndexCommandRunnerFullScanTests.cs
+  - tests/CodeIndex.Tests/IndexCommandRunnerReferenceIndexBulkLoadTests.cs
+  - tests/CodeIndex.Tests/McpServerToolsCallTests.cs
+  - tests/CodeIndex.Tests/ReferenceSecondaryIndexBulkLoadGuardTests.cs
+  - tests/CodeIndex.Tests/DatabaseTests.cs
+  - DEVELOPER_GUIDE.md
+---
+
+## English
+
+- **Authoritative empty-database CLI indexing now defers two reference-persistence indexes during the raw load** — after transaction-local fresh-ownership revalidation, the initial full scan omits the reference-line foreign-key and file/line lookup indexes until candidate or graph work begins, then restores both inside the same outer transaction. Rebuilds, existing-database updates, raced fresh claims, and MCP indexing keep both indexes available, while cancellation and failure retain atomic schema rollback.
+
+## 日本語
+
+- **authoritativeな空DB初回CLI indexで、raw load中のreference永続化index 2本を遅延するようになりました** — transaction-localなfresh ownership再確認後、初回full scanはreference-line foreign-key用とfile/line lookup用のindexをcandidateまたはgraph処理の開始まで外し、同じouter transaction内で2本とも復元します。rebuild、既存DB update、競合で失効したfresh claim、MCP indexingでは常時維持し、cancellationや失敗時もschema rollbackの原子性を保ちます。

changelog.d/unreleased/+authoritative-fresh-raw-bulk-inserts.changed.md

@@ -0,0 +1,14 @@
+---
+category: internal
+affected:
+  - tests/CodeIndex.Tests/IndexCommandRunnerInitialFullIndexPerformanceTests.cs
+  - TESTING_GUIDE.md
+---
+
+## English
+
+- **Cold full-index performance work now has an end-to-end multilingual regression contract** — a fresh external database fixture exercises the real CLI pipeline for C#, TypeScript, Python, Java, Go, Rust, C++, and Kotlin, checks phase ordering, persisted counts, readiness, workspace status, FTS searchability, and database integrity under a broad runaway budget, and provides a larger opt-in manual smoke using the same assertions.
+
+## 日本語
+
+- **空DBの初回フルインデックス高速化に、多言語のend-to-end回帰契約を追加しました** — 外部の新規DBを使うfixtureがC#、TypeScript、Python、Java、Go、Rust、C++、Kotlinの実CLI pipelineを通り、広いrunaway budgetのもとでphase順序、永続化件数、readiness、workspace status、FTS検索可能性、DB integrityを検証します。同じassertionを使う大きめのopt-in手動smokeも追加しました。

changelog.d/unreleased/+authoritative-fresh-reference-index-deferral.changed.md

@@ -0,0 +1,23 @@
+---
+category: internal
+affected:
+  - src/CodeIndex/Indexer/Symbols/SymbolExtractor.CSharpScanner.cs
+  - src/CodeIndex/Indexer/Symbols/SymbolExtractor.ExtractCore.cs
+  - src/CodeIndex/Indexer/Symbols/SymbolExtractor.ExtractionPhases.cs
+  - src/CodeIndex/Indexer/Symbols/SymbolExtractor.PatternExtraction.cs
+  - src/CodeIndex/Indexer/Symbols/SymbolExtractor.PatternFlow.cs
+  - src/CodeIndex/Indexer/Symbols/SymbolExtractor.PatternLoop.cs
+  - src/CodeIndex/Indexer/Symbols/SymbolExtractor.PatternMatching.cs
+  - src/CodeIndex/Indexer/Symbols/SymbolExtractor.Patterns.cs
+  - src/CodeIndex/Indexer/Symbols/SymbolExtractor.cs
+  - tests/CodeIndex.Tests/SymbolExtractorCSharpRegexProbeTests.cs
+  - DEVELOPER_GUIDE.md
+---
+
+## English
+
+- **C# symbol extraction now avoids regex probes that cannot succeed** — completed property inputs and plain-field inputs without a required terminator are rejected with allocation-free character checks, while wrapped-modifier recovery caches absent and present lookups and materializes confirmed prefixes once. Multiline default arguments, incomplete-attribute recovery, same-line offsets, diagnostics, and cancellation behavior remain unchanged.
+
+## 日本語
+
+- **C# symbol extraction が成功不可能な regex probe を回避するようになりました** — 完結済み property input と必須終端を含まない plain-field input を allocation-free な文字判定で除外し、wrapped-modifier recovery は有無どちらの lookup も cache し、確定した prefix を1回だけ materialize します。複数行 default argument、不完全 attribute recovery、same-line offset、diagnostic、cancellation の振る舞いは変わりません。

changelog.d/unreleased/+cold-full-index-contract.internal.md

@@ -0,0 +1,16 @@
+---
+category: changed
+affected:
+  - src/CodeIndex/Database/DbWriter.References.cs
+  - tests/CodeIndex.Tests/DatabaseTests.cs
+  - DEVELOPER_GUIDE.md
+  - TESTING_GUIDE.md
+---
+
+## English
+
+- **Cold C# graph finalization now resolves unqualified instantiations with set-based family facts** — Rank-5 constructor and type candidates materialize unique families and explicit-constructor summaries once, preserving exact case, arity, partial-type representatives, overloads, implicit defaults, ambiguity, and lower-rank semantics while avoiding repeated correlated symbol probes.
+
+## 日本語
+
+- **C# graph の cold finalization は無修飾 instantiation を family fact の集合処理で解決するようになりました** — rank 5 の constructor / type candidate は一意 family と明示 constructor summary を一度だけ materialize し、exact case、arity、partial type の代表、overload、implicit default、ambiguity、lower-rank の意味を維持しながら、相関 symbol probe の反復を避けます。

changelog.d/unreleased/+csharp-impossible-regex-probes.internal.md

@@ -0,0 +1,15 @@
+---
+category: changed
+affected:
+  - src/CodeIndex/Indexer/BoundedRegex.cs
+  - src/CodeIndex/Indexer/Symbols/
+  - tests/CodeIndex.Tests/SymbolExtractorCSharpRegexProbeTests.cs
+---
+
+## English
+
+- **Cold C# indexing avoids duplicate declaration regex probes** — the declaration loop and its recoverable-pattern check now share candidate-local deterministic misses for the same physical input, while merged multiline inputs, successful matches, timeouts, pattern priority, output, and cancellation behavior retain their previous contracts.
+
+## 日本語
+
+- **C# の初回インデックスで宣言 regex の重複 probe を回避します** — declaration loop と recoverable-pattern 判定が、同じ物理 input に対する candidate-local な deterministic miss を共有するようになりました。multiline 結合 input、成功 match、timeout、pattern priority、出力、cancellation の従来契約は維持します。

changelog.d/unreleased/+csharp-instantiation-fallback.changed.md

@@ -0,0 +1,15 @@
+---
+category: internal
+affected:
+  - tests/CodeIndex.Tests/IndexCommandRunnerReferenceIndexBulkLoadTests.cs
+  - tests/CodeIndex.Tests/McpServerToolsCallTests.cs
+  - TESTING_GUIDE.md
+---
+
+## English
+
+- **Fresh reference integration contracts now observe the complete raw and schema surface** — the full-scan lifecycle fixture includes raw reference-line RETURNING work and verifies its deferred-index snapshot, while the MCP readiness-failure fixture includes the `reference_lines` table when comparing the persisted schema with the canonical reference-index set.
+
+## 日本語
+
+- **fresh reference のintegration契約がraw処理とschemaの全範囲を観測するようになりました** — full-scan lifecycle fixtureはraw reference-line RETURNING処理とその遅延index snapshotを含め、MCP readiness失敗fixtureは永続化済みschemaをcanonical reference-index集合と比較するときに`reference_lines` tableも対象にします。

changelog.d/unreleased/+csharp-recoverable-regex-cache.changed.md

@@ -0,0 +1,19 @@
+---
+category: changed
+affected:
+  - src/CodeIndex/Database/DbWriter.BatchParameters.cs
+  - src/CodeIndex/Database/DbWriter.ChunkSymbolBatches.cs
+  - src/CodeIndex/Database/DbWriter.Issues.cs
+  - src/CodeIndex/Database/DbWriter.ReferenceSql.cs
+  - tests/CodeIndex.Tests/DatabaseTests.cs
+  - tests/CodeIndex.Tests/FreshReferenceResolutionTests.cs
+  - DEVELOPER_GUIDE.md
+---
+
+## English
+
+- **Index write batches now use compact SQLite numeric parameter slots across every persistence table** — chunk, symbol, issue, reference-line, and reference inserts share a one-origin `?1` through `?N` contract, reducing parameter-resolution work during full indexing while retaining the 32-parameter caller-transaction cap, exact tail statements, prepared-command reuse, NULL handling, and fresh-reference semantics.
+
+## 日本語
+
+- **インデックス書き込みbatchが全永続化tableでcompactなSQLite numeric parameter slotを使うようになりました** — chunk、symbol、issue、reference-line、reference insertを1-originの`?1`〜`?N`契約へ統一し、32 parameterのcaller transaction上限、正確なtail statement、prepared command再利用、NULL処理、fresh reference semanticsを保ったまま、full index中のparameter解決処理を削減します。