Merge pull request #5048 from Widthdom/codex/large-codebase-indexing-performance-20260809

Speed up large-codebase indexing

Author: Widthdom

DEVELOPER_GUIDE.md

@@ -0,0 +1,18 @@
+---
+category: changed
+affected:
+  - src/CodeIndex/Cli/IndexCommandRunner.FullScan.cs
+  - src/CodeIndex/Cli/IndexCommandRunner.FullScan.Readiness.cs
+  - src/CodeIndex/Cli/IndexCommandRunner.Update.cs
+  - src/CodeIndex/Cli/IndexCommandRunner.Update.Readiness.cs
+  - src/CodeIndex/Mcp/McpToolHandlers.Indexing.Execution.cs
+  - src/CodeIndex/Database/DbWriter.TypeScriptAugmentations.cs
+---
+
+## English
+
+- **Large TypeScript indexes avoid refreshing the reference graph twice** — CLI full scans, scoped updates, and MCP indexing now let a planned declaration-augmentation rebuild perform the single graph-finalization pass while keeping candidate-index staging active through that pass. Empty results still finalize deletions or an inherited graph pass, marker-only validation skips a whole-graph scan, late validation failures fall back before partial readiness, and `--memory-trace` attributes the graph sample after the augmentation work. Authoritative scans without TypeScript stamp the contract without rebuilding augmentation rows.
+
+## 日本語
+
+- **大規模 TypeScript index で reference graph を2回 refresh しないようになりました** — CLI full scan、scoped update、MCP indexing は、candidate index の遅延を維持したまま予定された declaration augmentation rebuild に1回だけの graph finalization を担当させます。結果が空でも削除または引き継いだgraph passは確定し、marker検証だけなら全graph走査を省きます。late validation failureではpartial readiness前にfallbackし、`--memory-trace` の graph sample はaugmentation作業後へ帰属させます。TypeScriptのないauthoritative scanはaugmentation rowを再構築せずcontractをstampします。

TESTING_GUIDE.md

@@ -0,0 +1,17 @@
+---
+category: changed
+affected:
+  - src/CodeIndex/Cli/IndexCommandRunner.Update.cs
+  - src/CodeIndex/Cli/IndexCommandRunner.Update.SecondaryIndexPreflight.cs
+  - src/CodeIndex/Database/ReferenceSecondaryIndexSql.cs
+  - src/CodeIndex/Database/ReferenceSecondaryIndexBulkLoadGuard.cs
+  - src/CodeIndex/Database/DbContext.SchemaInitialization.cs
+---
+
+## English
+
+- **Large reference graphs no longer maintain the reverse candidate-symbol index row by row during candidate materialization** — Fresh, rebuilt, and high-churn full-graph indexing keeps this query-only index during raw reference persistence and drops it only when a graph refresh will actually repopulate candidates, including a TypeScript augmentation-owned pass. A filtered stat preflight skips reference and hotspot secondary-index staging when high-cardinality scoped targets are unchanged or only a sparse subset is expected to mutate indexed state; staging begins only when that estimate crosses the existing bulk threshold. Marker-only work also avoids rebuilding the candidate index. The candidate primary key remains available and success, cancellation, and recovery still converge on the identical final schema.
+
+## 日本語
+
+- **大規模 reference graph の candidate 構築中に candidate-symbol reverse index を行ごとに保守しなくなりました** — fresh、rebuild、および高 churn の full-graph indexing は、この query 専用 index を raw reference persistence 中は維持し、TypeScript augmentation が担当する pass を含め、graph refresh が candidate を実際に再構築するときだけ外します。filtered stat preflight は高 cardinality scoped target が全て unchanged、または indexed state を変更し得る target が疎な場合に reference / hotspot secondary-index staging を省き、その見積もり件数が既存 bulk threshold を超えた場合だけ staging を開始します。marker-only work も candidate index の再構築を避けます。candidate primary key を維持したまま、成功・cancellation・recovery の全経路で同一の最終 schema に収束します。

changelog.d/unreleased/+coalesce-typescript-graph-refresh.changed.md

@@ -0,0 +1,15 @@
+---
+category: changed
+affected:
+  - src/CodeIndex/Cli/IndexCommandRunner.FullScan.ExtractionPipeline.cs
+  - src/CodeIndex/Cli/IndexCommandRunner.FullScan.ExtractionWorkers.cs
+  - src/CodeIndex/Indexer/Scanning/FileIndexer.cs
+---
+
+## English
+
+- **Parallel full scans start large tail files earlier** — extraction now skips metadata probes with a single worker or when all work fits in the first worker wave; otherwise it checks only a fixed-size suffix of at most 64 work items and starts known in-limit files largest-first, reducing the final worker-wave tail without adding a repository-wide metadata pass.
+
+## 日本語
+
+- **parallel full scanが末尾の大きなfileを早く開始します** — workerが1つ、または全workが最初のworker waveに収まる場合はmetadata probeを省き、それ以外では最大64件の固定長suffixだけを確認してsize取得済みで上限内のfileを大きい順に開始するため、repository全体のmetadata passを増やさず最終worker waveの長い尾を短縮します。

changelog.d/unreleased/+defer-reference-candidate-index.changed.md

@@ -0,0 +1,14 @@
+---
+category: changed
+affected:
+  - src/CodeIndex/Indexer/Symbols/SymbolExtractor.ExtractionPhases.cs
+  - src/CodeIndex/Indexer/Symbols/SymbolExtractor.JavaScriptScopes.cs
+---
+
+## English
+
+- **JavaScript and TypeScript symbol extraction reuses its sanitized snapshot** — module, supplemental-symbol, and private-scope analysis now share one column-preserving snapshot, avoiding a duplicate full-file lexical pass on scope-heavy files while retaining the flat-file pre-scan fast path.
+
+## 日本語
+
+- **JavaScript / TypeScript の symbol extraction が sanitized snapshot を再利用します** — module、supplemental symbol、private-scope 解析は列位置を保つ snapshot を共有し、flat file の pre-scan fast path を維持しながら scope の多い file で重複していた全file lexical passを避けます。

changelog.d/unreleased/+prioritize-full-scan-tail.changed.md

@@ -0,0 +1,19 @@
+---
+category: changed
+affected:
+  - src/CodeIndex/Database/HotspotReferenceAggregateSql.cs
+  - src/CodeIndex/Database/DbWriter.HotspotReferenceRefreshScope.cs
+  - src/CodeIndex/Database/DbContext.SchemaInitialization.cs
+  - src/CodeIndex/Database/DbContext.ReadMigrations.cs
+  - src/CodeIndex/Cli/IndexCommandRunner.FullScan.cs
+  - src/CodeIndex/Cli/IndexCommandRunner.Update.cs
+  - src/CodeIndex/Mcp/McpToolHandlers.Indexing.Execution.cs
+---
+
+## English
+
+- **Large indexing runs rebuild hotspot aggregate query indexes once instead of maintaining four trees row by row** — Full scans, rebuilds, high-churn scoped updates, and MCP bulk indexing now stage the cross-language hotspot indexes inside the existing aggregate refresh transaction when at least 64 dirty files account for most aggregate rows, including an empty pre-refresh aggregate during fresh/rebuild work. An exact probe bounded by dirty-row cardinality keeps the indexes for small or highly skewed updates, and success, cancellation, and failure preserve the same ready state and final schema.
+
+## 日本語
+
+- **大規模 indexing では hotspot aggregate の query index 4本を行ごとに保守せず、最後に1回だけ再構築するようになりました** — full scan、rebuild、高 churn の scoped update、MCP bulk indexing は、64件以上のdirty fileがaggregate rowの大半を占める場合に既存のaggregate refresh transaction内で全言語共通のhotspot indexを遅延し、fresh / rebuildでrefresh前aggregateが空の場合も対象にします。dirty-row cardinalityで上限を設けたexact probeにより小規模または偏りの大きいupdateではindexを維持し、成功・cancellation・失敗の各経路で同じready stateと最終schemaを保ちます。

changelog.d/unreleased/+reuse-javascript-typescript-sanitized-snapshot.changed.md

@@ -0,0 +1,13 @@
+---
+category: changed
+affected:
+  - src/CodeIndex/Database/DbWriter.Fts.cs
+---
+
+## English
+
+- **Large FTS rebuilds avoid repeated automatic segment merges before their final optimize** — The standard and trigram indexes now disable routine automerge only around each transactional rebuild, restore their prior settings before commit, and retain crisis merging as a safety valve. Cancellation and failure roll back the configuration together with the rebuild.
+
+## 日本語
+
+- **大規模 FTS rebuild では最終 optimize 前の自動 segment merge を繰り返さないようになりました** — standard / trigram index は各 transaction 内の rebuild 中だけ通常 automerge を無効化し、commit 前に以前の設定へ戻します。crisis merge は安全弁として維持し、cancellation / failure 時は設定と rebuild を一緒に rollback します。

changelog.d/unreleased/+stage-hotspot-aggregate-indexes.changed.md

@@ -113,6 +113,13 @@ private readonly record struct FullScanExtractionScheduling(
         bool Parallelize,
         string? Reason);
 
+    private readonly record struct FullScanExtractionTailCandidate(
+        int WorkOrdinal,
+        long? Length);
+
+    private const int FullScanExtractionTailWorkerWaves = 4;
+    internal const int MaxFullScanExtractionTailProbeCount = 64;
+
     private static FullScanExtractionPipelineResult
         RunFullScanExtractionPipeline(
             FullScanExtractionPipelineContext context)
@@ -170,6 +177,98 @@ private static FullScanExtractionScheduling
         return new FullScanExtractionScheduling(parallelize, reason);
     }
 
+    internal static int[] BuildFullScanExtractionTailSchedule(
+        int workItemCount,
+        int workerCount,
+        long maxFileSizeBytes,
+        Func<int, long?> getFileLength,
+        CancellationToken cancellationToken)
+    {
+        ArgumentOutOfRangeException.ThrowIfNegative(workItemCount);
+        ArgumentOutOfRangeException.ThrowIfLessThan(workerCount, 1);
+        ArgumentOutOfRangeException.ThrowIfNegative(maxFileSizeBytes);
+        ArgumentNullException.ThrowIfNull(getFileLength);
+        cancellationToken.ThrowIfCancellationRequested();
+        if (workerCount <= 1 || workItemCount <= workerCount)
+            return [];
+
+        // Dynamic claiming balances the main body of the scan, but a large file at the
+        // input tail otherwise starts in the final worker wave. Probe only a fixed-size
+        // suffix so the remedy never turns into an all-repository metadata pass.
+        // 本体はdynamic claimで均等化し、末尾だけを固定上限でsize順にして全件statを避ける。
+        var workerBound = workerCount >= MaxFullScanExtractionTailProbeCount / FullScanExtractionTailWorkerWaves
+            ? MaxFullScanExtractionTailProbeCount
+            : workerCount * FullScanExtractionTailWorkerWaves;
+        var tailCount = Math.Min(
+            workItemCount,
+            Math.Min(workerBound, MaxFullScanExtractionTailProbeCount));
+        var tailStart = workItemCount - tailCount;
+        var candidates = new FullScanExtractionTailCandidate[tailCount];
+        for (var tailIndex = 0; tailIndex < tailCount; tailIndex++)
+        {
+            cancellationToken.ThrowIfCancellationRequested();
+            var workOrdinal = tailStart + tailIndex;
+            long? length;
+            try
+            {
+                length = getFileLength(workOrdinal);
+            }
+            catch (Exception ex) when (
+                ex is IOException
+                    or UnauthorizedAccessException
+                    or NotSupportedException
+                    or ArgumentException
+                    or System.Security.SecurityException)
+            {
+                length = null;
+            }
+
+            var eligibleLength = length is >= 0
+                && length <= maxFileSizeBytes
+                    ? length
+                    : null;
+            candidates[tailIndex] = new FullScanExtractionTailCandidate(
+                workOrdinal,
+                eligibleLength);
+        }
+
+        Array.Sort(
+            candidates,
+            static (left, right) =>
+            {
+                if (left.Length.HasValue != right.Length.HasValue)
+                    return left.Length.HasValue ? -1 : 1;
+
+                if (left.Length.HasValue)
+                {
+                    var lengthComparison = right.Length.GetValueOrDefault().CompareTo(
+                        left.Length.GetValueOrDefault());
+                    if (lengthComparison != 0)
+                        return lengthComparison;
+                }
+
+                return left.WorkOrdinal.CompareTo(right.WorkOrdinal);
+            });
+
+        var schedule = new int[candidates.Length];
+        for (var index = 0; index < candidates.Length; index++)
+            schedule[index] = candidates[index].WorkOrdinal;
+        return schedule;
+    }
+
+    internal static int ResolveFullScanExtractionFileIndex(
+        IReadOnlyList<int>? extractionFileIndexes,
+        int workOrdinal)
+        => extractionFileIndexes == null
+            ? workOrdinal
+            : extractionFileIndexes[workOrdinal];
+
+    private static long? ReadFullScanExtractionFileLength(string filePath)
+    {
+        var info = new FileInfo(filePath);
+        return info.Exists ? info.Length : null;
+    }
+
     private static FullScanExtractionConsumerState
         ExecuteFullScanExtractionPipeline(
             FullScanExtractionPipelineContext context,
@@ -200,6 +299,18 @@ private static FullScanExtractionConsumerState
             new LazyDisposable<SymbolExtractionWorkerClient>(
                 () => new SymbolExtractionWorkerClient(
                     context.Options.MaxFileSizeBytes));
+        var extractionTailSchedule = parallelizeExtraction
+            ? BuildFullScanExtractionTailSchedule(
+                context.ExtractionWorkItemCount,
+                extractionWorkerCount,
+                context.Indexer.MaxFileSizeBytes,
+                workOrdinal => ReadFullScanExtractionFileLength(
+                    context.FileTargets[
+                        ResolveFullScanExtractionFileIndex(
+                            context.ExtractionFileIndexes,
+                            workOrdinal)].FilePath),
+                context.CancellationToken)
+            : [];
         var workers = StartFullScanExtractionWorkers(
             new FullScanExtractionWorkerContext
             {
@@ -213,6 +324,7 @@ private static FullScanExtractionConsumerState
                     context.ExtractionWorkItemCount,
                 ExtractionWorkerCount = extractionWorkerCount,
                 ParallelizeExtraction = parallelizeExtraction,
+                ExtractionTailSchedule = extractionTailSchedule,
                 CSharpWorkspace = context.GetCSharpWorkspace(),
                 CSharpWorkspaceFileSnapshots =
                     context.GetCSharpWorkspaceFileSnapshots(),

changelog.d/unreleased/+suppress-fts-rebuild-automerge.changed.md

@@ -18,6 +18,7 @@ private sealed class FullScanExtractionWorkerContext
         internal required int ExtractionWorkItemCount { get; init; }
         internal required int ExtractionWorkerCount { get; init; }
         internal required bool ParallelizeExtraction { get; init; }
+        internal required int[] ExtractionTailSchedule { get; init; }
         internal required CSharpStaticInterfaceWorkspaceSymbols CSharpWorkspace { get; init; }
         internal Dictionary<string, CSharpStaticInterfacePrepass.FileStatSnapshot>? CSharpWorkspaceFileSnapshots { get; init; }
         internal required PostExtractionHookRunner PostExtractionHooks { get; init; }
@@ -38,6 +39,7 @@ private static Task[] StartFullScanExtractionWorkers(
         var extractionWorkItemCount = context.ExtractionWorkItemCount;
         var extractionWorkerCount = context.ExtractionWorkerCount;
         var parallelizeExtraction = context.ParallelizeExtraction;
+        var extractionTailSchedule = context.ExtractionTailSchedule;
         var csharpWorkspace = context.CSharpWorkspace;
         var csharpWorkspaceFileSnapshots = context.CSharpWorkspaceFileSnapshots;
         var postExtractionHooks = context.PostExtractionHooks;
@@ -58,9 +60,13 @@ private static Task[] StartFullScanExtractionWorkers(
                     if (extractionIndex >= extractionWorkItemCount)
                         break;
 
-                    var fileIndex = extractionFileIndexes == null
-                        ? extractionIndex
-                        : extractionFileIndexes[extractionIndex];
+                    var tailStart = extractionWorkItemCount - extractionTailSchedule.Length;
+                    var workOrdinal = extractionIndex >= tailStart
+                        ? extractionTailSchedule[extractionIndex - tailStart]
+                        : extractionIndex;
+                    var fileIndex = ResolveFullScanExtractionFileIndex(
+                        extractionFileIndexes,
+                        workOrdinal);
                     var target = fileTargets[fileIndex];
                     var filePath = target.FilePath;
                     var relativeFilePath = target.RelativePath;