Speed up indexing for large workspaces

[Widthdom]

Summary

• Defer and stage reference indexes around bulk persistence and graph finalization.
• Materialize reusable C# graph identity and reference facts while shrinking the reference-index working set.
• Parallelize authoritative C# update extraction.
• Skip impossible reference-extraction patterns across C#, Java, Kotlin, JavaScript, TypeScript, Go, Dockerfile, markup/XAML, Terraform, JSON, and GitHub Actions paths.
• Replace direct regex match enumeration with lazy concrete struct enumerators that avoid enumerator allocations.
• Compute normalized content facts once and reuse them across chunk construction and validation.
• Preserve project-marker family scopes across mixed case-sensitive and case-insensitive filesystem boundaries without post-scan probes.

Why

Large workspace updates were spending substantial time maintaining secondary reference indexes during row-by-row writes, rebuilding facts that were already available, scanning regexes that could not match, and serially extracting authoritative C# updates. Normalization-derived facts were also recomputed separately by chunking and validation.

These changes move expensive work out of the per-row hot path, reuse already-materialized facts, parallelize independent extraction, and add conservative early exits while preserving indexing and reference-graph contracts.

Impact

On the same large-workspace update scenario, the observed duration moved from 302.547 seconds to a three-run median of 219.339 seconds (about 27.5% faster). The final HEAD run completed in 223.424 seconds for 1,130 updated files with a peak working set of about 526 MiB.

The exact file count changed slightly between runs, so these figures are indicative rather than a strict microbenchmark comparison.

Validation

• Debug and Release solution builds: 0 warnings, 0 errors.
• net8.0 full suite at ff9397594: 11,141 passed, 7 skipped, 0 failed.
• Final marker-scope change: all FileIndexerTests passed on net8.0 and net9.0 (633 each).
• Final HEAD net9.0 full suite: 10,678 passed, 411 skipped, 0 failed.
• BoundedRegex and normalized-content targeted correctness/allocation suites passed on both target frameworks.
• Changelog validator: 13 fragments valid.
• Repository self-index: fresh at HEAD, complete reference graph, 0 extraction-cap hits.
• Independent adversarial review: no remaining blocking or actionable findings.

Collaborator checklist

• The change follows the existing code style.
• The change includes tests when behavior changes.
• The change includes a changelog fragment when user-visible behavior changes.
• Documentation has been updated when needed.