Fix FileIndexer race and symlink handling (#1654 #1655 #1656 #1711)

Author: Widthdom

DEVELOPER_GUIDE.md

@@ -114,13 +114,13 @@ The lock files for projects with zero direct `PackageReference` entries (e.g. `t
 ### Indexing pipeline
 
 ```
-Directory scan / shared path filter (built-in skip lists + `.gitignore` / `.cdidxignore` + reparse/Windows Hidden/System attribute pruning)
+Directory scan / shared path filter (built-in skip lists + `.gitignore` / `.cdidxignore` + directory symlink policy + reparse/Windows Hidden/System attribute pruning)
   → Parallel extraction workers (`--parallelism`, `CDIDX_INDEX_PARALLELISM`; default CPU count capped at 16) read UTF-8, split chunks, extract symbols/references, and validate content
   → Single SQLite writer checks unchanged-file reuse, UPSERTs file records, runs post-extraction hooks, and inserts chunks + symbols + references + issues in per-file transactions
   → Populate FTS5 index
 ```
 
-Scoped `--files` / `--commits` refreshes reuse the same path filter as full scans. Before scanning a nested project root, `FileIndexer` loads ignore files from the resolved ignore-rule root through each existing ancestor directory down to the project root's parent, then loads the project directory's own rules during the normal walk. Within each directory, `FileIndexer` loads `.gitignore` before `.cdidxignore`, appends both rule sets in that order, and honors later `!` patterns as re-includes. If an ancestor ignore directory cannot be read, scanning fails closed with a scan error instead of silently skipping those rules; `ScanFilesResult.AncestorIgnoreDirectories` records the resolved ancestor list for troubleshooting. If a commit-scoped refresh includes `.gitignore` or `.cdidxignore` changes, `IndexCommandRunner` falls back to a full scan so newly ignored files are purged safely. Malformed ignore lines are reported as scan errors and skipped instead of aborting the whole run. On Windows, files and directories with Hidden or System attributes are rejected before language detection; clear those attributes before indexing project-owned sources because ignore rules cannot re-include them.
+Scoped `--files` / `--commits` refreshes reuse the same path filter as full scans. Before scanning a nested project root, `FileIndexer` loads ignore files from the resolved ignore-rule root through each existing ancestor directory down to the project root's parent, then loads the project directory's own rules during the normal walk. Within each directory, `FileIndexer` loads `.gitignore` before `.cdidxignore`, appends both rule sets in that order, and honors later `!` patterns as re-includes. If an ancestor ignore directory cannot be read, scanning fails closed with a scan error instead of silently skipping those rules; `ScanFilesResult.AncestorIgnoreDirectories` records the resolved ancestor list for troubleshooting. If a commit-scoped refresh includes `.gitignore` or `.cdidxignore` changes, `IndexCommandRunner` falls back to a full scan so newly ignored files are purged safely. Malformed ignore lines are reported as scan errors and skipped instead of aborting the whole run. Directory symlinks default to `--follow-symlinks none`; `internal` follows only targets that resolve under the workspace root, and `all` preserves the broad historical behavior. Dangling symlinks are counted and warned separately. On Windows, files and directories with Hidden or System attributes are rejected before language detection; clear those attributes before indexing project-owned sources because ignore rules cannot re-include them.
 
 Incremental refreshes that mutate `fts_chunks` increment `codeindex_meta.fts_incremental_writes_since_optimize`. When the counter reaches `DbWriter.DefaultFtsOptimizeIncrementalWriteThreshold`, the update path runs `INSERT INTO fts_chunks(fts_chunks) VALUES('optimize')`, resets the counter, and stamps `fts_last_optimized_at`. Users can run the same maintenance directly with `cdidx optimize --db <path>` or `cdidx index <projectPath> --optimize`; this may briefly hold the writer lock on large indexes.
 

changelog.d/unreleased/1654.fixed.md

@@ -0,0 +1,16 @@
+---
+category: fixed
+issues:
+  - 1654
+affected:
+  - src/CodeIndex/Indexer/Scanning/FileIndexer.cs
+  - tests/CodeIndex.Tests/FileIndexerTests.cs
+---
+
+## English
+
+- **File reads now retry once when mtime changes during indexing (#1654)** — `FileIndexer` rechecks `LastWriteTimeUtc` after reading content and retries once before persisting metadata, reducing stale mtime/content races.
+
+## 日本語
+
+- **index 中に mtime が変わったファイル読み取りを 1 回 retry するようになりました (#1654)** — `FileIndexer` は content 読み取り後に `LastWriteTimeUtc` を再確認し、metadata 保存前に 1 回 retry することで stale mtime/content race を減らします。

changelog.d/unreleased/1655.fixed.md

@@ -0,0 +1,16 @@
+---
+category: fixed
+issues:
+  - 1655
+affected:
+  - src/CodeIndex/Indexer/Scanning/FileIndexer.cs
+  - tests/CodeIndex.Tests/FileIndexerTests.cs
+---
+
+## English
+
+- **Mid-scan deletes remain non-fatal purge candidates (#1655)** — files that disappear during scan probing are recorded as skipped non-indexable paths with warnings, preserving directory purge authority instead of leaving orphan rows behind.
+
+## 日本語
+
+- **scan 中に削除されたファイルを非 fatal な purge 候補として扱います (#1655)** — probe 中に消えたファイルは warning 付きの non-indexable path として記録され、directory purge の authority を失わず orphan row を残しにくくなります。

changelog.d/unreleased/1656.fixed.md

@@ -0,0 +1,18 @@
+---
+category: fixed
+issues:
+  - 1656
+affected:
+  - src/CodeIndex/Indexer/Scanning/FileIndexer.cs
+  - src/CodeIndex/Cli/IndexCommandRunner.FullScan.cs
+  - src/CodeIndex/Cli/JsonOutputContracts.cs
+  - tests/CodeIndex.Tests/FileIndexerTests.cs
+---
+
+## English
+
+- **Dangling symlinks are now reported distinctly (#1656)** — directory symlink targets that cannot be resolved are warned as dangling symlinks and counted as `dangling_symlinks_skipped` in full-scan JSON summaries.
+
+## 日本語
+
+- **dangling symlink を個別に報告するようになりました (#1656)** — 解決できない directory symlink target は dangling symlink として warning され、full-scan JSON summary の `dangling_symlinks_skipped` に計上されます。

changelog.d/unreleased/1711.added.md

@@ -0,0 +1,20 @@
+---
+category: added
+issues:
+  - 1711
+affected:
+  - src/CodeIndex/Indexer/Scanning/FileIndexer.cs
+  - src/CodeIndex/Cli/IndexCommandRunner.Parse.cs
+  - src/CodeIndex/Cli/ConsoleUi.cs
+  - src/CodeIndex/Cli/IndexWatchRunner.cs
+  - DEVELOPER_GUIDE.md
+  - tests/CodeIndex.Tests/FileIndexerTests.cs
+---
+
+## English
+
+- **Added `--follow-symlinks none|internal|all` for directory scans (#1711)** — indexing now defaults to not following directory symlinks, can opt into workspace-internal targets, or can opt into all targets explicitly.
+
+## 日本語
+
+- **directory scan 向けに `--follow-symlinks none|internal|all` を追加しました (#1711)** — indexing は既定で directory symlink を辿らず、workspace 内 target のみ、または全 target を明示 opt-in できます。

src/CodeIndex/Cli/ConsoleUi.cs

@@ -60,7 +60,7 @@ public static class ConsoleUi
 
     private static readonly (string Command, string Usage)[] CommandUsageLines =
     [
-        ("index", "cdidx index <projectPath> [--db <path>] [--rebuild] [--optimize] [--verbose] [--dry-run] [--force] [--quiet] [--json] [--duration-format <auto|seconds|hms>] [--max-file-bytes <bytes>] [--include-symbol-kind <kind>[,<kind>]] [--exclude-symbol-kind <kind>[,<kind>]] [--watch [--debounce <ms>]]"),
+        ("index", "cdidx index <projectPath> [--db <path>] [--rebuild] [--optimize] [--verbose] [--dry-run] [--force] [--quiet] [--json] [--duration-format <auto|seconds|hms>] [--max-file-bytes <bytes>] [--follow-symlinks <none|internal|all>] [--include-symbol-kind <kind>[,<kind>]] [--exclude-symbol-kind <kind>[,<kind>]] [--watch [--debounce <ms>]]"),
         ("hooks", "cdidx hooks <install|uninstall|status> [--project <path>] [--force] [--json]"),
         ("backfill-fold", "cdidx backfill-fold [--db <path>] [--json]"),
         ("optimize", "cdidx optimize [--db <path>] [--json]"),
@@ -776,6 +776,7 @@ private static void PrintFlagReference(Action<string> WriteHelpLine)
         Console.WriteLine("  --duration-format <format> Index elapsed time format: `auto` (default), `seconds`, or `hms`; JSON keeps raw elapsed_ms");
         WriteHelpLine("  --max-file-bytes <bytes>  Index only files up to this size (default: 4MiB; also honors CDIDX_MAX_FILE_BYTES; accepts K/M/G suffixes)");
         WriteHelpLine("  --parallelism <n>         Full-scan extraction workers (default: CPU count capped at 16; also honors CDIDX_INDEX_PARALLELISM)");
+        WriteHelpLine("  --follow-symlinks <mode>  Directory symlink policy: none (default), internal, or all");
         WriteHelpLine("  --include-symbol-kind <kind>[,<kind>]  Keep only matching symbol kinds during indexing");
         WriteHelpLine("  --exclude-symbol-kind <kind>[,<kind>]  Drop matching symbol kinds during indexing");
         Console.WriteLine("  --commits <id> [id ...]    Update only files changed in the specified git commits (preferred after commits)");

src/CodeIndex/Cli/IndexCommandRunner.DryRun.cs

@@ -15,7 +15,7 @@ private static int RunDryRun(
         CancellationToken cancellationToken)
     {
         var projectPath = options.ProjectPath!;
-        var dryIndexer = new FileIndexer(projectPath, ignoreCase, ignoreRuleRoot, options.MaxFileSizeBytes);
+        var dryIndexer = new FileIndexer(projectPath, ignoreCase, ignoreRuleRoot, options.MaxFileSizeBytes, directoryIgnoreCaseProbe: null, symlinkPolicy: options.SymlinkPolicy);
         IReadOnlyList<string> dryCandidates;
         var errorList = new List<CliJsonMessage>();
         var dryScanErrorKeys = new HashSet<string>(StringComparer.Ordinal);

src/CodeIndex/Cli/IndexCommandRunner.FullScan.cs

@@ -1325,6 +1325,7 @@ void StopJsonHeartbeat()
                     FilesScanned = files.Count,
                     FilesSkipped = skipped,
                     FilesPurged = purged,
+                    DanglingSymlinksSkipped = scanResult.DanglingSymlinks.Count,
                     Warnings = warnings,
                     Errors = errors,
                     SymbolsDroppedByKindFilter = symbolsDroppedByKindFilter,
@@ -1370,6 +1371,7 @@ void StopJsonHeartbeat()
             Console.WriteLine(ConsoleUi.FormatSummaryLine("Symbols", $"{totalSymbols:N0}", indent: "  "));
             Console.WriteLine(ConsoleUi.FormatSummaryLine("Refs", $"{totalReferences:N0}", indent: "  "));
             if (skipped > 0) Console.WriteLine(ConsoleUi.FormatSummaryLine("Skipped", $"{skipped:N0} (unchanged)", indent: "  "));
+            if (scanResult.DanglingSymlinks.Count > 0) Console.WriteLine(ConsoleUi.FormatSummaryLine("Dangling symlinks", $"{scanResult.DanglingSymlinks.Count:N0} skipped", indent: "  "));
             if (options.Verbose && scanResult.UnknownExtensionFiles.Count > 0)
             {
                 Console.WriteLine($"  Unknown extension files: {scanResult.UnknownExtensionFiles.Count:N0}");

src/CodeIndex/Cli/IndexCommandRunner.Parse.cs

@@ -14,7 +14,7 @@ public static partial class IndexCommandRunner
     [
         "--db", "--data-dir", "--rebuild", "--verbose", "--json", "--dry-run", "--force",
         "--yes", "--watch", "--debounce", "--duration-format", "--max-file-bytes",
-        "--parallelism",
+        "--parallelism", "--follow-symlinks",
         "--commits", "--changed-between", "--files", "--solution", "--project",
         "--include-symbol-kind", "--exclude-symbol-kind", "--optimize", "--help",
         "--read-only", "--immutable",
@@ -41,6 +41,7 @@ public static IndexCommandOptions ParseArgs(string[] args)
         var durationFormat = DurationOutputFormat.Auto;
         long? maxFileSizeBytes = ReadMaxFileSizeBytesFromEnvironment();
         var parallelism = ReadIndexParallelismFromEnvironment();
+        var symlinkPolicy = FileIndexer.SymlinkPolicy.None;
         string? easterEgg = null;
         int spinnerFlagCount = 0;
         bool randomSpinner = false;
@@ -144,6 +145,12 @@ public static IndexCommandOptions ParseArgs(string[] args)
                 case var option when option.StartsWith("--parallelism=", StringComparison.Ordinal):
                     parallelism = ParseIndexParallelism(option["--parallelism=".Length..], parallelism, "--parallelism");
                     break;
+                case "--follow-symlinks" when i + 1 < args.Length:
+                    symlinkPolicy = ParseSymlinkPolicy(args[++i], symlinkPolicy, ref parseError);
+                    break;
+                case var option when option.StartsWith("--follow-symlinks=", StringComparison.Ordinal):
+                    symlinkPolicy = ParseSymlinkPolicy(option["--follow-symlinks=".Length..], symlinkPolicy, ref parseError);
+                    break;
                 case "--commits":
                     while (i + 1 < args.Length && !args[i + 1].StartsWith('-'))
                     {
@@ -289,10 +296,27 @@ public static IndexCommandOptions ParseArgs(string[] args)
             DurationFormat = durationFormat,
             MaxFileSizeBytes = maxFileSizeBytes,
             Parallelism = parallelism,
+            SymlinkPolicy = symlinkPolicy,
             SymbolKindFilter = SymbolKindFilter.Create(includeSymbolKinds, excludeSymbolKinds, symbolKindFilterError),
         };
     }
 
+    private static FileIndexer.SymlinkPolicy ParseSymlinkPolicy(string value, FileIndexer.SymlinkPolicy fallback, ref string? parseError)
+    {
+        switch (value.Trim().ToLowerInvariant())
+        {
+            case "none":
+                return FileIndexer.SymlinkPolicy.None;
+            case "internal":
+                return FileIndexer.SymlinkPolicy.Internal;
+            case "all":
+                return FileIndexer.SymlinkPolicy.All;
+            default:
+                parseError ??= $"invalid --follow-symlinks value '{value}': expected none, internal, or all";
+                return fallback;
+        }
+    }
+
     private static string BuildUnknownIndexOptionError(string token)
     {
         var name = TrimInlineValue(token);

src/CodeIndex/Cli/IndexCommandRunner.cs

@@ -239,7 +239,7 @@ internal static int Run(string[] indexArgs, JsonSerializerOptions jsonOptions, C
         AddToGitExclude(options.ProjectPath, dbPath);
 
         var writer = new DbWriter(db);
-        var indexer = new FileIndexer(options.ProjectPath, ignoreCase, ignoreRuleRoot, options.MaxFileSizeBytes);
+        var indexer = new FileIndexer(options.ProjectPath, ignoreCase, ignoreRuleRoot, options.MaxFileSizeBytes, directoryIgnoreCaseProbe: null, symlinkPolicy: options.SymlinkPolicy);
         var currentHotspotFamilyMarkerFingerprints = GetHotspotFamilyMarkerFingerprints(indexer);
         var projectRoot = Path.GetFullPath(options.ProjectPath!);
 
@@ -1172,6 +1172,7 @@ public sealed class IndexCommandOptions
     public DurationOutputFormat DurationFormat { get; init; } = DurationOutputFormat.Auto;
     public long? MaxFileSizeBytes { get; init; }
     public int Parallelism { get; init; } = IndexCommandRunner.DefaultIndexParallelism();
+    public FileIndexer.SymlinkPolicy SymlinkPolicy { get; init; } = FileIndexer.SymlinkPolicy.None;
     public SymbolKindFilter SymbolKindFilter { get; init; } = SymbolKindFilter.Empty;
 }