Raise the index file size cap to 500KB, and say when a file is skipped - #53
Merged
Conversation
…skipped The silence was the defect; the number was secondary. A file over the limit was dropped with `return`, so a user whose symbols were missing from find_references had nothing to go on. Same shape as the scan truncation in #40, smaller in blast radius: raising the limit only moves the cliff, naming it is what makes falling off it observable. 100 KB was also low for real source. This repo's own p99 is 41 KB — only one file exceeds 100 KB and it is a test — but generated TypeScript users legitimately want indexed (GraphQL types, protobuf output, OpenAPI clients, large const tables) routinely passes it. The pathological files a cap exists to stop, minified bundles and vendored trees, mostly sit in directories INDEX_EXCLUDE_DIRS already removes, so the cap was doing less protective work than its size implied. Measured with tree-sitter, roughly linear: 136 KB -> 81ms 272 KB -> 130ms 680 KB -> 289ms 500 KB costs ~215ms worst case against a full index. Past that, skipping is right — a multi-MB `.ts` is almost certainly generated or minified and its symbols would be noise. Found while reconciling the live graph against a direct tree-sitter run: every per-file count matched, and the only aggregate gaps were __corpus__ fixtures (#52) and one 136 KB file silently over this limit. Mutation-checked independently: reverting the cap fails two tests, silencing the log fails one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
✅ All checks passed
Posted by SideCarAI-Bot |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The silence was the defect; the number was secondary.
The silence
A file over the limit was dropped without a word, so a user whose symbols were
missing from
find_referenceshad no way to discover why. Same shape as thescan truncation in #40, smaller in blast radius.
Raising the limit only moves the cliff. Naming it is what makes falling off it
observable — so this does both, and the logging is the half I would keep if only
one were possible.
Why 100 KB was also too low
This repo's own distribution argues against urgency — p99 is 41 KB and exactly
one file exceeds 100 KB, a test file. The case is other people's repos:
generated TypeScript that users legitimately want indexed (GraphQL types,
protobuf output, OpenAPI clients, large const tables) routinely passes 100 KB.
Meanwhile the pathological files a cap exists to stop — minified bundles,
vendored trees — mostly sit in directories
INDEX_EXCLUDE_DIRSalready removes,so the cap was doing less protective work than its size implied.
Cost, measured
Tree-sitter, roughly linear:
500 KB costs ~215ms worst case against a full index. Past that, skipping is
genuinely right — a multi-MB
.tsis almost certainly generated or minified andits symbols would be noise, so this keeps a cap rather than removing one.
How it was found
Reconciling the live symbol graph against a direct tree-sitter run after the
tree-sitter fixes landed. Every per-file count matched exactly; the only
aggregate gaps were
__corpus__fixtures (#52) and one 136 KB file silentlyover this limit. The arithmetic closed to the symbol once both were accounted
for — no extraction defect, just two file-set differences, one of which was
invisible.
Testing
Mutation-checked independently: reverting the cap to 100 KB fails two tests,
making the skip silent again fails one.
npm run check: green.🤖 Generated with Claude Code