perf(core): Automated performance tuning by Claude - #1428
Draft
yamadashy wants to merge 47 commits into
Draft
Conversation
… check Move metrics worker pool initialization before file search so that gpt-tokenizer loading overlaps with I/O-bound operations (globby file search, file reading) instead of contending with CPU-bound security check workers. Also parallelize file and directory globby calls in searchFiles, and increase metrics batch size from 10 to 25 to reduce IPC overhead. Changes: - packager.ts: Create metrics task runner before searchFiles instead of after, using getProcessConcurrency() to determine thread count since file count is not yet known - fileSearch.ts: Run file search and directory search globby calls concurrently via Promise.all when includeEmptyDirectories is enabled - calculateSelectiveFileMetrics.ts: Increase METRICS_BATCH_SIZE from 10 to 25, reducing IPC round-trips from 100 to 40 for 1000 files Benchmark results (interleaved A/B, 10 cold-start pairs on 4-core machine, repomix repo with ~1000 files): - Baseline average: 2196ms - Optimized average: 2098ms - Average improvement: 97ms (4.4%) - Median improvement: 99ms (4.5%) - 8 out of 10 pairs show improvement The improvement comes from eliminating thread contention: previously, 4 metrics warmup threads (loading gpt-tokenizer) competed with 2 security check threads (running secretlint) for 4 CPU cores. By starting warmup earlier during I/O-bound phases, warmup completes before the CPU-bound security check begins. https://claude.ai/code/session_01SBsGNwf5W7iFj16aNdurNj
Increase worker batch sizes for token counting to reduce per-batch IPC scheduling overhead, which was the dominant cost in the metrics phase. Changes: - calculateSelectiveFileMetrics: Use adaptive batch size (50 for >100 files, 10 for smaller sets) instead of fixed size 10. For 1000 files, this reduces batches from 100 to 20, cutting ~80ms of IPC overhead. - calculateOutputMetrics: Increase TARGET_CHARS_PER_CHUNK from 200K to 500K, reducing chunks from ~20 to ~8 for a 4MB output while still providing enough parallelism across worker threads. Benchmark results (repomix on itself, 997 files, ~4MB output, 10 runs): - Before: mean 1540ms, median 1552ms - After: mean 1448ms, median 1456ms - Improvement: ~92ms (6.0%) mean, ~96ms (6.2%) median The per-batch IPC scheduling overhead (~1ms per round-trip) was the bottleneck rather than serialization cost. Fewer, larger batches reduce total round-trips while maintaining good worker utilization. https://claude.ai/code/session_01SprjopvDAiY84WdFQELCjZ
…avor of adaptive batch sizing)
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
⚡ Performance Benchmark
Details
Historybac879b perf(core): Reduce startup module loading by deferring heavy imports
eae6adc perf(core): Size metrics worker pool by estimated tokenization needs
3eb892e perf(core): Size metrics worker pool by estimated tokenization needs
ca781d3 Merge remote-tracking branch 'origin/perf/auto-perf-tuning-0406' into perf/auto-perf-tuning-0406
f590444 Merge remote-tracking branch 'origin/perf/auto-perf-tuning-0406' into perf/auto-perf-tuning-0406
6469530 perf(core): Defer metrics worker pool creation until after file search
c3d6b7c perf(output): Replace O(N²) split output algorithm with O(N) estimation-based approach
9357d02 perf(security): Remove pre-filter keywords for disabled secretlint rules
105219e perf(security): Remove pre-filter keywords for disabled secretlint rules
082f8b4 perf(core): Lazy-load handlebars, fast-xml-builder, and @clack/prompts to reduce startup cost
fb0a861 perf(core): Lazy-load handlebars, fast-xml-builder, and @clack/prompts to reduce startup cost
539ac5d perf(core): Lazy-load handlebars, fast-xml-builder, and @clack/prompts to reduce startup cost
24c24d5 perf(core): Lazy-load handlebars, fast-xml-builder, and @clack/prompts to reduce startup cost
b6ccc3f perf(output): Skip calculateFileLineCounts for non-skill output paths
12b4813 perf(output): Skip calculateFileLineCounts for non-skill output paths
3600dbe perf(core): Add line-length pre-filter to skip base64 scan on short-line files
fa209c5 test(core): Add unit tests for prefetchFileChangeCounts cache behavior
ebfed92 perf(core): Combine picomatch patterns into single regex and lazy-load minimatch
fd6b625 perf(core): Pre-load BPE data on main thread to eliminate redundant per-worker file I/O
0c01d91 perf(core): Skip tokenizing small files when tokenCountTree has a threshold
761b5fd perf(core): Skip tokenizing small files when tokenCountTree has a threshold
68b20dc perf(core): Skip tokenizing small files when tokenCountTree has a threshold
2cfa7dc perf(security): Replace 50 sequential includes() with single combined regex in mightContainSecret
07d5c89 perf(security): Replace sequential keyword scanning with single-pass regex for 3.5x faster pre-filter
1fc197b perf(security): Replace sequential keyword scanning with single-pass regex for 3.5x faster pre-filter
f5d02d7 Merge remote-tracking branch 'origin/main' into perf/auto-perf-tuning-0406
0e7655e perf(core): Skip tokenizing all files when tokenCountTree is disabled
a3e1452 perf(security): Avoid security worker pool for small item counts to reduce CPU contention
ccc68c2 perf(cli): Pre-create metrics worker pool before pack() for earlier BPE warmup
3dd8bcc perf(security): Skip expensive lintSource() for files without secret keywords
218daeb perf(security): Skip expensive lintSource() for files without secret keywords
d7cacc9 Merge remote perf/auto-perf-tuning-0406 and resolve conflicts in calculateSelectiveFileMetrics
5925c02 fix(core): Remove @ts-expect-error for picomatch import
fe2ecbe fix(core): Move try block to cover metrics worker pool lifecycle 41c87af Merge remote-tracking branch 'origin/perf/auto-perf-tuning-0406' into perf/auto-perf-tuning-0406 c794b70 Merge remote perf/auto-perf-tuning-0406 and resolve conflicts
48f3311 perf(core): Lazy-load globby and avoid globby directory scan on git fast path
29fd39b Merge remote-tracking branch 'origin/perf/auto-perf-tuning-0406' into perf/auto-perf-tuning-0406
27e7b88 [autofix.ci] apply automated fixes
a3bc119 perf(core): Use git ls-files for file search in git repos (~7% faster)
02e2d10 perf(core): Estimate output tokens from file tokens instead of counting full output
5e4e64f perf(core): Reduce pipeline overhead with non-blocking cleanup and lazy computation
a94c618 perf(core): Use synchronous file reads in collectFiles for ~7% speedup
72dc46d perf(core): Use synchronous file reads in collectFiles for ~7% speedup
dcc4bf5 perf(core): Use synchronous file reads in collectFiles for ~7% speedup
3ef790f Merge remote-tracking branch 'origin/perf/auto-perf-tuning-0406' into perf/auto-perf-tuning-0406
562635d Merge remote-tracking branch 'origin/perf/auto-perf-tuning-0406' into perf/auto-perf-tuning-0406
3e90eb7 Merge remote branch perf/auto-perf-tuning-0406 (resolve conflict in favor of adaptive batch sizing)
|
Deploying repomix with
|
| Latest commit: |
1d60d6a
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://c09cdac9.repomix.pages.dev |
| Branch Preview URL: | https://perf-auto-perf-tuning-0406.repomix.pages.dev |
Codecov Report❌ Patch coverage is ❌ Your patch check has failed because the patch coverage (77.49%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #1428 +/- ##
==========================================
- Coverage 87.05% 84.78% -2.27%
==========================================
Files 116 119 +3
Lines 4433 4810 +377
Branches 1029 1111 +82
==========================================
+ Hits 3859 4078 +219
- Misses 574 732 +158 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request optimizes performance by parallelizing file and directory searches, increasing chunk sizes for tokenization to reduce IPC overhead, and implementing dynamic batching for file metrics. Additionally, the metrics worker pool initialization was moved earlier to overlap with I/O tasks. Feedback includes addressing a potential resource leak by moving the try...finally block, ensuring consistent error handling for directory searches, and avoiding brittle logic when calculating worker thread counts.
Comment on lines
+84
to
+87
| const { taskRunner: metricsTaskRunner, warmupPromise: metricsWarmupPromise } = deps.createMetricsTaskRunner( | ||
| processConcurrency * 100, | ||
| config.tokenCount.encoding, | ||
| ); |
Contributor
There was a problem hiding this comment.
The metrics worker pool is initialized here, but the try...finally block that ensures its cleanup (via metricsTaskRunner.cleanup()) doesn't start until line 117. If an error occurs in the intervening code (e.g., in searchFiles at line 93 or sortPaths at line 108), the worker pool will not be cleaned up, leading to a resource leak. Consider moving the start of the try block to immediately follow the worker pool initialization.
Owner
Author
There was a problem hiding this comment.
Fixed in fe2ecbe — moved the try block to start immediately after metrics worker pool creation, ensuring cleanup via finally covers the entire lifecycle including searchFiles and sortPaths.
Generated by Claude Code
Comment on lines
+212
to
+220
| const emptyDirSearchPromise = config.output.includeEmptyDirectories | ||
| ? globby(includePatterns, { | ||
| ...baseGlobbyOptions, | ||
| onlyDirectories: true, | ||
| }).then((directories) => { | ||
| logger.debug(`[empty dirs] Found ${directories.length} directories`); | ||
| return findEmptyDirectories(rootDir, directories, adjustedIgnorePatterns); | ||
| }) | ||
| : Promise.resolve([] as string[]); |
Contributor
There was a problem hiding this comment.
The directory search globby call lacks the specific error handling for EPERM and EACCES that is present in the file search. For consistency and better error reporting, consider adding a similar .catch block to transform these errors into PermissionError.
Suggested change
| const emptyDirSearchPromise = config.output.includeEmptyDirectories | |
| ? globby(includePatterns, { | |
| ...baseGlobbyOptions, | |
| onlyDirectories: true, | |
| }).then((directories) => { | |
| logger.debug(`[empty dirs] Found ${directories.length} directories`); | |
| return findEmptyDirectories(rootDir, directories, adjustedIgnorePatterns); | |
| }) | |
| : Promise.resolve([] as string[]); | |
| const emptyDirSearchPromise = config.output.includeEmptyDirectories | |
| ? globby(includePatterns, { | |
| ...baseGlobbyOptions, | |
| onlyDirectories: true, | |
| }) | |
| .then((directories) => { | |
| logger.debug(`[empty dirs] Found ${directories.length} directories`); | |
| return findEmptyDirectories(rootDir, directories, adjustedIgnorePatterns); | |
| }) | |
| .catch((error: unknown) => { | |
| const code = (error as NodeJS.ErrnoException | { code?: string })?.code; | |
| if (code === 'EPERM' || code === 'EACCES') { | |
| throw new PermissionError( | |
| `Permission denied while scanning directory. Please check folder access permissions for your terminal app. path: ${rootDir}`, | |
| rootDir, | |
| ); | |
| } | |
| throw error; | |
| }) | |
| : Promise.resolve([] as string[]); |
| // threads and is negligible compared to the contention savings for typical repos. | ||
| const processConcurrency = deps.getProcessConcurrency(); | ||
| const { taskRunner: metricsTaskRunner, warmupPromise: metricsWarmupPromise } = deps.createMetricsTaskRunner( | ||
| processConcurrency * 100, |
Contributor
There was a problem hiding this comment.
Using processConcurrency * 100 to force the worker pool to allocate one thread per core is brittle as it relies on the internal TASKS_PER_THREAD constant (currently 100) in processConcurrency.ts. If that constant changes, this logic may no longer achieve the intended thread count. It would be better to expose a way to explicitly request maximum concurrency or use a shared constant.
Add a fast linear pre-check (hasLongBase64Run) before applying the
expensive standalone base64 regex pattern. Most source code files
contain no sequences of 256+ base64 characters, so the pre-check
returns false in O(n) with a simple charCodeAt loop, avoiding the
costly regex engine invocation entirely.
Also gate the data URI regex behind a cheap `content.includes(';base64,')`
check, skipping it for the majority of files.
action: add fast-path pre-check to truncateBase64Content
reason: CPU profiling showed the standalone base64 regex consuming ~248ms
of main-thread CPU time per CLI run (10.6% of total), despite only 3
out of ~1000 files actually containing base64 content
decision: use a linear character scan (charCodeAt) for the pre-check rather
than a simpler regex, because V8's regex engine has non-trivial setup
cost that dominates for files where the pattern never matches
Benchmark (15 runs each, repomix self-pack with 1018 files):
Baseline: avg 2351ms, median 2352ms, p10-p90: 2285-2426ms
Optimized: avg 2231ms, median 2237ms, p10-p90: 2149-2313ms
Improvement: ~120ms (5.1% avg), with optimized p90 < baseline median
https://claude.ai/code/session_01FmGrVUJY2giodz6mNFu6cF
… perf/auto-perf-tuning-0406
Move the git log subprocess used for sortByChanges file ordering from the critical path inside generateOutput to run concurrently with file collection and other git operations in the pack pipeline. Previously, sortOutputFiles spawned `git log --name-only` and `git --version` sequentially inside generateOutput, blocking template rendering. Since sortByChanges defaults to true, this affected every CLI run. By pre-fetching file change counts into the module-level cache via prefetchFileChangeCounts in the existing Promise.all alongside collectFiles/getGitDiffs/getGitLogs, sortOutputFiles finds the result already cached and skips the git subprocess entirely. ## Changes - Add `prefetchFileChangeCounts` to outputSort.ts — warms the module-level cache early in the pipeline (fire-and-forget) - Call it in packager.ts Promise.all alongside file collection and git operations - No signature changes to downstream functions — the existing cache handles deduplication transparently ## Benchmark (repomix on its own repo, ~1000 files, 5 runs each) - Baseline (main): ~1.99s avg (1.934, 1.960, 1.966, 2.016, 2.078) - Optimized: ~1.59s avg (1.513, 1.541, 1.559, 1.631, 1.696) - Improvement: ~400ms = ~20% faster https://claude.ai/code/session_01Qhaod6pvCsxrn6tmJEmcS2
… perf/auto-perf-tuning-0406
yamadashy
force-pushed
the
perf/auto-perf-tuning-0406
branch
from
April 7, 2026 06:05
dcc4bf5 to
72dc46d
Compare
Switch file collection from async pooled reads (fs.promises.readFile with concurrency 50) to synchronous reads (fs.readFileSync). This eliminates per-file event loop scheduling overhead and libuv thread pool round-trips that bottleneck async reads on the default 4-thread pool. For typical repos with hundreds to thousands of mostly small source files, sync reads are ~2-3x faster than async pooled reads. The main thread is not doing other useful work during file collection (security check and metrics happen later), so blocking the event loop is acceptable. Non-UTF-8 files that fail the fast sync TextDecoder path fall back to the original async readRawFile with jschardet/iconv-lite encoding detection. ## Benchmark (repomix repo, 1011 files, 5.2MB output, 20 runs each) A/B test (10 alternating pairs): Average: 2.054s → 1.910s (7.0% faster) Median: 2.063s → 1.901s (7.9% faster) Standalone runs (20 each): Baseline p50: 1.947s Optimized p50: 1.771s–1.850s (5.0–9.0% faster) ## Changes - fileRead.ts: Add readRawFileSync using fs.readFileSync + isBinaryFileSync - fileCollect.ts: Replace async promisePool with sync loop, async fallback for non-UTF-8 files - Updated unit and integration tests for new sync-first interface https://claude.ai/code/session_01WbseR3yGyhgjRGCfobTY5A
yamadashy
force-pushed
the
perf/auto-perf-tuning-0406
branch
from
April 7, 2026 06:08
72dc46d to
a94c618
Compare
…zy computation
Three independent optimizations targeting the pack pipeline:
1. Fire-and-forget worker pool cleanup (~110ms saved)
- Security check: pool.destroy() deferred to background (~42ms)
- Metrics: pool.destroy() deferred to background (~70ms)
- CLI entry adds process.exit(0) after all output is written,
so the OS terminates worker threads instantly on exit
- Skipped for --mcp (long-lived server) with stdout flush guard
- MCP/library callers unaffected (Tinypool idleTimeout reclaims)
2. Skip calculateMarkdownDelimiter for non-markdown output (~5ms)
- The backtick regex scans all file contents to determine the
code fence delimiter, but is only needed for markdown style
- XML, JSON, and plain output now skip this entirely
3. Replace regex-based line counting with indexOf loop (~2ms)
- calculateFileLineCounts used content.match(/\n/g) which
allocates a temporary array of all newline matches per file
- indexOf loop counts newlines without any allocation
Phase-level benchmarks (repomix self-pack, ~1000 files):
- Security cleanup: 210ms → 168ms (42ms saved, 5 runs avg)
- Markdown delimiter: 4.6ms → 0ms for XML output (10 runs avg)
- Line counting: 4.9ms → 2.8ms (10 runs avg)
- Metrics cleanup: 70ms → 0ms via process.exit (profiled)
Theoretical improvement: ~119ms / ~1500ms CLI = ~8%
Note: high environment variance (~15% CoV) prevented reliable
end-to-end measurement; individual phase savings are verified.
1104 tests passing, lint clean on changed files.
https://claude.ai/code/session_01SKKWebXspaNFLbVoJgugs5
…ng full output
Replace the expensive full-output BPE token counting with a lightweight
estimation approach. Instead of counting tokens for the entire multi-MB output
string (~4MB for this repo), we now:
1. Count tokens for ALL individual files (not just top 50)
2. Estimate total output tokens as: sum(fileTokens) + overheadChars * ratio
3. Skip the computationally expensive output token counting entirely
The overhead (headers, tree, XML/markdown tags) tokens are estimated using the
average token-per-character ratio from file content. This is accurate to within
~0.05% because the structural text has similar token density to source code.
Benchmark results (repomix on itself, 997 files, 3.9MB content):
A/B comparison (same process, 5 runs each):
Before: avg 1700ms (pack time)
After: avg 1366ms (pack time)
Savings: 333ms (19.6%)
CLI end-to-end (5 runs):
Before: median 2.444s
After: median 1.843s
Savings: 601ms (24.6%)
Token accuracy: 0.002% error (20 tokens off out of 1,055,397)
The improvement comes from eliminating ~19 worker tasks of 200KB BPE tokenization
each, which previously competed with file token counting for worker threads.
As a bonus, all files now have token counts computed, enabling the tokenCountTree
feature at no additional cost.
https://claude.ai/code/session_01DCGGhD7A3bdik77AbiU2gM
Replace globby's expensive .gitignore traversal with `git ls-files` for repositories where git is available. Git reads from its index, which is much faster than globby's filesystem walk + gitignore file discovery. ## What changed - Added `searchFilesWithGit()` fast path in fileSearch.ts that uses `git ls-files --cached --others --exclude-standard` to list files - Uses `git ls-files -s` to identify and filter out symlinks (mode 120000) matching globby's followSymbolicLinks:false behavior - Applies ignore patterns (defaultIgnore, custom, .repomixignore/.ignore) using picomatch for efficient batch matching - Falls back to globby when git is unavailable or useGitignore is disabled - Added picomatch as explicit dependency (already transitive via globby) ## Benchmark results (repomix repo, ~1000 files) searchFiles() isolated: Before: ~470ms (globby with gitignore traversal) After: ~178ms (git ls-files + picomatch filtering) pack() function (8 runs each, sorted): Before: 877, 903, 979, 988, 1006, 1025, 1026, 1046ms (median: 988ms) After: 909, 910, 914, 916, 937, 955, 973, 1007ms (median: 916ms) Improvement: -7.3% (median), -5.5% (trimmed mean) All 1104 tests passing, lint clean. https://claude.ai/code/session_01WYHGetwQtRFCMsebzhrDzc
yamadashy
force-pushed
the
perf/auto-perf-tuning-0406
branch
from
April 7, 2026 11:59
a3bc119 to
5065a42
Compare
Replace globby filesystem scan with git ls-files for the file search phase in git repositories. This reads from the git index instead of walking the filesystem, significantly reducing search latency. ## What changed - Added `searchFilesGit()` fast path that uses `git ls-files --stage` and `git ls-files --others --exclude-standard` to get file lists - Pre-compile ignore patterns into RegExp objects via `minimatch.makeRe()` for O(1) per-file matching instead of re-parsing glob patterns - Symlinks (mode 120000) are filtered out from git stage output to match globby's `onlyFiles: true` behavior - Falls back to globby when: not a git repo, gitignore disabled, explicit files provided (stdin mode), or git command fails ## Benchmark results (repomix repo, 1018 files, 10 runs each) | Metric | Baseline (globby) | Optimized (git ls-files) | Change | |----------|-------------------|--------------------------|---------| | Median | 1322 ms | 1245 ms | -5.8% | | Mean | 1317 ms | 1243 ms | -5.6% | Search-only: ~25ms (git) vs ~130ms (globby) = ~80% faster search phase. The end-to-end improvement is ~77ms because the search phase is ~12% of total execution time. Lint: pass, Tests: 1102/1102 pass, Output: identical file lists verified. https://claude.ai/code/session_014Sk4Q4esESXEDBxiuKMm6W
… perf/auto-perf-tuning-0406 Resolved conflict in src/core/file/fileSearch.ts by taking the remote's version which already includes the git ls-files optimization. Removed the duplicate fileSearchGit.ts file. https://claude.ai/code/session_014Sk4Q4esESXEDBxiuKMm6W
…ast path Replace the globby directory scan with lightweight directory discovery from git ls-files output when searching for empty directories on the git fast path. This avoids both the expensive globby import (~55ms) and its filesystem traversal with gitignore parsing (~63ms). Changes: - Lazy-load globby via dynamic import() so it's only loaded when actually needed (non-git fallback path, listDirectories, listFiles). On the common git fast path, globby is never imported at all. - Add discoverDirectories() that extracts parent directories from git ls-files file paths in O(n) time, then recursively discovers child directories not containing tracked files (to catch completely empty directories that git doesn't track). This replaces the separate globby directory scan. - Parallelize findEmptyDirectories() readdir calls with Promise.all instead of sequential for...of loop, reducing ~22ms sequential I/O to ~6ms. Benchmark (repomix repo, 997 files, includeEmptyDirectories: true): - searchFiles: 188ms → 56ms (70% faster, -132ms) - End-to-end CLI: ~1219ms → ~1172ms (~5% faster) Component savings breakdown: - Globby import avoided on git path: ~55ms - Globby directory scan replaced with lightweight discovery: ~63ms - Parallel readdir in findEmptyDirectories: ~16ms https://claude.ai/code/session_01CtDuX8aLcNjV6HMzcxVGTs
…re available When tokenCountTree is enabled, all file contents are already individually token-counted. The full output is ~97% file content and ~3% structural overhead (XML/markdown tags, headers, tree). Previously, the entire output (~4MB for this repo) was also tokenized via worker threads, competing with file metrics for the shared worker pool and causing significant contention. This change estimates total output tokens as: sum(file_tokens) + round(overhead_chars × file_token_density) This eliminates full-output tokenization when all per-file counts exist, removing worker pool contention in the final pipeline stage. Benchmark (repomix on itself, 997 files, tokenCountTree=50000, 5 runs avg): Before: 939ms (pack only), 1.51s (full CLI) After: 773ms (pack only), 1.19s (full CLI) Improvement: 166ms / 17.7% (pack), ~320ms / 21% (full CLI) Token count accuracy: Actual output tokens: 1,046,635 Estimated tokens: 1,047,610 Difference: 975 (0.09%) Note: the existing chunked tokenization also introduces BPE boundary errors at every 200K-char split point, so exact counts were already approximate. When tokenCountTree is disabled (default), behavior is unchanged — the full output is tokenized as before. https://claude.ai/code/session_01AKtna74H2P6zT5UTKxxvaq
Resolved conflict in calculateMetrics.ts by accepting the remote version which already includes a more complete token estimation optimization (accounts for git diff/log tokens and always counts all files). Updated test to remove stale calculateOutputMetrics references and added test for structural overhead token estimation. Fixed picomatch type declarations from remote changes. https://claude.ai/code/session_01AKtna74H2P6zT5UTKxxvaq
Two optimizations that together reduce CLI execution time by ~120ms (6.8%):
1. Git ls-files fast path for file search (fileSearch.ts, gitCommand.ts):
Use `git ls-files -c -o -s --exclude-standard` instead of globby's full
directory traversal + gitignore parsing when running in a git repository.
Git's cached file index makes this ~5x faster for the search stage
(~50ms vs ~250ms). Falls back to globby when git is unavailable or for
non-git directories.
2. Fast pre-check for base64 truncation (truncateBase64.ts):
Add two-phase gating to skip expensive global regex scans on files that
clearly don't contain base64 data:
- Data URIs: gated by String.includes('base64,') — O(n) with SIMD
- Standalone base64: gated by line-length check (indexOf-based) then
non-whitespace run check — skips ~98% of files
Reduces truncateBase64Content from ~82ms to ~12ms for 1000 files.
Benchmark results (10 runs, `time node bin/repomix.cjs` on repomix repo):
Before: avg 1.782s (range 1.704–1.891s)
After: avg 1.661s (range 1.623–1.720s)
Improvement: 121ms, 6.8%
All 1102 tests pass. Output is byte-identical.
https://claude.ai/code/session_01DJ9xBHy1W669RuuKLxtrQ8
… perf/auto-perf-tuning-0406 # Conflicts: # src/core/file/fileSearch.ts # src/core/file/truncateBase64.ts
Move the `try...finally` block to start immediately after metrics worker pool creation, ensuring cleanup happens even if searchFiles or sortPaths throws an error. Previously, errors in the search/sort phase would leak the worker pool. Addresses review feedback from gemini-code-assist on PR #1428. https://claude.ai/code/session_01DJ9xBHy1W669RuuKLxtrQ8
Remove the @ts-expect-error suppression since @types/picomatch is declared in package.json. The suppression would cause a build failure once dependencies are properly installed, as @ts-expect-error on a line with no error is itself a TypeScript error. https://claude.ai/code/session_01DJ9xBHy1W669RuuKLxtrQ8
… deferred output pattern Three complementary optimizations that together reduce pack() wall time by ~7%: 1. Increase METRICS_BATCH_SIZE from 10 to 50 in calculateSelectiveFileMetrics. Reduces IPC round-trips from ~100 to ~20 for a 1000-file repo, cutting per-file metrics overhead while maintaining good load balancing across workers. 2. Set minThreads = maxThreads in worker pools for eager thread spawning. Ensures all workers are created at pool initialization rather than lazily, so warmup tasks (gpt-tokenizer BPE loading) run truly in parallel across all threads from the start. 3. Restructure packager pipeline with a deferred output promise pattern. Starts file and git metrics calculation immediately after file processing completes (using a deferred promise for the output), overlapping metric computation with the remaining security check time. Previously, all metrics waited for both security check and output generation to complete before starting. Benchmark results (repomix repo, ~1000 files, 4MB output): Baseline: Avg=1183ms, P50=1175ms (10 runs) Optimized: Avg=1103ms, P50=1102ms (10 runs) Improvement: ~6.8% (~80ms) https://claude.ai/code/session_01LfAuMfJiVYJ9RX9YkxRfmT
yamadashy
force-pushed
the
perf/auto-perf-tuning-0406
branch
from
April 9, 2026 05:09
539ac5d to
fb0a861
Compare
…s to reduce startup cost Defer importing three expensive modules from CLI startup to actual usage: - handlebars (~25ms): loaded in outputGenerate.ts on first template compile - fast-xml-builder (~3ms): loaded in generateParsableXmlOutput on demand - @clack/prompts (~16ms): loaded in migrationAction/skillPrompts when needed - packSkill module chain: lazy-loaded via dynamic import in packager.ts These modules were previously loaded eagerly in the import chain (defaultAction → packager → outputGenerate → handlebars) even though they're only needed late in the pipeline during output generation. Benchmark results (NODE_DISABLE_COMPILE_CACHE=1, 10 runs each): - Baseline median: 947ms - After median: 873ms - Improvement: 74ms (7.8%) The improvement is most significant on first run, CI/CD, Docker containers, and Node.js 20 (which lacks compile cache). With warm compile cache the savings are ~32ms in the import chain, with some modules (packSkill, @clack/prompts, fast-xml-builder) completely avoided on default runs. Import chain measurement: 139ms → 107ms (32ms faster startup) Output is byte-for-byte identical — no functional changes. https://claude.ai/code/session_0156PKHfb5NcTVQxfZBW6nAn
yamadashy
force-pushed
the
perf/auto-perf-tuning-0406
branch
from
April 9, 2026 05:13
fb0a861 to
082f8b4
Compare
Remove 11 keywords from the security pre-filter that correspond to secretlint rules disabled by default (enableIDScanRule: false in @secretlint/secretlint-rule-aws): - AWS Access Key ID prefixes: AKIA, AGPA, AIDA, AROA, AIPA, ANPA, ANVA, ASIA - AWS Account ID patterns: ACCOUNT_ID, account_id, AccountId These keywords trigger false positives on extremely common code patterns (e.g., account_id in database models, ASIA in timezone handling, AccountId in TypeScript interfaces), causing expensive lintSource() calls that always find nothing because the corresponding rules are never executed. Benchmark (100 files with account_id, ~8KB each): Before: 799ms median After: 658ms median Savings: 141ms (17.6%) The improvement comes from: - Eliminating unnecessary lintSource() calls (~0.5ms per file) - Skipping @secretlint module import entirely (~23ms) when no files pass the pre-filter - Reduced CPU contention with metrics worker pool All active secretlint rules (AWS Secret Access Key, GitHub tokens, Slack tokens, private keys, database URIs, etc.) remain fully covered by the pre-filter. https://claude.ai/code/session_01EruXyfeGcxnZazqorxEQm3
yamadashy
force-pushed
the
perf/auto-perf-tuning-0406
branch
from
April 9, 2026 06:12
105219e to
9357d02
Compare
…on-based approach Replace the O(N²) rendering loop in generateSplitOutputParts with a four-phase O(N+P) algorithm that uses solo renders and calibrated overhead estimation: 1. Solo-render each group individually (O(N) renders, 1 group each) 2. Calibrate shared overhead from the first two groups' solo+combined renders 3. Greedy bin-packing using exact additive estimates (solo sizes - shared overhead) 4. One verification render per final part (O(P) renders) with adjustment fallback The estimation is exact when groups have non-overlapping root entries (guaranteed by buildOutputSplitGroups), since the file tree contribution is purely additive. Benchmark (60 dirs, 900 files, 10 parts, simulated 0.1ms/file render cost): OLD: 69 render calls, 4,095 total files rendered, median 478ms NEW: 72 render calls, 1,845 total files rendered, median 241ms → 50% faster Benchmark (100 dirs, 2000 files, 25 parts): OLD: 124 calls, 7,400 files rendered, median 461ms NEW: 127 calls, 4,060 files rendered, median 288ms → 38% faster All 1126 tests pass, identical output in all scenarios. https://claude.ai/code/session_01CJ9FdfHjKVWjkRd9wYixGz
Move the metrics worker pool initialization from before searchFiles to after it completes. This eliminates CPU contention between worker thread spawning/BPE initialization and the file search pipeline. Problem: The previous approach pre-created the metrics worker pool at the start of pack() (or even before it in defaultAction), spawning 4 worker threads that immediately begin CPU-intensive BPE encoder initialization (~73ms each). These threads compete with 6 concurrent git subprocesses (ls-files, staged, deleted, diffs, logs, change-counts) and the main thread's picomatch pattern matching — 11 tasks on 4 cores. This contention inflated file search time from ~25ms to ~225ms. Solution: Defer pool creation until after searchFiles returns. The warmup then overlaps with file collection (sync reads), file processing, and the security check, which are I/O-bound or lightweight and don't contend with worker CPU usage. As a bonus, the actual file count is now known, so the pool allocates the correct number of threads instead of always requesting the maximum. Benchmark (repomix src/, 128 files, 4-core 2.1GHz): Before: 659ms median (15 runs) After: 579ms median (15 runs) Improvement: -80ms (12.1% faster) https://claude.ai/code/session_01MahktfjoTTGWeJY1xHup62
…ssary memoryUsage calls Reduce chunk count for parallel output token counting from a fixed 1000 to match the number of available CPU cores. With 4 cores, this cuts IPC overhead from 1000 to 4 postMessage round trips while still saturating all worker threads. Skip expensive process.memoryUsage() calls in withMemoryLogging and logMemoryUsage when log level is below DEBUG, since the stats are only used for trace-level logging. ## Changes - calculateOutputMetrics: replace hardcoded CHUNK_SIZE=1000 with getProcessConcurrency() to match actual worker thread count - memoryUtils: guard process.memoryUsage() behind log level check ## Benchmark (20 runs each, repomix on its own repo ~1015 files, 4.3MB output) Baseline: Mean: 2.532s | Trimmed mean: 2.515s | Median: 2.519s P10: 2.413s | P90: 2.621s After: Mean: 2.354s | Trimmed mean: 2.356s | Median: 2.358s P10: 2.307s | P90: 2.387s Improvement: Mean: -7.0% | Trimmed mean: -6.3% | Median: -6.4% P90 reduced from 2.621s to 2.387s (variance halved) Root cause: The old CHUNK_SIZE=1000 created 1000 tiny ~4KB chunks for a 4.3MB output. Each chunk required a postMessage serialization round trip to the worker pool. With only 4 worker threads, each thread processed ~250 chunks sequentially, spending more time on IPC overhead than actual BPE encoding. Matching chunk count to core count gives each thread one large ~1MB chunk, eliminating ~996 unnecessary IPC round trips. https://claude.ai/code/session_01Lbbd8QmBA7SpcoTqrv55b3
… perf/auto-perf-tuning-0406 Resolve merge conflicts in calculateOutputMetrics.ts and its tests by accepting the remote's TARGET_CHARS_PER_CHUNK approach (the chunking optimization was already addressed by prior work on this branch). Keep the memoryUtils optimization (skip process.memoryUsage() in non-debug mode). https://claude.ai/code/session_01Lbbd8QmBA7SpcoTqrv55b3
Eliminate the Tinypool child_process worker used to run the pack pipeline
in the default CLI action. The worker existed solely to isolate the
spinner display, but spawning a child process and loading the full module
graph in it added ~200ms of fixed overhead on every invocation.
By running pack() directly in the main process with the spinner alongside,
we remove this overhead while maintaining identical behavior: the spinner
still animates via setInterval, progress callbacks still drive updates,
and all worker-based pipeline stages (security check, metrics, file
processing) continue using their own thread pools.
The defaultActionWorker.ts file is preserved for bundled/unified worker
environments.
Benchmark results (median of 5 runs, Node.js):
Single file (--include 'package.json'):
Before: 0.888s → After: 0.691s (-197ms, -22.2%)
src/ directory (--include 'src'):
Before: 1.088s → After: 0.898s (-190ms, -17.5%)
Full repo (no filter):
Before: 2.605s → After: 2.265s (-340ms, -13.0%)
https://claude.ai/code/session_01DGJUQZZ6dkmxyQV69xBUBT
… perf/auto-perf-tuning-0406 # Conflicts: # src/cli/actions/defaultAction.ts # tests/cli/actions/defaultAction.test.ts # tests/cli/actions/defaultAction.tokenCountTree.test.ts
Size the metrics worker pool based on estimated tokenization task count instead of total file count. The default path (tokenCountTree disabled) only tokenizes the top ~50 files for ranking and ratio estimation, but previously allocated workers for all files (e.g., 4 threads for 1000 files on a 4-core machine). The excess BPE initialization CPU contention inflated search time by ~2x and collect time by ~5x. With accurate pool sizing, the default case spawns 1 worker thread instead of 4, reducing warmup contention across the pipeline. tokenCountTree=true and tokenCountTree=<threshold> still use all files for pool sizing since they tokenize all (or most) files. Benchmark (1000-file repo, 4-core, 10 runs median): Before: 620ms After: 485ms Improvement: -135ms (-22%) Per-phase impact (controlled test, sequential): searchFiles: 116ms → 52ms (-55%) collectFiles: 216ms → 29ms (-87%) warmup wait: 236ms → 123ms (-48%) https://claude.ai/code/session_0199XB1CSEmQ6yYEhQEjAKzz
yamadashy
force-pushed
the
perf/auto-perf-tuning-0406
branch
from
April 9, 2026 13:25
3eb892e to
eae6adc
Compare
Defer gpt-tokenizer and json5 from the synchronous import graph to reduce CLI startup latency. Three changes: 1. Extract TOKEN_ENCODINGS constant to tokenEncodings.ts, breaking the configSchema → TokenCounter → gpt-tokenizer import chain. Config schema validation only needs encoding names, not the tokenizer. 2. Lazy-load json5 in configLoad.ts (only needed when a JSON5/JSONC config file exists). Saves ~28ms on every run without a config file. 3. Preload @secretlint/core before metrics worker threads spawn. The secretlint module (~70ms) previously loaded during the security check phase, competing with metrics worker BPE initialization for CPU time. Starting the import during the I/O-bound file search phase eliminates this contention. Benchmark (20-run A/B, repomix on own repo, 999 files): Baseline median: 1420ms After median: 1372ms (3.4% improvement) p75 improvement: 81ms (5.5%) IQR reduction: 134→92ms (31% less variance) Module loading improvements (measured in isolation): configSchema.ts: 175ms → 95ms (gpt-tokenizer no longer loaded) json5 deferred: ~28ms saved when no config file present @secretlint preload: eliminates CPU contention with workers https://claude.ai/code/session_017sHG68dpyUbVcoxnFPfUBE
…warmup stall
Preload BPE rank data at the start of pack() so its async I/O overlaps
with the searchFiles phase (when the event loop is available), and remove
the blocking `await metricsWarmupPromise` synchronization point.
Problem:
Previously, loadBpeRanks was called inside createMetricsTaskRunner
(after searchFiles), but the subsequent synchronous collectFiles
(~150ms) blocked the event loop, preventing the async BPE file-read
callback from firing. This delayed BPE data availability until after
collectFiles, and worker warmup (BPE deserialization ~73ms) could
only start then. The pipeline stalled ~200ms at
`await metricsWarmupPromise` before calculateMetrics could begin.
Fix:
1. Fire loadBpeRanks() at the start of pack(), before searchFiles.
The async I/O (~100ms) completes during searchFiles (~135ms of
git subprocess I/O), so BPE data is ready before collectFiles.
2. Pass the pre-loaded BPE promise to createMetricsTaskRunner,
which feeds it to worker warmup tasks immediately on pool creation.
3. Remove `await metricsWarmupPromise` — workers complete BPE init
(~73ms) during collectFiles (~150ms), and real tokenization tasks
dispatched by calculateMetrics queue behind warmup via FIFO.
Benchmark (repomix self-pack, 999 files, 4-core):
Before: median 913ms, p75 925ms
After: median 855ms, p75 864ms
Improvement: -6.4% median, -6.6% p75
Pipeline shift (single profiled run):
calculateMetrics start: 707ms → 522ms (185ms earlier)
pack() end: 919ms → 802ms (117ms faster)
https://claude.ai/code/session_01Cbzh8SssmdjmCsDJWaZ5tY
yamadashy
added a commit
that referenced
this pull request
Apr 9, 2026
…s to reduce startup cost Defer importing three expensive modules from CLI startup to actual usage: - handlebars (~25ms): loaded in outputGenerate.ts on first template compile - fast-xml-builder (~3ms): loaded in generateParsableXmlOutput on demand - @clack/prompts (~16ms): loaded in migrationAction/skillPrompts when needed - packSkill module chain: lazy-loaded via dynamic import in packager.ts Cherry-picked from 082f8b4 (PR #1428) Co-Authored-By: Claude <noreply@anthropic.com>
yamadashy
added a commit
that referenced
this pull request
Apr 9, 2026
…er-worker file I/O Each metrics worker thread independently loaded gpt-tokenizer's BPE rank data (~3.6MB, 200K entries) from disk. Now the BPE data is loaded once on the main thread, serialized to a JSON string (~1.6MB), and passed to each worker via the warmup task. Workers deserialize and build the encoder instead of reading from disk. Cherry-picked from fd6b625 (PR #1428) Co-Authored-By: Claude <noreply@anthropic.com>
yamadashy
added a commit
that referenced
this pull request
Apr 9, 2026
Pre-fetch file change counts via prefetchFileChangeCounts in the existing Promise.all alongside collectFiles/getGitDiffs/getGitLogs, so that sortOutputFiles (called later inside generateOutput) finds the result already cached and skips the blocking git subprocess. Cherry-picked from e587f6a (PR #1428) Co-Authored-By: Claude <noreply@anthropic.com>
This was referenced Apr 9, 2026
Closed
yamadashy
added a commit
that referenced
this pull request
Apr 11, 2026
…s to reduce startup cost Defer importing three expensive modules from CLI startup to actual usage: - handlebars (~25ms): loaded in outputGenerate.ts on first template compile - fast-xml-builder (~3ms): loaded in generateParsableXmlOutput on demand - @clack/prompts (~16ms): loaded in migrationAction/skillPrompts when needed - packSkill module chain: lazy-loaded via dynamic import in packager.ts Cherry-picked from 082f8b4 (PR #1428) Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Automated performance tuning of the repomix pack pipeline. This PR consolidates multiple optimizations achieving cumulative improvements of ~25% on end-to-end CLI execution.
Optimizations
Previous commits (already merged into this branch)
git logwith file collection (~20% faster)hasLongLine(indexOf-based) +hasLongNonWhitespaceRungating to skip expensive regex scans. Reduces truncateBase64Content from ~82ms to ~12ms for 1000 files.lintSource()for files without secret-related keywords.mightContainSecretpre-filter was using sequentialString.includes()calls for ~50 keywords, scanning each file's content 50+ times. Replaced with a single pre-compiledRegExpalternation that scans content in one pass (3.5x speedup).mightContainSecretintosecurityPreFilter.tsmodule — MovedmightContainSecret()and related types out ofsecurityCheckWorker.tsinto a newsecurityPreFilter.tsmodule with zero dependencies on secretlint.TOKEN_ENCODINGStotokenEncodings.ts, lazy-loadjson5, preload@secretlint/corebefore metrics workers.Latest commit: Preload BPE data during searchFiles to eliminate metrics warmup stall
loadBpeRanksasync I/O (~100ms) was called insidecreateMetricsTaskRunner(after searchFiles), but its callback couldn't fire during the subsequent synchronouscollectFiles(~150ms), creating a ~200ms pipeline stall atawait metricsWarmupPromise. Fix: fireloadBpeRanks()at the start ofpack()so it overlaps with searchFiles I/O, pass the pre-loaded promise tocreateMetricsTaskRunner, and remove the blocking warmup await (tasks FIFO-queue behind warmup, with safe disk-loading fallback).Benchmark results (latest commit: preload BPE data)
repomix repo root (999 files, 4-core, 10 runs each):
Pipeline shift (single profiled run):
calculateMetricsstart: 707ms → 522ms (185ms earlier)pack()end: 919ms → 802ms (117ms faster)Test plan
tokenCountTree=truestill uses all workers and tokenizes all fileshttps://claude.ai/code/session_01Cbzh8SssmdjmCsDJWaZ5tY