feat(cli): add --threads auto to cap worker concurrency for large files (#77 P1)#79
Open
luojiyin1987 wants to merge 1 commit into
Open
feat(cli): add --threads auto to cap worker concurrency for large files (#77 P1)#79luojiyin1987 wants to merge 1 commit into
luojiyin1987 wants to merge 1 commit into
Conversation
1d79d96 to
0b24dee
Compare
Closes P1 of issue #77 (CLI adaptive concurrency). When the user passes '--threads auto', resolveAdaptiveConcurrency stats the input files and caps the Piscina pool based on the largest file size: max < 1 MiB -> availableParallelism() (no cap) 1 MiB <= max<5 -> min(cpuLimit, 2) max >= 5 MiB -> 1 Numeric --threads N keeps the existing behavior verbatim. The cap is applied before Piscina is created, so worker count (and therefore peak RSS) shrinks in proportion. --dev mode logs the applied cap. Files: - src/types.ts: add ThreadCount = number | 'auto' - src/utils/configure.ts: getThreadCount returns 'auto' for that input - src/utils/batch-lint.ts: add getMaxFileSize + resolveAdaptiveConcurrency - src/lint-md.ts: resolve effective concurrency, dev log when capped - scripts/benchmark-memory.mjs: accept --threads auto - CHANGELOG.md: Unreleased entry - __tests__/*: configure auto, resolveAdaptiveConcurrency unit tests, threads-validation auto CLI tests Benchmark on this machine (8 files of 1 MiB, 4 files of 5 MiB, runs=3): scenario --threads 4 RSS auto RSS reduction 8x 1 MiB 2.88 GiB 1.39 GiB ~52% 4x 5 MiB 5.95 GiB 2.72 GiB ~54% Trade-off: wall time goes up (e.g. 5 MiB 4x goes 5.3s -> 14.4s at 1 worker) which is the explicit cost documented in the issue. Out of scope for this PR (issue #77 follow-ups): - --max-memory budget - fix-mode write concurrency cap - docs/performance.md
0b24dee to
fc3d398
Compare
This was referenced Jul 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #78 (P1 sub-issue of #77).
落地
--threads auto第一版:纯文件大小阈值,不引入内存预算。行为
--threads N字节级保持原行为--threads)行为保持原行为--threads auto时只在--dev模式下,且 cap 真正收窄了并发数时打印一行Adaptive concurrency: requested auto, effective N, max file X.XX MiB设计
src/types.ts新增ThreadCount = number | 'auto'src/utils/configure.ts:getThreadCount在收到'auto'时直接返回该字符串,数值路径完全不动src/utils/batch-lint.ts新增两个纯函数:getMaxFileSize(paths):Promise.all(stat)取最大 size,错误向上抛resolveAdaptiveConcurrency(threads, paths):实现上述规则表src/lint-md.ts:在mdFiles就绪后调用resolveAdaptiveConcurrency,把结果作为number传给batchLint(batchLint公共签名不变);--dev模式下补打一行scripts/benchmark-memory.mjs:--threads接受auto测试
__tests__/configure.spec.ts:getThreadCount('auto')返回'auto'__tests__/batch-lint.spec.ts:新增describe('getMaxFileSize')和describe('resolveAdaptiveConcurrency'),覆盖数值路径与 auto 路径的全部阈值边界__tests__/threads-validation.spec.ts:新增stdin + --threads auto和files + --threads auto两组 CLI 烟测npm test→ 74/74 passing;npm run lintcleanBenchmark(本机 16 cores,runs=3 median)
--threads 4RSS--threads autoRSS复现命令:
wall time 5 MiB 4x 场景从 5.3s 升到 14.4s(1 worker),1 MiB 8x 场景从 2.4s 升到 3.5s。这是 OOM 风险下降的显式代价,与 issue 描述一致。
行为 / API 边界
batchLint的公共签名未变化(仍为number),未破坏任何下游调用--threads auto模式在Promise.all(stat)阶段会先于 worker 暴露 stat 失败(与现有lintWorker读文件失败的错误处理边界一致)--fix写文件并发继续使用effectiveThreads,但--fix路径下若threads === 'auto'且收窄严重,对写并发也会有降级效果。Issue [performance] 性能优化路线图:在文件读取移入 worker 后继续降低内存压力 #77 P2/P6 之后再评估是否需要为--fix单独放行非目标(issue #77 后续子项)
--max-memory <size>内存预算(issue [performance] 性能优化路线图:在文件读取移入 worker 后继续降低内存压力 #77 步骤 2)--fix写文件独立并发(issue [performance] 性能优化路线图:在文件读取移入 worker 后继续降低内存压力 #77 P6)docs/performance.md(issue [performance] 性能优化路线图:在文件读取移入 worker 后继续降低内存压力 #77 P7)