Skip to content

feat(cli): add --threads auto to cap worker concurrency for large files (#77 P1)#79

Open
luojiyin1987 wants to merge 1 commit into
masterfrom
feat/cli-adaptive-concurrency
Open

feat(cli): add --threads auto to cap worker concurrency for large files (#77 P1)#79
luojiyin1987 wants to merge 1 commit into
masterfrom
feat/cli-adaptive-concurrency

Conversation

@luojiyin1987

Copy link
Copy Markdown
Contributor

Closes #78 (P1 sub-issue of #77).

落地 --threads auto 第一版:纯文件大小阈值,不引入内存预算。

行为

threads=N (任意正整数)   →  N  (clamped to [1, fileCount])  行为不变
threads='auto':
  max < 1 MiB           →  availableParallelism()
  1 MiB <= max<5 MiB    →  min(availableParallelism(), 2)
  max >= 5 MiB          →  1
  • 数值型 --threads N 字节级保持原行为
  • 默认(不传 --threads)行为保持原行为
  • --threads auto 时只在 --dev 模式下,且 cap 真正收窄了并发数时打印一行 Adaptive concurrency: requested auto, effective N, max file X.XX MiB
  • 不改变 lint 输出
  • 不修改 parser / rule / worker 实现

设计

  • src/types.ts 新增 ThreadCount = number | 'auto'
  • src/utils/configure.tsgetThreadCount 在收到 'auto' 时直接返回该字符串,数值路径完全不动
  • src/utils/batch-lint.ts 新增两个纯函数:
    • getMaxFileSize(paths)Promise.all(stat) 取最大 size,错误向上抛
    • resolveAdaptiveConcurrency(threads, paths):实现上述规则表
  • src/lint-md.ts:在 mdFiles 就绪后调用 resolveAdaptiveConcurrency,把结果作为 number 传给 batchLintbatchLint 公共签名不变);--dev 模式下补打一行
  • scripts/benchmark-memory.mjs--threads 接受 auto

测试

  • __tests__/configure.spec.tsgetThreadCount('auto') 返回 'auto'
  • __tests__/batch-lint.spec.ts:新增 describe('getMaxFileSize')describe('resolveAdaptiveConcurrency'),覆盖数值路径与 auto 路径的全部阈值边界
  • __tests__/threads-validation.spec.ts:新增 stdin + --threads autofiles + --threads auto 两组 CLI 烟测
  • npm test → 74/74 passing;npm run lint clean

Benchmark(本机 16 cores,runs=3 median)

场景 --threads 4 RSS --threads auto RSS 降幅
8 × 1 MiB 2.88 GiB 1.39 GiB ~52%
4 × 5 MiB 5.95 GiB 2.72 GiB ~54%

复现命令:

npm run build
node scripts/benchmark-memory.mjs --files 8 --bytes-per-file 1048576 --threads 4     --runs 3
node scripts/benchmark-memory.mjs --files 8 --bytes-per-file 1048576 --threads auto  --runs 3
node scripts/benchmark-memory.mjs --files 4 --bytes-per-file 5242880 --threads 4     --runs 3
node scripts/benchmark-memory.mjs --files 4 --bytes-per-file 5242880 --threads auto  --runs 3

wall time 5 MiB 4x 场景从 5.3s 升到 14.4s(1 worker),1 MiB 8x 场景从 2.4s 升到 3.5s。这是 OOM 风险下降的显式代价,与 issue 描述一致。

行为 / API 边界

  • batchLint 的公共签名未变化(仍为 number),未破坏任何下游调用
  • --threads auto 模式在 Promise.all(stat) 阶段会先于 worker 暴露 stat 失败(与现有 lintWorker 读文件失败的错误处理边界一致)
  • --fix 写文件并发继续使用 effectiveThreads,但 --fix 路径下若 threads === 'auto' 且收窄严重,对写并发也会有降级效果。Issue [performance] 性能优化路线图:在文件读取移入 worker 后继续降低内存压力  #77 P2/P6 之后再评估是否需要为 --fix 单独放行

非目标(issue #77 后续子项)

@luojiyin1987 luojiyin1987 force-pushed the feat/cli-adaptive-concurrency branch from 1d79d96 to 0b24dee Compare July 8, 2026 11:10
Closes P1 of issue #77 (CLI adaptive concurrency). When the user passes
'--threads auto', resolveAdaptiveConcurrency stats the input files and
caps the Piscina pool based on the largest file size:

  max < 1 MiB     -> availableParallelism() (no cap)
  1 MiB <= max<5  -> min(cpuLimit, 2)
  max >= 5 MiB    -> 1

Numeric --threads N keeps the existing behavior verbatim. The cap is
applied before Piscina is created, so worker count (and therefore peak
RSS) shrinks in proportion. --dev mode logs the applied cap.

Files:
- src/types.ts: add ThreadCount = number | 'auto'
- src/utils/configure.ts: getThreadCount returns 'auto' for that input
- src/utils/batch-lint.ts: add getMaxFileSize + resolveAdaptiveConcurrency
- src/lint-md.ts: resolve effective concurrency, dev log when capped
- scripts/benchmark-memory.mjs: accept --threads auto
- CHANGELOG.md: Unreleased entry
- __tests__/*: configure auto, resolveAdaptiveConcurrency unit tests,
  threads-validation auto CLI tests

Benchmark on this machine (8 files of 1 MiB, 4 files of 5 MiB, runs=3):

  scenario         --threads 4 RSS   auto RSS   reduction
  8x 1 MiB         2.88 GiB          1.39 GiB   ~52%
  4x 5 MiB         5.95 GiB          2.72 GiB   ~54%

Trade-off: wall time goes up (e.g. 5 MiB 4x goes 5.3s -> 14.4s at 1
worker) which is the explicit cost documented in the issue.

Out of scope for this PR (issue #77 follow-ups):
- --max-memory budget
- fix-mode write concurrency cap
- docs/performance.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[cli] 为大 Markdown 文件增加自适应并发控制

1 participant