refactor(update-links): divided into features instead of layers#9
Merged
Conversation
…ency Fix concurrency bug: replace .fill(runWorker()) with Array.from factory callback so each worker slot starts its own independent promise.
…agnostics Inline all LEGACY regexes, WIN10_REGEX, normalizeLegacyUrl, pickWin10Preferred, and LegacyDownloadLinks into legacy.ts; add describeLegacyFailure. Remove matching.ts and clean up regex.ts/types.ts.
Inline PRIMARY/FALLBACK/REACT_PROPS regexes, ReleaseEntry/PageData/DownloadLinks types, and private extractReactProps/decodeHtmlEntities into releases.ts; delete now-empty html.ts, regex.ts, and types.ts.
|
thx Haru🥰 |
There was a problem hiding this comment.
Pull request overview
This PR refactors the scripts/update-links scraper from a by-layer layout into a by-feature structure, introduces shared utilities, and adds targeted Vitest coverage for key parsing/conmsumption helpers.
Changes:
- Introduces new utilities (
fetchText, regex match helpers, version extraction, and concurrency-limited mapping) and adds dedicated tests. - Refactors archive/version discovery and releases/legacy parsing to use the new utilities and removes now-obsolete shared modules (
types.ts,regex.ts,matching.ts). - Fixes the concurrency worker startup bug by ensuring each worker loop is started with a distinct promise.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/update-links/version.test.ts | Adds unit tests for version extraction from headers and URLs. |
| tests/update-links/regex-match.test.ts | Adds tests for regex match helpers and patch-link filtering. |
| tests/update-links/legacy.test.ts | Adds tests for legacy URL normalization and win10 preference selection. |
| tests/update-links/http.test.ts | Adds tests for fetchText success/failure behavior. |
| tests/update-links/concurrency.test.ts | Adds tests for result ordering and concurrency limiting. |
| scripts/update-links/utils/version.ts | Extracts version parsing logic into a focused utility module. |
| scripts/update-links/utils/regex-match.ts | Adds reusable regex match helpers and patch filtering. |
| scripts/update-links/utils/http.ts | Centralizes fetch + status checking into fetchText. |
| scripts/update-links/utils/concurrency.ts | Adds concurrency-limited async mapping helper (fixing prior worker startup issue). |
| scripts/update-links/archive.ts | Refactors archive scraping to use fetchText and emits normalized entry.version. |
| scripts/update-links/releases.ts | Refactors modern download page parsing; improves legacy failure diagnostics integration. |
| scripts/update-links/legacy.ts | Refactors legacy parsing to use new utilities and adds failure description helper. |
| scripts/update-links/index.ts | Switches to new archive entry shape and shared concurrency utility. |
| scripts/update-links/constants.ts | Removes CUDA_ARCHIVE_URL export (now localized to archive feature). |
| scripts/update-links/types.ts | Removes centralized types (moved/embedded where used). |
| scripts/update-links/regex.ts | Removes centralized regex constants (moved/embedded where used). |
| scripts/update-links/matching.ts | Removes centralized matching helpers (moved into feature/utils modules). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
mzwing
changed the title
mzwing
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
结构(by-layer → by-feature)
archive.ts— 版本清单发现releases.ts— 现代下载页解析(React props)legacy.ts— 旧版下载页解析 + 失败诊断utils/:http.ts(fetchText)、version.ts(版本号提取)、regex-match.ts(pickFirstMatch/pickAllMatches)、concurrency.ts(mapWithConcurrency)regex.ts/matching.ts/html.ts/types.ts(内容按「谁在用」归位)顺带修复与改进
mapWithConcurrency并发 bug:原Array.from({length:N}).fill(runWorker())复用了同一个 promise,实际只起 1 个 worker → 脚本退化成串行抓取。改为工厂回调
Array.from({length:N}, () => runWorker()),并发上限limit才真正生效。fetchText:消除archive.ts/releases.ts中重复的fetch + 状态检查。LEGACY_LINUX_RUNFILE_FALLBACK_REGEX与主正则逐字符相同,
?? fallback永不生效。实测 69 个 CUDA 版本(其中 16 个走 legacy 路径)全部被主正则覆盖,0 个需要 fallback,故删除而非放宽。
releases.ts不再直接依赖 legacy 正则做失败诊断,改为调用legacy.describeLegacyFailure()。archive.ts,index.ts直接用entry.version。