feat(build): parallel build across worker threads - #1741
Merged
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
🚧 Files skipped from review as they are similar to previous changes (3)
📝 WalkthroughWalkthroughAdds optional CPU-aware parallel template building via tinypool, extracts per-template rendering into buildTemplate, adds renderer source override support for beforeRender hooks, implements worker orchestration and config, and expands tests and packaging to support the new flow. ChangesParallel Template Build Feature
Sequence DiagramssequenceDiagram
participant build
participant resolveParallel
participant runParallelBuild
participant tinypool
participant worker_mjs
participant buildWorker
participant buildTemplate
participant FileSystem
build->>resolveParallel: evaluate config, CPU, template count
resolveParallel-->>build: { enabled, workers }
alt parallel enabled
build->>runParallelBuild: submit sharded template batches
runParallelBuild->>tinypool: create pool with worker.mjs
runParallelBuild->>tinypool: run each shard
tinypool->>worker_mjs: invoke with shard data
worker_mjs->>buildWorker: load impl and call run(data)
buildWorker->>buildTemplate: build each template in shard
buildTemplate->>FileSystem: write HTML + plaintext outputs
buildTemplate-->>buildWorker: return files[], sfcAfterBuildCount
buildWorker-->>runParallelBuild: aggregated results
runParallelBuild-->>build: aggregated files + total sfcAfterBuildCount
else sequential
build->>buildTemplate: create renderer, build per-template
buildTemplate->>FileSystem: write outputs
buildTemplate-->>build: files[], sfcCount
end
build->>FileSystem: copy static files
build->>build: fire afterBuild event
🎯 4 (Complex) | ⏱️ ~60 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/build.ts`:
- Around line 197-210: Detect and prevent duplicate output targets before
spawning parallel workers: before calling Promise.all([...pool.run(...)])
flatten the batches and, using the same output-resolution logic used by
buildTemplate (extract that logic into or call a new helper like
computeOutputTarget(templatePath, configPath, configData, outputPath,
outputExtension) if necessary), compute each template's final output file path
deterministically, check for duplicates and fail-fast with a clear error if any
collisions exist; alternatively you can change pool.run workers to write to
per-worker temp files and then, on the main thread after results return,
atomically move/merge those temp files into the real output directory in a
deterministic order while checking for duplicates — reference batches, pool.run,
buildTemplate (or the new computeOutputTarget helper), results and files when
implementing the change.
- Around line 164-165: The current return forces sequential runs for
single-template builds because enabled is computed as workers >= 2 && count >=
2; update the condition to respect explicit parallel/threshold settings: keep
const workers = Math.min(maxWorkers, count) but return { enabled: workers >= 2
&& (count >= 2 || parallel === true || threshold === 0), workers } (or use the
actual option names in scope if different) so that explicit parallel:true or
threshold:0 will enable parallel even when count === 1; adjust the function
signature or captured variables to reference the surrounding parallel and
threshold config values used elsewhere in this module.
In `@src/render/buildTemplate.ts`:
- Around line 122-124: The code uses basename(templatePath) when setting
ptOutputPath in the sfcPlaintext branch, causing different templates with the
same filename (e.g., emails/a/welcome.vue and emails/b/welcome.vue) to collide;
change it to preserve the template's relative directory structure by computing
the path relative to your templates root (or a common templates base) and
include that directory when joining the destination: compute relDir =
relative(templatesRoot, dirname(templatePath)) and set ptOutputPath =
join(resolve(sfcPlaintext.destination), relDir, `${name}.${ptExtension}`)
(ensure destination dirs are created); apply the same relative-path-preserving
change to the global-destination branch that sets ptOutputPath in the other
block.
- Around line 153-169: The code currently picks the first non-negated glob in
computeContentBase which breaks multi-root content setups; change
computeContentBase to derive a common root from all positive patterns by
extracting each pattern's staticPart (the segment before the first glob char),
normalizing/dirname'ing them, and computing their longest common path (common
ancestor) instead of returning the first match; keep resolveOutputPath as-is (it
can continue to use the computed contentBase) so relative(...) no longer
produces leading ".." for templates in other configured roots.
- Around line 82-91: The post-render hooks are being called with the original
renderConfig instead of the merged effective template config
(rendered.templateConfig), causing hooks to see stale settings; update both
calls to events.fireAfterRender and events.fireAfterTransform to pass
templateConfig (the merged rendered.templateConfig) as the config argument
(i.e., replace renderConfig with templateConfig) so hooks receive the effective
config used by subsequent logic (ensure you reference rendered.templateConfig
when building the templateConfig variable used in those calls).
In `@src/render/parallel/buildWorker.ts`:
- Line 40: The current call const config = defu(configData, reloaded) in
buildWorker.ts uses defu's default array-concatenation semantics and can
duplicate array entries; change the merge to one where configData overrides
reloaded and arrays are replaced (not concatenated). Replace the defu call with
a merge utility/configured deep-merge that treats arrays as replace (e.g.,
deepmerge or a custom merge function) and pass arguments in the correct
precedence so configData wins; ensure the symbol names remain config, configData
and reloaded and that the resulting type is still castable to MaizzleConfig.
Validate behavior for plugin/source lists to confirm arrays from configData
fully replace those from reloaded.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 4f484aa9-f23a-4f2f-8be6-e5bc966f69fa
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (9)
package.jsonsrc/build.tssrc/render/buildTemplate.tssrc/render/createRenderer.tssrc/render/parallel/buildWorker.tssrc/render/parallel/worker.mjssrc/tests/build.test.tssrc/types/config.tstsdown.config.ts
Comment on lines
+164
to
+165
| const workers = Math.min(maxWorkers, count) | ||
| return { enabled: workers >= 2 && count >= 2, workers } |
There was a problem hiding this comment.
parallel: true is still forced to sequential for a single template.
Line 165 requires workers >= 2 && count >= 2, so parallel: true and threshold: 0 do not actually force parallel for single-template builds, which conflicts with the documented behavior in this file.
Suggested fix
- return { enabled: workers >= 2 && count >= 2, workers }
+ return { enabled: workers >= 1 && count >= 1, workers }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| const workers = Math.min(maxWorkers, count) | |
| return { enabled: workers >= 2 && count >= 2, workers } | |
| const workers = Math.min(maxWorkers, count) | |
| return { enabled: workers >= 1 && count >= 1, workers } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/build.ts` around lines 164 - 165, The current return forces sequential
runs for single-template builds because enabled is computed as workers >= 2 &&
count >= 2; update the condition to respect explicit parallel/threshold
settings: keep const workers = Math.min(maxWorkers, count) but return { enabled:
workers >= 2 && (count >= 2 || parallel === true || threshold === 0), workers }
(or use the actual option names in scope if different) so that explicit
parallel:true or threshold:0 will enable parallel even when count === 1; adjust
the function signature or captured variables to reference the surrounding
parallel and threshold config values used elsewhere in this module.
Comment on lines
+197
to
+210
| const results = await Promise.all( | ||
| batches.map(templatePaths => pool.run({ | ||
| templatePaths, | ||
| configPath, | ||
| configData, | ||
| outputPath, | ||
| outputExtension, | ||
| contentBase, | ||
| })), | ||
| ) | ||
|
|
||
| return { | ||
| files: results.flatMap(r => r.files), | ||
| sfcAfterBuildCount: results.reduce((n, r) => n + r.sfcAfterBuildCount, 0), |
There was a problem hiding this comment.
Parallel workers can nondeterministically overwrite the same output file.
At Line 197, batches are executed concurrently and each worker writes to disk inside buildTemplate(). If two templates resolve to the same output path (for example via custom output mappings), the final file becomes timing-dependent in parallel mode, while sequential mode is deterministic.
Please add collision protection (fail-fast on duplicate output targets, or write to per-worker temp locations and merge deterministically on the main thread).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/build.ts` around lines 197 - 210, Detect and prevent duplicate output
targets before spawning parallel workers: before calling
Promise.all([...pool.run(...)]) flatten the batches and, using the same
output-resolution logic used by buildTemplate (extract that logic into or call a
new helper like computeOutputTarget(templatePath, configPath, configData,
outputPath, outputExtension) if necessary), compute each template's final output
file path deterministically, check for duplicates and fail-fast with a clear
error if any collisions exist; alternatively you can change pool.run workers to
write to per-worker temp files and then, on the main thread after results
return, atomically move/merge those temp files into the real output directory in
a deterministic order while checking for duplicates — reference batches,
pool.run, buildTemplate (or the new computeOutputTarget helper), results and
files when implementing the change.
…FC plaintext destinations
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parallel build across worker threads
Large projects (hundreds–thousands of templates) build slowly because templates render sequentially through a single Vite SSR renderer. This adds an opt-in/auto parallel build that shards templates across worker threads.
Config
New top-level
parallelkey:true- always parallel,min(CPU − 1, 8)workersfalse- always sequential{ workers, threshold }-workersthread count (defaultmin(CPU − 1, 8)),thresholdtemplate count to trigger parallel (default50,0= always)How it works
beforeCreateonce, shards the template list, and runsafterBuildonce with the full file list.buildTemplate()the sequential path uses, firing per-template events (beforeRender/afterRender/afterTransform) in-thread.tinypool; the worker entry is a tiny jiti shim that loads the compiled.jsin dist and the.tssource in dev/tests.Benchmarks
Setup: 24-core machine, default
two-factor.vueMaizzle 6 template.The worker count matters more than the threshold, over-provisioning hurts:
So the default caps at 8 workers. With that cap the crossover is ~25 templates:
Behavior & limits
afterBuildhandlers can't run in a worker (they'd need to fire once on the main thread with the aggregate file list) - they're counted and a warning is logged; use the configafterBuildhook instead.Tests
resolveParallelgating (threshold, worker cap, explicit override,false, inline-object fallback) + two end-to-end parallel buildsverifying config events fire in workers and output matches sequential.
Summary by CodeRabbit
New Features
paralleloption with worker/threshold controls.Tests