Skip to content

feat(build): parallel build across worker threads #1741

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cossssmin merged 5 commits into from
Jun 6, 2026
Merged

feat(build): parallel build across worker threads #1741

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
69f03ca
feat(build): parallel build across worker threads
cossssmin Jun 5, 2026
89841de
fix(build): preserve directory structure for multi-root content and S…
cossssmin Jun 6, 2026
8baa999
fix(build): pass effective template config to afterRender and afterTr…
cossssmin Jun 6, 2026
f1bd4b2
fix(build): replace arrays instead of concatenating when merging work…
cossssmin Jun 6, 2026
d353b10
test: cover computeContentBase common ancestor and parallel afterBuil…
cossssmin Jun 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@
"reka-ui": "^2.9.3",
"string-strip-html": "^13.5.3",
"tinyglobby": "^0.2.15",
"tinypool": "^2.1.0",
"tw-animate-css": "^1.4.0",
"typescript": "^5.9.3",
"unplugin-auto-import": "^21.0.0",
Expand Down
260 changes: 145 additions & 115 deletions src/build.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,14 @@
import { readFileSync, writeFileSync, mkdirSync, cpSync, existsSync, rmSync } from 'node:fs'
import { resolve, dirname, basename, relative, join, parse as parsePath } from 'node:path'
import { mkdirSync, cpSync, existsSync, rmSync } from 'node:fs'
import { resolve, dirname, relative, join } from 'node:path'
import { fileURLToPath } from 'node:url'
import { availableParallelism } from 'node:os'
import { glob } from 'tinyglobby'
import ora from 'ora'
import { resolveConfig } from './config/index.ts'
import { EventManager } from './events/index.ts'
import { runTransformers } from './transformers/index.ts'
import { createRenderer } from './render/createRenderer.ts'
import { createPlaintext } from './plaintext.ts'
import { stripForHtml, stripForPlaintext } from './utils/output-markers.ts'
import { normalizeComponentSources } from './utils/componentSources.ts'
import { _setCurrentTemplate } from './composables/useCurrentTemplate.ts'
import defu from 'defu'
import { buildTemplate, computeContentBase } from './render/buildTemplate.ts'
import type { MaizzleConfig } from './types/index.ts'

export interface BuildResult {
Expand Down Expand Up @@ -55,103 +53,47 @@ export async function build(configInput?: Partial<MaizzleConfig> | string): Prom
rmSync(outputPath, { recursive: true, force: true })
}

const renderer = await createRenderer({ markdown: config.markdown, root: config.root, componentDirs: normalizeComponentSources(config.components?.source, process.cwd()), vite: config.vite })
const outputFiles: string[] = []
let droppedAfterBuild = 0

try {
for (const templatePath of templateFiles) {
const absolutePath = resolve(templatePath)
const parsedPath = parsePath(absolutePath)
const template = { source: readFileSync(absolutePath, 'utf-8'), path: parsedPath }

_setCurrentTemplate(parsedPath)

try {
await events.fireBeforeRender({ config, template })

const rendered = await renderer.render(absolutePath, config)

/**
* Register SFC event handlers collected during render so they take
* part in the post-render events (afterRender / afterTransform).
* They're cleared at the end of the iteration so they don't
* leak into the next template.
*/
for (const { name, handler } of rendered.sfcEventHandlers) {
events.on(name, handler)
}

let html = await events.fireAfterRender({ config, template, html: rendered.html })

/**
* Use the per-template merged config (from defineConfig() in the SFC) so
* that template-level overrides like css.safe: false are respected
* by transformers.
*/
const templateConfig = rendered.templateConfig

const doctype = rendered.doctype ?? templateConfig.doctype ?? '<!DOCTYPE html>'

if (templateConfig.useTransformers !== false) {
html = await runTransformers(html, templateConfig, absolutePath, doctype, rendered.tailwindBlocks)
}

html = await events.fireAfterTransform({ config, template, html })
if (doctype) html = `${doctype}\n${html}`

const htmlOut = stripForHtml(html)
const sfcOutputPath = rendered.outputPath
let outputFilePath: string

if (sfcOutputPath) {
const parsed = parsePath(resolve(sfcOutputPath))
const ext = parsed.ext ? parsed.ext.slice(1) : outputExtension
outputFilePath = join(parsed.dir, `${parsed.name}.${ext}`)
} else {
outputFilePath = resolveOutputPath(templatePath, outputPath, outputExtension, contentBase)
}

mkdirSync(dirname(outputFilePath), { recursive: true })
writeFileSync(outputFilePath, htmlOut)
outputFiles.push(outputFilePath)

// Generate plaintext version if configured
const globalPlaintext = templateConfig.plaintext
const sfcPlaintext = rendered.plaintext

if (globalPlaintext || sfcPlaintext) {
const globalCfg = typeof globalPlaintext === 'object' ? globalPlaintext : {}
const stripOptions = defu(sfcPlaintext?.options, globalCfg.options)
const plaintext = createPlaintext(stripForPlaintext(html), stripOptions)
const ptExtension = sfcPlaintext?.extension ?? globalCfg.extension ?? 'txt'

let ptOutputPath: string

if (sfcPlaintext?.destination) {
const name = basename(templatePath).replace(/\.(vue|md)$/, '')
ptOutputPath = join(resolve(sfcPlaintext.destination), `${name}.${ptExtension}`)
} else if (sfcOutputPath) {
const parsed = parsePath(outputFilePath)
ptOutputPath = join(parsed.dir, `${parsed.name}.${ptExtension}`)
} else if (globalCfg.destination) {
ptOutputPath = resolveOutputPath(templatePath, resolve(globalCfg.destination), ptExtension, contentBase)
} else {
ptOutputPath = resolveOutputPath(templatePath, outputPath, ptExtension, contentBase)
}

mkdirSync(dirname(ptOutputPath), { recursive: true })
writeFileSync(ptOutputPath, plaintext)
}
} finally {
_setCurrentTemplate(undefined)
events.clearSfcHandlers()
}
}
const parallel = resolveParallel(config, templateFiles.length, configInput)

if (parallel.enabled) {
spinner.text = `Building ${templateFiles.length} templates across ${parallel.workers} workers...`

const result = await runParallelBuild({
templateFiles,
workers: parallel.workers,
config,
configInput,
outputPath,
outputExtension,
contentBase,
})

outputFiles.push(...result.files)
droppedAfterBuild = result.sfcAfterBuildCount

await copyStatic(config, outputPath)
await events.fireAfterBuild({ files: outputFiles, config })
} finally {
await renderer.close()
} else {
const renderer = await createRenderer({ markdown: config.markdown, root: config.root, componentDirs: normalizeComponentSources(config.components?.source, process.cwd()), vite: config.vite })

try {
for (const templatePath of templateFiles) {
const { files } = await buildTemplate(templatePath, { config, renderer, events, outputPath, outputExtension, contentBase })
outputFiles.push(...files)
}

await copyStatic(config, outputPath)
await events.fireAfterBuild({ files: outputFiles, config })
} finally {
await renderer.close()
}
}

if (droppedAfterBuild > 0) {
console.warn(`[maizzle] Skipped ${droppedAfterBuild} SFC-registered afterBuild handler(s): afterBuild can't run inside a parallel build worker. Move build-completion logic to the config's afterBuild hook.`)
}

const duration = ((Date.now() - start) / 1000).toFixed(2)
Expand All @@ -165,30 +107,118 @@ export async function build(configInput?: Partial<MaizzleConfig> | string): Prom
}

/**
* Extract the static (non-glob) prefix from content patterns.
* Default template count above which parallel build turns on. Benchmarked
* crossover (with the worker cap below) is ~25 templates; 50 leaves margin so
* auto-parallel only kicks in where it's a reliable win across hardware.
* Override per project with `parallel: { threshold }`.
*/
const DEFAULT_PARALLEL_THRESHOLD = 50

/**
* Default worker cap. Each worker runs a full Vite SSR renderer, so startup +
* contention outweighs added parallelism past ~8 — benchmarks showed 8 beating
* 12/16/23 at every size. Override with `parallel: { workers }`.
*/
const DEFAULT_MAX_WORKERS = 8

/**
* Decide whether to build in parallel and with how many workers.
*
* For example, `['/abs/path/emails/**\/*.vue']` → `'/abs/path/emails'`
* `config.parallel`:
* - omitted → parallel when `count > 50`, min(CPU count − 1, 8) workers
* - `true` → always parallel (ignores threshold), default workers
* - `false` → always sequential
* - `{ workers, threshold }` → parallel when `count > threshold` (default 50),
* using `workers` threads (default min(CPU count − 1, 8))
*
* This is used to strip the content base from template paths
* so the output preserves only the subdirectory structure.
* Workers reload the config file to recover function hooks, so parallel only
* applies to file-based configs (a path or the default cwd config) — an inline
* config object has no file to reload and always builds sequentially.
*/
function computeContentBase(patterns: string[]): string {
// Use the first non-negated pattern
const pattern = patterns.find(p => !p.startsWith('!')) ?? patterns[0]
export function resolveParallel(
config: MaizzleConfig,
count: number,
configInput: Partial<MaizzleConfig> | string | undefined,
): { enabled: boolean; workers: number } {
const setting = config.parallel
if (setting === false) return { enabled: false, workers: 0 }

const fileBased = typeof configInput === 'string' || configInput == null
if (!fileBased) return { enabled: false, workers: 0 }

const cpus = availableParallelism()
const defaultWorkers = Math.min(Math.max(1, cpus - 1), DEFAULT_MAX_WORKERS)

let maxWorkers = defaultWorkers
let threshold = DEFAULT_PARALLEL_THRESHOLD
// `true` opts in regardless of count; object/omitted stay threshold-gated.
const ignoreThreshold = setting === true

if (typeof setting === 'object' && setting !== null) {
if (typeof setting.workers === 'number' && setting.workers > 0) maxWorkers = Math.floor(setting.workers)
if (typeof setting.threshold === 'number' && setting.threshold >= 0) threshold = Math.floor(setting.threshold)
}

// Split on first glob character (* { ? [) and take the directory part
const staticPart = pattern.split(/[*{?[]/)[0]
if (!ignoreThreshold && count <= threshold) return { enabled: false, workers: 0 }

// Ensure we have a clean directory path (not a partial segment)
return resolve(staticPart.endsWith('/') ? staticPart : dirname(staticPart))
const workers = Math.min(maxWorkers, count)
return { enabled: workers >= 2 && count >= 2, workers }
Comment on lines +164 to +165

@coderabbitai coderabbitai Bot Jun 5, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

parallel: true is still forced to sequential for a single template.

Line 165 requires workers >= 2 && count >= 2, so parallel: true and threshold: 0 do not actually force parallel for single-template builds, which conflicts with the documented behavior in this file.

Suggested fix 
-  return { enabled: workers >= 2 && count >= 2, workers }
+  return { enabled: workers >= 1 && count >= 1, workers }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const workers = Math.min(maxWorkers, count)
return { enabled: workers >= 2 && count >= 2, workers }
const workers = Math.min(maxWorkers, count)
return { enabled: workers >= 1 && count >= 1, workers }
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/build.ts` around lines 164 - 165, The current return forces sequential
runs for single-template builds because enabled is computed as workers >= 2 &&
count >= 2; update the condition to respect explicit parallel/threshold
settings: keep const workers = Math.min(maxWorkers, count) but return { enabled:
workers >= 2 && (count >= 2 || parallel === true || threshold === 0), workers }
(or use the actual option names in scope if different) so that explicit
parallel:true or threshold:0 will enable parallel even when count === 1; adjust
the function signature or captured variables to reference the surrounding
parallel and threshold config values used elsewhere in this module.

}

function resolveOutputPath(templatePath: string, outputDir: string, extension: string, contentBase: string): string {
const name = basename(templatePath).replace(/\.(vue|md)$/, '')
const absTemplate = resolve(templatePath)
const rel = relative(contentBase, dirname(absTemplate))
/**
* Run the build across worker threads. Each worker reloads the config (for its
* function hooks), builds its batch via the same `buildTemplate` as the
* sequential path, and returns the files it wrote. beforeCreate/afterBuild stay
* on the main thread (handled by the caller).
*/
async function runParallelBuild(opts: {
templateFiles: string[]
workers: number
config: MaizzleConfig
configInput: Partial<MaizzleConfig> | string | undefined
outputPath: string
outputExtension: string
contentBase: string
}): Promise<{ files: string[]; sfcAfterBuildCount: number }> {
const { templateFiles, workers, config, configInput, outputPath, outputExtension, contentBase } = opts

const { default: Tinypool } = await import('tinypool')
const workerPath = resolve(dirname(fileURLToPath(import.meta.url)), 'render/parallel/worker.mjs')

const configPath = typeof configInput === 'string' ? configInput : undefined
// Serializable snapshot of the post-beforeCreate config (functions dropped).
const configData = JSON.parse(JSON.stringify(config)) as Partial<MaizzleConfig>

const batches = shardEvenly(templateFiles, workers)

const pool = new Tinypool({ filename: workerPath, minThreads: batches.length, maxThreads: batches.length })

try {
const results = await Promise.all(
batches.map(templatePaths => pool.run({
templatePaths,
configPath,
configData,
outputPath,
outputExtension,
contentBase,
})),
)

return {
files: results.flatMap(r => r.files),
sfcAfterBuildCount: results.reduce((n, r) => n + r.sfcAfterBuildCount, 0),
Comment on lines +197 to +210

@coderabbitai coderabbitai Bot Jun 5, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

Parallel workers can nondeterministically overwrite the same output file.

At Line 197, batches are executed concurrently and each worker writes to disk inside buildTemplate(). If two templates resolve to the same output path (for example via custom output mappings), the final file becomes timing-dependent in parallel mode, while sequential mode is deterministic.

Please add collision protection (fail-fast on duplicate output targets, or write to per-worker temp locations and merge deterministically on the main thread).

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/build.ts` around lines 197 - 210, Detect and prevent duplicate output
targets before spawning parallel workers: before calling
Promise.all([...pool.run(...)]) flatten the batches and, using the same
output-resolution logic used by buildTemplate (extract that logic into or call a
new helper like computeOutputTarget(templatePath, configPath, configData,
outputPath, outputExtension) if necessary), compute each template's final output
file path deterministically, check for duplicates and fail-fast with a clear
error if any collisions exist; alternatively you can change pool.run workers to
write to per-worker temp files and then, on the main thread after results
return, atomically move/merge those temp files into the real output directory in
a deterministic order while checking for duplicates — reference batches,
pool.run, buildTemplate (or the new computeOutputTarget helper), results and
files when implementing the change.

}
} finally {
await pool.destroy()
}
}

return join(outputDir, rel, `${name}.${extension}`)
/** Round-robin items into up to `buckets` non-empty groups for even balance. */
function shardEvenly<T>(items: T[], buckets: number): T[][] {
const out: T[][] = Array.from({ length: buckets }, () => [])
items.forEach((item, i) => out[i % buckets].push(item))
return out.filter(b => b.length > 0)
}

async function copyStatic(config: MaizzleConfig, outputPath: string): Promise<void> {
Expand Down
Loading
Loading