Run commands concurrently with --jobs, ship as 0.5.0 - #2
Merged
Merged
Conversation
Three additive changes to lib/manifest.js, all of which keep getCommands()
output byte-identical (its existing expectations pass untouched):
- getCommandsByType() walks contexts × tags exactly as getCommands() does but
returns ordered [type, commands] pairs, dropping types which produced no
command. getCommands() is now just its flattening, so the two cannot drift.
Commands within a type are independent of each other while types are not
(nothing is tested before it is built), which is the grouping a parallel
executor needs — that follow-up will consume this.
- A template which renders to nothing (trimmed) no longer emits a command;
previously it produced an empty string that ran as `sh -c ''`. This lets a
manifest wrap a template in `{{^alias}}…{{/alias}}` so tags which are pure
retags of another image are skipped for that type while still being pushed.
- `structureTest: false` (and its structure_test/structure-test aliases) is an
explicit opt-out from the parent → child config concatenation, which until
now no tag could escape. As a child it wins outright; as a parent it means
"nothing to inherit", so a tag under an opted-out context can re-enable with
its own configs. Semantics are documented at mergeStructureTest().
Tests: 59 → 80, with new describes for the grouping and ordering, a
getCommands()/getCommandsByType() equivalence check over the fixture's cases,
the empty-render skip (new test/retag-manifest.yml fixture, kept separate so
the shared test/manifest.yml expectations in indexTest/executableTest stay
valid), and the opt-out at tag, context and global level. manifest.js branch
coverage 90.81% → 92.72%, statements/lines/functions still 100%; the remaining
uncovered branches are pre-existing defensive fallbacks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dave's commands were always independent within a type (builds of different tags don't touch each other) but strictly ordered across types (nothing can be tested before it is built). The executor now reflects exactly that: main() walks manifest.getCommandsByType() with a barrier between types — every command of a type finishes before the next type starts, and a type with failures ends the run. Within a type: * -j 1 (default) keeps the old path bit-for-bit: runSerial wraps the untouched runCommand (spawn with inherited stdio), so output still streams live and a first failure aborts with the classic 'Command failed: <cmd>' error and exit 1. * --jobs N runs a bounded worker pool (runPool): min(N, #commands) workers pull from a shared queue front-to-back, so commands start in the same deterministic sorted order the manifest produces. Each child runs through runCommandCaptured — spawn with piped stdio, stdout+stderr chunks buffered together in arrival order and replayed as ONE contiguous block on completion, bracketed by 'Running:' and 'Finished (Ns):' lines. Buffering is deliberately uncapped: exec()'s maxBuffer would kill a verbose child (the same pitfall runCommand already documents), and capturing is the whole point here, so we accumulate raw chunks ourselves. Concurrent children therefore never interleave a single line of output. * On failure the pool stops handing out new work but never kills in-flight children; --keep-going/-k drains the rest of the type's queue instead (later types still never start). A single failure throws the classic error; several throw an 'N command(s) failed during <type>:' summary with one line and exit code per command. --jobs validates via coerce (whole numbers >= 1 only) and argv parsing now uses .fail(false) so bad options reject cleanly instead of yargs exiting mid-parse; the entry-point handler already reports and exits 1. Timing sanity (8 tags, build = sleep 1): 8.2s at -j 1 vs 2.1s at -j 4. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Publishing shipped nearly the whole repo — most of all the 9.3 MB decorative GIF under .images, plus tests and coverage output — because .npmignore had gone stale: it still listed .eslintrc.yml and .travis.yml and never covered anything added since. Replace it with a "files" allowlist, which fails safe in the other direction: whatever lands in the repo next stays out of the tarball unless it is listed. The package goes from 9.5 MB across 43 files to 10.8 kB across 5. Document --jobs/-j and --keep-going/-k in the README, together with the two manifest-level opt-outs added earlier in this branch: a template which renders to nothing skips the command, and structureTest: false drops the configs which would otherwise trickle down. Add a CHANGELOG whose first entry is 0.5.0, and bump the version to match. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
package.json and the README have always declared MIT, but the repo (and therefore the published npm tarball) carried no license text. npm auto- includes LICENSE regardless of the "files" allowlist, so 0.5.0 now ships it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dave has always executed its rendered commands strictly serially — one
for…of awaitloop. For a manifest like docker-dataset's (~105 buildable tags), that means every
build, test, and push runs one at a time. This PR adds opt-in parallelism plus the
manifest features needed to skip alias tags, and trims the published package.
New flags
--jobs/-j N— run up to N commands of the same type concurrently.Guarantees: types stay strictly ordered (
build → test → structure-test → push,hard barrier between them); at
-j 1(the default) behavior is byte-identical to0.4.1 with live-streaming output; at
-j > 1each command's output is bufferedand printed as one contiguous block when it finishes — never interleaved.
--keep-going/-k— on failure, remaining commands of the same type stillrun; later types never start; all failures are summarized and the exit code is
non-zero.
The pool path uses
spawnwith self-managed pipe buffering (uncapped), so theexec()maxBuffer cliff that 79e19e0 fixed cannot regress.New manifest features
produces no command. Wrapping a template in
{{^retagFrom}}…{{/retagFrom}}letsalias tags (e.g.
latest) skip build/test/push entirely.structureTest: false— per-tag/per-context opt-out from inheritedstructure-test configs (previously configs always concatenated down).
getCommandsByType();getCommands()is reimplemented on top ofit, output unchanged.
Packaging
.npmignoreis replaced with a"files"allowlist. The published tarball dropsfrom 9.5 MB to 12.7 kB (a decorative 9.3 MB GIF was shipping in every install).
A LICENSE file is added (package.json always declared MIT; the tarball now
actually carries the text).
Compatibility
Fully backward compatible: no flags → same serial, live-streaming execution as
0.4.1. Two minor behavior notes (also in CHANGELOG.md): an empty rendered template
no longer runs
sh -c '', and invalid flag values print the error without thefull usage dump.
Testing
running dave as a child process at
-j 3; type barrier and keep-going coveredby dedicated fixtures.
sleep 1→ 8.2 s at-j 1, 2.1 s at-j 4.npm pack --dry-run: 6 files, 12.7 kB.After merging: tag
v0.5.0andnpm publish— the docker-dataset CI redesignpins
DAVE_VERSION: 0.5.0and must not merge before the package is live.🤖 Generated with Claude Code