Unpack the remote cache with several tar processes - #21
Open
kozak wants to merge 4 commits into
Open
Conversation
Restoring a cache bundle is dominated by per-file filesystem work, not by the download. The diagnostic added in the previous commit measured, across seven tasks on CI, 13.6s of downloading against 420.7s of unpacking - 97% of the time. The cost is per entry rather than per byte: a representative bundle is 80 MiB holding 56,845 entries, 77% of them under 4 KiB, and it takes ~23s to unpack, i.e. ~400us each. That is filesystem latency, so it parallelises. ParallelUnpack splits the tar stream across several concurrent `tar -x` processes. It parses only enough of each header to find where the entry ends and forwards the bytes verbatim, so permissions, mtimes, symlinks, hardlinks, long names and sparse files are all still handled by real tar rather than reimplemented. The archive format does not change, so bundles saved by any previous version still restore. Two things cannot be done concurrently and are held back for a final sequential pass: hardlinks, which need their target to exist, and directory entries, whose metadata must be applied after the files inside them. Separately, the workers must never create a directory themselves - two tar processes racing to auto-create the same parent silently loses files - so the splitter creates every directory before dispatching the entry that needs it. Verified against a real production bundle (56,845 entries) and against an archive built to cover the awkward cases: contents, modes, mtimes, symlink targets, hardlink inode sharing and sparse allocation all match a plain `tar -x` exactly. The worker count defaults to the processor count clamped to 4-8 and can be set with TASKRUNNER_UNPACK_WORKERS; 1 restores the previous single-process path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follows the review on #20. Decompressing in-process for the parallel path means a tap can go between zstd and tar, so restore now separates network from decompression from the file writes - which #20 could not do, since there zstd runs inside tar. As on the save path the taps nest rather than partition, so decompression is the difference between them. Two corrections to the previous numbers: - Dropped the feed loop's own timing. It blocks whenever a worker's pipe is full, so its wall clock conflated this thread's work with waiting for tar, and a large value read as "the splitter is slow" when it actually meant "the workers are the bottleneck". The tap either side of zstd attributes that correctly. - Report how long tar keeps extracting after the last byte is handed over. That tail is outside the pipeline's accounting, so without it the figures visibly failed to add up: a 700 MB single-file archive reported 0.00s of stalling against 0.63s elapsed, because all the work happened after feeding finished. All four combinations now sum to the elapsed time: 20k small files, 8 workers download 0.00s decompress 0.01s unpack 0.61s (0.63s) 20k small files, 1 worker download 0.00s decompress+unpack 0.42s tar tail 0.49s (0.91s) one 700 MB file, 8 workers download 0.00s decompress 0.04s unpack 0.57s (0.62s) one 700 MB file, 1 worker download 0.00s decompress+unpack 0.00s tar tail 0.62s (0.63s) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The diagnostic from #20 measured, across seven tasks on CI, 13.6s downloading against 420.7s unpacking — 97% of restore time. The cost is per entry, not per byte: a representative bundle (
concierge_BOOTSTRAP) is 80 MiB holding 56,845 entries, 77% of them under 4 KiB, and takes ~23s to unpack — roughly 400µs each. That is filesystem latency, so it parallelises.What
ParallelUnpacksplits the decompressed tar stream across N concurrenttar -xprocesses. It parses only enough of each 512-byte header to find where the entry ends and forwards the bytes verbatim, so permissions, mtimes, symlinks, hardlinks, GNU long names and sparse files are still handled by real tar rather than reimplemented.The archive format is unchanged, so bundles saved by any previous version still restore.
Two things cannot be done concurrently and go to a final sequential pass:
Separately, the workers must never create a directory themselves. Two
tarprocesses racing to auto-create the same parent silently loses files — at 8 workers an early version dropped 3,887 of 56,848 paths with no error. The splitter now creates every directory itself, single-threaded, before dispatching the entry that needs it.Numbers
This box is ~10x faster per file than CI (41µs/entry vs ~400µs), so the local win understates the expected one:
tar -x --zstd(today)The new debug line says why it does not go lower here:
splitting 1.24s, draining 0.00s— the tar workers finish as fast as they are fed, so the splitter is the local ceiling (~13µs/entry, isolated by comparing 444 MB in 11 entries against the same bytes in 56,845). On CI, where each entry costs ~10x more, the workers should be the ceiling instead.Verification
Against the real production bundle and a purpose-built torture archive, output is byte-identical to plain
tar -x: contents, modes, mtimes, symlink targets (44), hardlink inode sharing, sparse allocation, 5,717 paths over 100 chars, unicode and spaces in names, empty directories, a read-only 0500 directory, fifos, plus empty and single-huge-file archives. A corrupt header fails loudly instead of producing a broken tree.New golden test
remote-cache-parallel-unpackrestores the same cache with 4 workers and with 1 (the single-process path) and asserts both trees match what was saved, including modes and mtimes.Full suite: 53/53 with S3, 48/48 without.
Tuning
Worker count defaults to the processor count clamped to 4–8, overridable with
TASKRUNNER_UNPACK_WORKERS. Setting it to1restores the previous single-process path exactly.🤖 Generated with Claude Code