zlib: add ZIP archive support to zlib (ZipFile,ZipBuffer,ZipEntry) - #64339
Conversation
nodejs-github-bot
added
lib / src
pipobscure
changed the title
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #64339 +/- ##
==========================================
+ Coverage 90.17% 90.31% +0.13%
==========================================
Files 746 760 +14
Lines 242776 246928 +4152
Branches 45741 46578 +837
==========================================
+ Hits 218929 223003 +4074
- Misses 15347 15411 +64
- Partials 8500 8514 +14
🚀 New features to boost your workflow:
|
This comment was marked as outdated.
This comment was marked as outdated.
0f2b879 to
4258e94
Compare
Codecov flagged low patch coverage on lib/internal/zip.js and lib/internal/vfs/providers/archive.js in nodejs#64339. Add tests exercising Zip64 extra-field parsing, DOS date/time edge cases, streaming-entry state guards, decodeMemberStream/decodeMemberSync's duplicated error branches, the ZipBuffer/ZipFile iteration protocols, several on-disk ZipFile error paths, and ArchiveFileHandle's direct read/write/stat/ truncate surface plus a handful of provider-level error branches the existing tests didn't reach.
2a56a15 to
27ab6b4
Compare
|
See also #45651 |
Thanks @bakkot !!! I think the time has come for it on the one hand, and on the other I added some „motivation“ links earlier. Here some more detail: Based on this, we can modify the loader to directly load from an archive. If we do that, we get application bundles. pipobscure#5 & pipobscure#6 Which can then in turn be used to easily create application bundles: https://github.com/pipobscure/experimental-sea So the world had changed enough that it‘s worth proposing again. |
f157686 to
233f865
Compare
|
I like this a lot. It's likely better to split this into 2 PRs, one for Zip support and one for VFS-Zip, so the Zip support could theoretically be backportable on its own. |
This comment was marked as outdated.
This comment was marked as outdated.
|
As per @mcollina I split this into two PRs. I have the vfs-provider ready to go as follow up one (as it depends on this being merged) I also made sure that the streaming side of things was actually as clean as I intended, and added a few more tests. |
pipobscure
changed the title
pipobscure
changed the title
I've added a bunch more tests and compared to what go/python/info-zip do. (info-zip is cli exercise, so it's not entirely clear what gets tested). I also gave zip.js an once over and concluded that it was too large a file (it originated from separate files that I've had for ages combined into one). So I split it back out so it would be easier to review. And gave that another look. |
So we can't easily see and track the requested diff now. Unfortunate as otherwise we could review increments instead. |
Apologies. I thought the “requested modus operandi “ was to keep it to a single commit. If that’s wrong then I’ll change the approach for the future. |
No worries. FYI the commit-queue can land fixup commits (autosquash), or we can apply a label to squash, or if the PR has multiple things that can/should be backported independently (e.g. part is semver-major, part isn't) we can also land with rebase. |
contentIterator() (and ZipFile.stream()) must not apply the default getMaxZipContentSize() limit the way content()/contentSync() do. That default guards a single large buffer allocation, which the streaming path never makes: streaming is the bounded-memory way to read arbitrarily large members. Applying the ceiling there rejected legitimate reads of members larger than it - including the >4 GiB streaming case in test/pummel/test-zlib-zip-slow.js. Output stays bounded per chunk to the declared uncompressed size, and a caller that wants an explicit cap can still pass options.maxSize.
|
Loving tests right now. It caught me breaking streaming when doing the sec-fixes earlier. |
|
Can someone trigger CI, so that we can get this over the line? |
github-actions
Bot
removed
the
request-ci
jasnell
added
commit-queue
nodejs-github-bot
added
commit-queue-failed
Commit Queue failed- Loading data for nodejs/node/pull/64339 ✔ Done loading data for nodejs/node/pull/64339 ----------------------------------- PR info ------------------------------------ Title zlib: add ZIP archive support to zlib (ZipFile,ZipBuffer,ZipEntry) (#64339) ⚠ Could not retrieve the email or name of the PR author's from user's GitHub profile! Branch pipobscure:ziparchives -> nodejs:main Labels zlib, semver-minor, lib / src, experimental, needs-ci, commit-queue-rebase Commits 2 - zlib: add ZipEntry, ZipFile, and ZipBuffer - zlib: do not cap ZIP streaming reads by the content-size ceiling Committers 1 - Philipp Dunkel <pip@pipobscure.com> PR-URL: https://github.com/nodejs/node/pull/64339 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com> Reviewed-By: Aviv Keller <me@aviv.sh> ------------------------------ Generated metadata ------------------------------ PR-URL: https://github.com/nodejs/node/pull/64339 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com> Reviewed-By: Aviv Keller <me@aviv.sh> -------------------------------------------------------------------------------- ℹ This PR was created on Tue, 07 Jul 2026 14:54:34 GMT ✔ Approvals: 4 ✔ - Matteo Collina (@mcollina) (TSC): https://github.com/nodejs/node/pull/64339#pullrequestreview-4832884784 ✔ - James M Snell (@jasnell) (TSC): https://github.com/nodejs/node/pull/64339#pullrequestreview-4816314636 ✔ - Marco Ippolito (@marco-ippolito) (TSC): https://github.com/nodejs/node/pull/64339#pullrequestreview-4816360046 ✔ - Aviv Keller (@avivkeller): https://github.com/nodejs/node/pull/64339#pullrequestreview-4833086662 ✔ Last GitHub CI successful ℹ Last Full PR CI on 2026-07-31T23:54:15Z: https://ci.nodejs.org/job/node-test-pull-request/75350/ - Querying data for job/node-test-pull-request/75350/ ✔ Build data downloaded ✔ Last Jenkins CI successful -------------------------------------------------------------------------------- ✔ No git cherry-pick in progress ✔ No git am in progress ✔ No git rebase in progress -------------------------------------------------------------------------------- - Bringing origin/main up to date... From https://github.com/nodejs/node * branch main -> FETCH_HEAD ✔ origin/main is now up-to-date - Downloading patch for 64339 From https://github.com/nodejs/node * branch refs/pull/64339/merge -> FETCH_HEAD ✔ Fetched commits as 67af6d1d3917..fb49acaa84c0 -------------------------------------------------------------------------------- [main ca13b3f1fe] zlib: add ZipEntry, ZipFile, and ZipBuffer Author: Philipp Dunkel <pip@pipobscure.com> Date: Wed Jul 8 17:42:06 2026 +0200 36 files changed, 9665 insertions(+) create mode 100644 lib/internal/zip.js create mode 100644 lib/internal/zip/archive.js create mode 100644 lib/internal/zip/binary.js create mode 100644 lib/internal/zip/buffer.js create mode 100644 lib/internal/zip/compression.js create mode 100644 lib/internal/zip/constants.js create mode 100644 lib/internal/zip/content-size.js create mode 100644 lib/internal/zip/dos.js create mode 100644 lib/internal/zip/entry.js create mode 100644 lib/internal/zip/extra-fields.js create mode 100644 lib/internal/zip/file.js create mode 100644 lib/internal/zip/fs-util.js create mode 100644 lib/internal/zip/header-builders.js create mode 100644 lib/internal/zip/headers.js create mode 100644 test/parallel/test-zlib-zip-coverage.js create mode 100644 test/parallel/test-zlib-zip-edgecases.js create mode 100644 test/parallel/test-zlib-zip-encoding.js create mode 100644 test/parallel/test-zlib-zip-experimental-warning.js create mode 100644 test/parallel/test-zlib-zip-files.js create mode 100644 test/parallel/test-zlib-zip-fuzz.js create mode 100644 test/parallel/test-zlib-zip-hardening.js create mode 100644 test/parallel/test-zlib-zip-internals.js create mode 100644 test/parallel/test-zlib-zip-interop.js create mode 100644 test/parallel/test-zlib-zip-metadata.js create mode 100644 test/parallel/test-zlib-zip-property.js create mode 100644 test/parallel/test-zlib-zip-security.js create mode 100644 test/parallel/test-zlib-zip-sync.js create mode 100644 test/parallel/test-zlib-zip-writable.js create mode 100644 test/parallel/test-zlib-zip-zip64.js create mode 100644 test/parallel/test-zlib-zip.js create mode 100644 test/pummel/test-zlib-zip-slow.js [main 5f7867586e] zlib: do not cap ZIP streaming reads by the content-size ceiling Author: Philipp Dunkel <pip@pipobscure.com> Date: Thu Jul 30 21:49:20 2026 +0200 3 files changed, 31 insertions(+), 17 deletions(-) ✔ Patches applied There are 2 commits in the PR. Attempting autorebase. (node:1608) [DEP0190] DeprecationWarning: Passing args to a child process with shell option true can lead to security vulnerabilities, as the arguments are not escaped, only concatenated. (Use `node --trace-deprecation ...` to show where the warning was created) Rebasing (2/4) Executing: git node land --amend --yes --------------------------------- New Message ---------------------------------- zlib: add ZipEntry, ZipFile, and ZipBuffer
createZipArchive() serializes a sequence of entries into an archive Signed-off-by: Philipp Dunkel <pip@pipobscure.com>
|
mcollina
added
commit-queue
nodejs-github-bot
removed
the
commit-queue
|
Landed in 5ba72ae |
Add ZIP archive support to the node:zlib module through three classes and a set of helpers: - ZipEntry: a single archive member, with buffered reads (content()), bounded-memory streaming reads (contentIterator()), and create()/createStream() for building members. - ZipFile: random access to an archive backed by a file descriptor, reading members lazily without retaining their content and writing new members in place; opened with open()/openSync(). - ZipBuffer: a zero-copy, in-memory view over an archive already held in a Buffer. createZipArchive() serializes a sequence of entries into an archive byte stream, and setMaxZipContentSize() bounds the default in-memory decompression size. Every operation has both an asynchronous and a synchronous form. Signed-off-by: Philipp Dunkel <pip@pipobscure.com> PR-URL: #64339 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com> Reviewed-By: Aviv Keller <me@aviv.sh>
aduh95
added
dont-land-on-v22.x
14 participants
Summary
Add ZIP archive support to the node:zlib module through three classes
and a set of helpers:
bounded-memory streaming reads (contentIterator()), and
create()/createStream() for building members.
reading members lazily without retaining their content and writing
new members in place; opened with open()/openSync().
in a Buffer.
createZipArchive() serializes a sequence of entries into an archive
byte stream, and setMaxZipContentSize() bounds the default in-memory
decompression size. Every operation has both an asynchronous and a
synchronous form.
P.S.: my CLA should be on file and I wrote this myself so COO is declared