Problem
ZIP archive creation allocates ~207 MB for 500 x 1 KB files, compared to 326 KB for the same workload with TAR. 99.6% of allocations originate inside the zip crate which re-creates the DEFLATE compressor for each entry.
Profiling data (dhat, 500 files x 1 KB)
| Format |
Total Allocated |
Peak Live |
| TAR |
326 KB |
152 KB |
| ZIP |
207.2 MB |
827 KB |
Root cause
zip::GenericZipWriter::switch_to allocates a new flate2::Compress (~380 KB) and flate2::zio::Writer buffer (~32 KB) on every start_file call instead of reusing them across entries.
Upstream
Reported at zip-rs/zip2#633
Workaround
None available at the library level. For workloads with many small files, using CompressionMethod::Stored avoids the issue entirely.
Problem
ZIP archive creation allocates ~207 MB for 500 x 1 KB files, compared to 326 KB for the same workload with TAR. 99.6% of allocations originate inside the
zipcrate which re-creates the DEFLATE compressor for each entry.Profiling data (dhat, 500 files x 1 KB)
Root cause
zip::GenericZipWriter::switch_toallocates a newflate2::Compress(~380 KB) andflate2::zio::Writerbuffer (~32 KB) on everystart_filecall instead of reusing them across entries.Upstream
Reported at zip-rs/zip2#633
Workaround
None available at the library level. For workloads with many small files, using
CompressionMethod::Storedavoids the issue entirely.