Repository navigation
Conversation
rshrj
force-pushed
the
zip-equal-blocks
branch
from
October 13, 2026 22:56
ff2dfbc to
f97e8ab
Compare
rshrj
marked this pull request as ready for review
October 13, 2026 22:57
rshrj
requested review from
dpiparo,
jblomer,
pcanal and
silverweed
as code owners
October 13, 2026 22:57
Buffers larger than kMAXZIPBUF are compressed in several blocks. So far all blocks but the last were kMAXZIPBUF bytes long. For a buffer just above a multiple of kMAXZIPBUF, the last block is only a few bytes, which cannot be compressed, and the whole buffer was then stored uncompressed: RNTuple pages of 2^24 bytes (and every full page with a 32 MiB page size limit), TTree baskets, TKey objects and TMessages of that size. Cut the buffer into the same number of blocks, all of the same size instead (ROOT::Internal::GetZipBlockSize). Readers take each block's size from its header, so the format and the reading code are unchanged. Fixes root-project#23706
rshrj
force-pushed
the
zip-equal-blocks
branch
from
October 13, 2026 23:00
f97e8ab to
62cb787
Compare
rshrj
changed the title
jblomer
added
in:I/O
experiment
Test Results 24 files 24 suites 4d 3h 52m 32s ⏱️ For more details on these failures, see this check. Results for commit c13eb40. ♻️ This comment has been updated with latest results. |
pcanal
reviewed
Oct 15, 2026
jblomer
approved these changes
Oct 21, 2026
jblomer
left a comment
jblomer
left a comment
Contributor
There was a problem hiding this comment.
LGTM but see open comments
silverweed
reviewed
Oct 21, 2026
| /// smaller by less than the number of blocks. Full blocks followed by the remainder would end a buffer just above a | ||
| /// multiple of kMAXZIPBUF in a block of a few bytes, which cannot be compressed, so that the whole buffer would be | ||
| /// stored uncompressed. | ||
| constexpr int GetZipBlockSize(long long nbytes) |
Contributor
There was a problem hiding this comment.
I would replace all instances of long long with std::uint64_t
Contributor
Author
There was a problem hiding this comment.
Done (c13eb40) for the parameter. The returned values stay int because the callers use Int_t and they're still smaller than kMAXZIPBUF
Replace GetZipBlockSize by GetZipBlocks, which returns the number of blocks and their size (RZipBlocks), so the five writers no longer compute the count themselves and both values cannot disagree. GetZipBlocks asserts a positive size. RNTupleCompressor::Zip and TBasket::WriteBuffer now handle empty buffers before calling it; the result for them is unchanged.
jblomer
reviewed
Oct 22, 2026
Comment on lines
+59
to
+60
| const unsigned int nZipBlocks = blocks.fNBlocks; | ||
| const int szBlock = blocks.fBlockSize; |
Comment on lines
+49
to
+50
| int fNBlocks; | ||
| int fBlockSize; |
Contributor
There was a problem hiding this comment.
Suggested change
| int fNBlocks; | |
| int fBlockSize; | |
| int fNBlocks = 0; | |
| int fBlockSize = 0; |
| { | ||
| assert(nbytes > 0); | ||
| const std::uint64_t nblocks = 1 + (nbytes - 1) / kMAXZIPBUF; | ||
| return RZipBlocks{static_cast<int>(nblocks), static_cast<int>((nbytes + nblocks - 1) / nblocks)}; |
Contributor
There was a problem hiding this comment.
Maybe let's add an R__ASSERT making sure that the two values stay within below the maximum value of int.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This Pull request:
Changes or fixes:
Buffers larger than
kMAXZIPBUF(2^24 − 1 bytes) are compressed in several blocks. So far all blocks but the last werekMAXZIPBUFbytes long, and if any block did not shrink the whole buffer was stored uncompressed. For a buffer just above a multiple ofkMAXZIPBUFthe last block is only a few bytes, which can never be compressed, so such buffers were always stored uncompressed (#23706): RNTuple pages of 2^24 bytes, every full page with a 32 MiB page size limit, and TTree baskets,TKeyobjects andTMessages of that size.This PR cuts the buffer into the same number of blocks as before, but all of (nearly) the same size:
ROOT::Internal::GetZipBlockSize(nbytes)inRZip.h, used by the five writers (RNTupleCompressor::Zip,TBasket::WriteBuffer, bothTKeyconstructors,TMessage::Compress). No block is larger thankMAXZIPBUF, and no block is tiny.The format does not change: all readers (
TBasket::ReadBasketBuffers,TKey::UnzipBuffer,TTreeCacheUnzip,TMessage::Uncompress,RNTupleDecompressor::Unzip) take each block's size from its header. Older ROOT versions read the new files. Buffers up tokMAXZIPBUFbytes are compressed exactly as before; larger ones now compress to different, valid bytes.Tests:
RNTupleZip.LargeWithOutputBufferasserted the old behaviour ("Trailing byte cannot be compressed, entire buffer returns uncompressed") and now checks that akMAXZIPBUF + 1buffer is compressed and read back. New:RNTupleZip.LargeJustAboveBlocks(sizeskMAXZIPBUF + 1,+ 2and 2^25, with ZLIB, LZMA, LZ4 and ZSTD) and, inio/io/test/ZipHeader.cxx,RZip.JustAboveBlockSize{TKey,TTree,Message}: an object, a basket and a message of exactlykMAXZIPBUF + 1bytes must be compressed and read back.Effect, with the reproducer of #23706 (ZSTD 5): RNTuple pages of 2^24, 2^24 + 1, + 19 and + 39 bytes take 566–574 bytes on storage instead of 16.8 MB; a TTree basket of 2^24 bytes takes 631 bytes instead of 16 777 281. On real data (CMS HGCAL hits written as RNTuple with 32 000 000-byte pages, 203 pages, 45 of them larger than one block), recompressing every page gives 312.9 instead of 346.5 MB with ZSTD 3, 308.3 instead of 341.8 MB with ZSTD 5 and 282.1 instead of 315.8 MB with LZMA 7; the difference is the pages that were stored uncompressed, the other pages hardly change.
On synthetic buffers (zeros, a compressible pattern and random data, 4 algorithms, sizes from
kMAXZIPBUF − 1to 10^8 bytes, 192 cases), 75 buffers that were stored uncompressed are now compressed, 94 are unchanged (all buffers up tokMAXZIPBUFand all random data), and 23 change by a few bytes or per cent in either direction. The largest increase is +5.5 % with LZMA, whose dictionary is capped at a quarter of each (now smaller) block.Checklist:
Tested on x86_64 (EL9, gcc 14.3) with two builds of 14a58f7 (master of 2026-10-12), with and without this change,
-Dminimal=ON -Dtesting=ON:ctest -R '^gtest-(io|tree|net)'passes (100 of 100) in both; the new tests fail without the change.This PR fixes #23706