[io] Compress buffers larger than 16 MiB in blocks of equal size by rshrj · Pull Request #23707 · root-project/root

rshrj · 2026-10-13T22:50:25Z

This Pull request:

Changes or fixes:

Buffers larger than kMAXZIPBUF (2^24 − 1 bytes) are compressed in several blocks. So far all blocks but the last were kMAXZIPBUF bytes long, and if any block did not shrink the whole buffer was stored uncompressed. For a buffer just above a multiple of kMAXZIPBUF the last block is only a few bytes, which can never be compressed, so such buffers were always stored uncompressed (#23706): RNTuple pages of 2^24 bytes, every full page with a 32 MiB page size limit, and TTree baskets, TKey objects and TMessages of that size.

This PR cuts the buffer into the same number of blocks as before, but all of (nearly) the same size: ROOT::Internal::GetZipBlockSize(nbytes) in RZip.h, used by the five writers (RNTupleCompressor::Zip, TBasket::WriteBuffer, both TKey constructors, TMessage::Compress). No block is larger than kMAXZIPBUF, and no block is tiny.

The format does not change: all readers (TBasket::ReadBasketBuffers, TKey::UnzipBuffer, TTreeCacheUnzip, TMessage::Uncompress, RNTupleDecompressor::Unzip) take each block's size from its header. Older ROOT versions read the new files. Buffers up to kMAXZIPBUF bytes are compressed exactly as before; larger ones now compress to different, valid bytes.

Tests: RNTupleZip.LargeWithOutputBuffer asserted the old behaviour ("Trailing byte cannot be compressed, entire buffer returns uncompressed") and now checks that a kMAXZIPBUF + 1 buffer is compressed and read back. New: RNTupleZip.LargeJustAboveBlocks (sizes kMAXZIPBUF + 1, + 2 and 2^25, with ZLIB, LZMA, LZ4 and ZSTD) and, in io/io/test/ZipHeader.cxx, RZip.JustAboveBlockSize{TKey,TTree,Message}: an object, a basket and a message of exactly kMAXZIPBUF + 1 bytes must be compressed and read back.

Effect, with the reproducer of #23706 (ZSTD 5): RNTuple pages of 2^24, 2^24 + 1, + 19 and + 39 bytes take 566–574 bytes on storage instead of 16.8 MB; a TTree basket of 2^24 bytes takes 631 bytes instead of 16 777 281. On real data (CMS HGCAL hits written as RNTuple with 32 000 000-byte pages, 203 pages, 45 of them larger than one block), recompressing every page gives 312.9 instead of 346.5 MB with ZSTD 3, 308.3 instead of 341.8 MB with ZSTD 5 and 282.1 instead of 315.8 MB with LZMA 7; the difference is the pages that were stored uncompressed, the other pages hardly change.

On synthetic buffers (zeros, a compressible pattern and random data, 4 algorithms, sizes from kMAXZIPBUF − 1 to 10^8 bytes, 192 cases), 75 buffers that were stored uncompressed are now compressed, 94 are unchanged (all buffers up to kMAXZIPBUF and all random data), and 23 change by a few bytes or per cent in either direction. The largest increase is +5.5 % with LZMA, whose dictionary is capped at a quarter of each (now smaller) block.

Checklist:

tested changes locally
updated the docs (if necessary): not needed, no interface change

Tested on x86_64 (EL9, gcc 14.3) with two builds of 14a58f7 (master of 2026-10-12), with and without this change, -Dminimal=ON -Dtesting=ON: ctest -R '^gtest-(io|tree|net)' passes (100 of 100) in both; the new tests fail without the change.

This PR fixes #23706

Buffers larger than kMAXZIPBUF are compressed in several blocks. So far all blocks but the last were kMAXZIPBUF bytes long. For a buffer just above a multiple of kMAXZIPBUF, the last block is only a few bytes, which cannot be compressed, and the whole buffer was then stored uncompressed: RNTuple pages of 2^24 bytes (and every full page with a 32 MiB page size limit), TTree baskets, TKey objects and TMessages of that size. Cut the buffer into the same number of blocks, all of the same size instead (ROOT::Internal::GetZipBlockSize). Readers take each block's size from its header, so the format and the reading code are unchanged. Fixes root-project#23706

github-actions · 2026-10-14T13:48:16Z

Test Results

21 files 21 suites 3d 15h 12m 12s ⏱️
3 888 tests 3 886 ✅ 0 💤 2 ❌
72 123 runs 72 121 ✅ 0 💤 2 ❌

For more details on these failures, see this check.

Results for commit 62cb787.

pcanal · 2026-10-15T18:04:28Z

      Int_t nbuffers = 1 + (fObjlen - 1)/kMAXZIPBUF;
+      const Int_t blockSize = ROOT::Internal::GetZipBlockSize(fObjlen);


What would happen if GetZipBlockSize::nblocks and nbuffers get out of sync? Should we either assert that (fObjlen + blockSize - 1) / blockSize == nbuffers or (probably better) pass nbuffers as a parameter? (or even return both nbuffers and blockSize for the function if we want to delegate the entire split to that function).

I like the latter the most (returning both, block size and number of blocks from the helper function).

I agree with this. I changed the helper in c13eb40: GetZipBlockSize to GetZipBlocks which implements this

jblomer

LGTM but see open comments

jblomer · 2026-10-16T12:23:45Z

+/// stored uncompressed.
+constexpr int GetZipBlockSize(long long nbytes)
+{
+   const long long nblocks = 1 + (nbytes - 1) / kMAXZIPBUF;


Suggested change

const long long nblocks = 1 + (nbytes - 1) / kMAXZIPBUF;

assert(nbytes > 0);

const long long nblocks = 1 + (nbytes - 1) / kMAXZIPBUF;

jblomer · 2026-10-21T11:03:52Z

         if (i == nbuffers - 1) bufmax = fObjlen - nzip;
-         else bufmax = kMAXZIPBUF;
+         else
+            bufmax = blockSize;


Maybe reformat the entire if-else block

jblomer · 2026-10-21T11:06:46Z

      Int_t nbuffers = 1 + (fObjlen - 1)/kMAXZIPBUF;
+      const Int_t blockSize = ROOT::Internal::GetZipBlockSize(fObjlen);


I like the latter the most (returning both, block size and number of blocks from the helper function).

silverweed · 2026-10-21T14:35:39Z

+/// smaller by less than the number of blocks. Full blocks followed by the remainder would end a buffer just above a
+/// multiple of kMAXZIPBUF in a block of a few bytes, which cannot be compressed, so that the whole buffer would be
+/// stored uncompressed.
+constexpr int GetZipBlockSize(long long nbytes)


I would replace all instances of long long with std::uint64_t

Done (c13eb40) for the parameter. The returned values stay int because the callers use Int_t and they're still smaller than kMAXZIPBUF

Replace GetZipBlockSize by GetZipBlocks, which returns the number of blocks and their size (RZipBlocks), so the five writers no longer compute the count themselves and both values cannot disagree. GetZipBlocks asserts a positive size. RNTupleCompressor::Zip and TBasket::WriteBuffer now handle empty buffers before calling it; the result for them is unchanged.

rshrj force-pushed the zip-equal-blocks branch from ff2dfbc to f97e8ab Compare October 13, 2026 22:56

rshrj marked this pull request as ready for review October 13, 2026 22:57

rshrj requested review from dpiparo, jblomer, pcanal and silverweed as code owners October 13, 2026 22:57

rshrj force-pushed the zip-equal-blocks branch from f97e8ab to 62cb787 Compare October 13, 2026 23:00

rshrj changed the title ~~[core,io,tree,ntuple,net] Compress large buffers in blocks of equal size~~ [io] Compress buffers larger than 16 MiB in blocks of equal size Oct 13, 2026

jblomer self-assigned this Oct 14, 2026

jblomer added in:I/O experiment Affects an experiment / reported by its software & computimng experts labels Oct 14, 2026

pcanal reviewed Oct 15, 2026

View reviewed changes

jblomer approved these changes Oct 21, 2026

View reviewed changes

silverweed reviewed Oct 21, 2026

View reviewed changes

Navigation Menu

Repository navigation

[io] Compress buffers larger than 16 MiB in blocks of equal size - #23707

[io] Compress buffers larger than 16 MiB in blocks of equal size#23707
rshrj wants to merge 2 commits into
root-project:masterfrom
rshrj:zip-equal-blocks

rshrj commented Oct 13, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

pcanal Oct 15, 2026 •

edited

Loading

Uh oh!

jblomer Oct 21, 2026

Uh oh!

rshrj Oct 22, 2026 •

edited

Loading

Uh oh!

jblomer left a comment

Uh oh!

jblomer Oct 16, 2026

Uh oh!

rshrj Oct 22, 2026

Uh oh!

jblomer Oct 21, 2026

Uh oh!

rshrj Oct 22, 2026

Uh oh!

jblomer Oct 21, 2026

Uh oh!

silverweed Oct 21, 2026

Uh oh!

rshrj Oct 22, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		Int_t nbuffers = 1 + (fObjlen - 1)/kMAXZIPBUF;
		const Int_t blockSize = ROOT::Internal::GetZipBlockSize(fObjlen);

	const long long nblocks = 1 + (nbytes - 1) / kMAXZIPBUF;
	assert(nbytes > 0);
	const long long nblocks = 1 + (nbytes - 1) / kMAXZIPBUF;

Conversation

rshrj commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

This Pull request:

Changes or fixes:

Checklist:

Uh oh!

github-actions Bot commented Oct 14, 2026

Test Results

Uh oh!

pcanal Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jblomer Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

rshrj Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jblomer left a comment

Choose a reason for hiding this comment

Uh oh!

jblomer Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

rshrj Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

jblomer Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

rshrj Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

jblomer Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

silverweed Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

rshrj Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rshrj commented Oct 13, 2026 •

edited

Loading

pcanal Oct 15, 2026 •

edited

Loading

rshrj Oct 22, 2026 •

edited

Loading

rshrj Oct 22, 2026 •

edited

Loading