Skip to content

[ntuple] Avoid keeping too many files open in RNTupleChainProcessor - #23879

Merged
vepadulano merged 1 commit into
root-project:masterfrom
vepadulano:ntuple-processor-getnentries-cleanup-open-files
Oct 31, 2026
Merged

vepadulano merged 1 commit into
root-project:masterfrom
vepadulano:ntuple-processor-getnentries-cleanup-open-files

Conversation

@vepadulano

Copy link
Copy Markdown
Member

RNTupleProcessor::GetNEntries is called (at the moment) by RNTupleProcessorDS. This commit proposes to not keep the page sources after having opened them, but to only get the number of entries and then close them. This avoids the issue in
datasource_ntuple_processor.cxx test "error: Too many files open", seen on the Windows CI and visible also on a UNIX system by calling 'ulimit -n' with a number lower than the number of files of the chain in the test (1000).

RNTupleProcessor::GetNEntries is called (at the moment) by
RNTupleProcessorDS. This commit proposes to not keep the page
sources after having opened them, but to only get the number of entries
and then close them. This avoids the issue in
datasource_ntuple_processor.cxx test "error: Too many files open", seen
on the Windows CI and visible also on a UNIX system by calling 'ulimit
-n' with a number lower than the number of files of the chain in the
test (1000).
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Test Results

    24 files      24 suites   4d 0h 18m 29s ⏱️
 3 883 tests  3 883 ✅ 0 💤 0 ❌
83 646 runs  83 646 ✅ 0 💤 0 ❌

Results for commit 8771300.

♻️ This comment has been updated with latest results.

@vepadulano

Copy link
Copy Markdown
Member Author

Stopped CI runs to avoid overloading runners before the rc, will re-run afterwards.

@vepadulano

Copy link
Copy Markdown
Member Author

Just to show that on my Fedora node I see:

  • before this change
$: ulimit -n 256
$: /datasource_ntuple_processor --gtest_filter=RNTupleProcessorDSTest.Chain
Running main() from /builddir/build/BUILD/gtest-1.17.0-build/googletest-1.17.0/googletest/src/gtest_main.cc
Note: Google Test filter = RNTupleProcessorDSTest.Chain
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RNTupleProcessorDSTest
[ RUN      ] RNTupleProcessorDSTest.Chain
unknown file: Failure
C++ exception with description "Cannot open 'RNTupleProcessorDS_test.root', error: Too many open files" thrown in the test body.
  • after this change
$: ulimit -n 256
$: ./datasource_ntuple_processor --gtest_filter=RNTupleProcessorDSTest.Chain
Running main() from /builddir/build/BUILD/gtest-1.17.0-build/googletest-1.17.0/googletest/src/gtest_main.cc
Note: Google Test filter = RNTupleProcessorDSTest.Chain
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RNTupleProcessorDSTest
[ RUN      ] RNTupleProcessorDSTest.Chain
[       OK ] RNTupleProcessorDSTest.Chain (3394 ms)
[----------] 1 test from RNTupleProcessorDSTest (3394 ms total)

@vepadulano vepadulano closed this Oct 29, 2026
@vepadulano vepadulano reopened this Oct 29, 2026

@enirolf enirolf left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@vepadulano vepadulano closed this Oct 30, 2026
@vepadulano vepadulano reopened this Oct 30, 2026
@vepadulano

Copy link
Copy Markdown
Member Author

/backport to 6.42

@vepadulano
vepadulano merged commit 754aee0 into root-project:master Oct 31, 2026
59 of 110 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants