Repository navigation
Conversation
Test Results 22 files 22 suites 3d 14h 42m 50s ⏱️ Results for commit 1575491. ♻️ This comment has been updated with latest results. |
jblomer
force-pushed
the
ntuple-huge-file-read
branch
from
October 30, 2026 08:44
e08aafa to
f8023ed
Compare
...in addition to the pinned clusters that induce pinning of their cluster group.
Move "searching for column index in a given cluster group" and "binary search for the column index in the clusters of a cluster group" into their own helper methods.
- Look first into the cluster group of the current cluster and one adjacent cluster group before full binary search. - Look through pinned cluster groups - "Top-level columns" (columns that have one element per entry) can use the more efficient entry-wise FindClusterId()
pcanal
reviewed
Oct 30, 2026
pcanal
reviewed
Oct 30, 2026
| const auto clusterGroupBudget = Internal::RNTupleReadOptionsManip::GetClusterGroupBudget(fOptions); | ||
| assert(clusterGroupBudget > 0); | ||
| if ((nClusterGroups >= clusterGroupBudget) && (nClusterGroups > fPinnedClusterGroups.size())) { | ||
| auto nRemove = std::min(nClusterGroups - fPinnedClusterGroups.size(), nClusterGroups - clusterGroupBudget + 1); |
Member
There was a problem hiding this comment.
Why the '+1'? I recommend adding a code comment (eg 'Remove at least cluster group if they are not all pinned' (or whatever the correct explanation is :)).
pcanal
reviewed
Oct 30, 2026
| assert(clusterGroupBudget > 0); | ||
| if ((nClusterGroups >= clusterGroupBudget) && (nClusterGroups > fPinnedClusterGroups.size())) { | ||
| auto nRemove = std::min(nClusterGroups - fPinnedClusterGroups.size(), nClusterGroups - clusterGroupBudget + 1); | ||
| for (auto lru = fClusterGroupAccessSequence.rbegin(); nRemove > 0;) { |
Member
There was a problem hiding this comment.
I recommend a code comment indicating the semantic reason to use the reverse iteration.
jblomer
force-pushed
the
ntuple-huge-file-read
branch
from
November 2, 2026 12:50
f8023ed to
1575491
Compare
Limit the number of concurrently loaded cluster group details. The corresponding read options stays internal for the time being.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Verified with a small benchmark that this change has the desired effect on memory consumption: jblomer/iotools@f355b8a