Repository navigation
Conversation
TFileCacheRead::ReadBufferExtNormal sorts the list of blocks to read, which marks it as sorted, before reading them all into fBuffer with TFile::ReadBuffers. When that read fails, e.g. on an XRootD socket timeout, only the request that triggered it reports an error: the following requests for blocks of the same list find them in the list and are served from fBuffer, which still holds the blocks of the previous list, or nothing at all. The baskets of the rest of the cluster are thus unzipped and streamed from unrelated data. Reading LHCb DSTs over XRootD, a single timeout is followed by "Inconsistency found in header (nin=0, nbuf=0)" errors and allocations of several GB from the bogus content. Reset the list of blocks when reading them fails, so that subsequent requests miss the cache and are read from the file. Assisted-by: claude-code:claude-opus-5-5
TTreeCacheUnzip::GetUnzipBuffer reads the requested basket from the file only when TTreeCache::ReadBufferExt reports a miss. When that call itself fails to read the blocks of the cache, it returns -1, and the basket is unzipped from whatever fCompBuffer holds: the previous basket, or uninitialised memory. This happens when learning prefill has filled the cache before the first basket is requested. Treat a failure as a miss, so that the basket is read from the file. Assisted-by: claude-code:claude-opus-5-5
chrisburr
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tangentially related to #23831 in that this has been found in response to seeing grid jobs in LHCb that allocate unreasonable amounts of memory. See https://gitlab.cern.ch/gaudi/Gaudi/-/work_items/348 for an example of the symptom I target in this PR.
It's hard to know exactly what is going on in the job however I do know that I see this when:
This led me to the first commit (47a4690) and it seems plausible that the cache would result in future reads consuming garbage data. Claude has written a test which seems like a reasonable way of mocking this failure mode and without the fix it reproduces the same error messages as shown in the Gaudi bug report.
Changes or fixes:
TTreeCacheUnziponly use the cached block if the read was successful. Falls back to trying the read a second time, else errors.Checklist:
This PR (hopefully) fixes https://gitlab.cern.ch/gaudi/Gaudi/-/work_items/348
This PR was prepared with the assistance of Claude Code (Opus 5.5).