A storage cycle needs the shard, not a share of it - #1353
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A storage cycle does not give memory back, and it asks for a great deal more.
A write that carries its page leaves an entry in two side maps, and both are released when the
storage manager materialises that page. So the obvious question about the per-object memory measured
next door is whether running a cycle returns some of it. It does not.
Twenty thousand objects, debug build:
The cycle needs about 263 MB of working memory for a twenty-thousand-object shard -- roughly
thirteen kilobytes per object, wanted at once.
It is a peak, not a leak, and that is why the probe runs the cycle twice. The second cycle costs
4 KB, so the first one's memory is reused rather than retained. A single reading could not tell
those apart: resident memory does not fall when an allocation is returned to the allocator, so one
number is consistent with both "the cycle leaks 263 MB" and "the cycle needs 263 MB" -- and those
call for completely different work. The first would be a bug to hunt; the second is a capacity
requirement to plan for.
What it means in practice: maintenance on a large shard needs headroom proportional to the shard,
because every phase is whole-shard. That is the memory face of what the two probes above measure in
time -- compaction holding the shard for 19.7 seconds at twenty thousand objects, and a dump costing
1,375 ms for a single-write delta. The design this follows bounds each phase per round, so its
working set and its stall are both functions of the round limit rather than of the store.
The probe is
#[ignore], so it costs nothing on a normal run.