Skip to content

What a dump costs as the shard grows - #1350

Merged
bjmeetsfo merged 1 commit into
mainfrom
perf/what-a-dump-costs
Sep 8, 2026
Merged

What a dump costs as the shard grows#1350
bjmeetsfo merged 1 commit into
mainfrom
perf/what-a-dump-costs

Conversation

@bjmeetsfo

Copy link
Copy Markdown
Collaborator

A dump costs the shard, not the change.

flush_shard_index writes the index out with no bound on how much of it that is. This measures
what that means, holding the delta fixed: one write between dumps at every size, so what CHANGED is
identical and only the shard size varies.

 1,000 objects         77 ms
 5,000 objects        333 ms
20,000 objects      1,375 ms

Twenty times the objects, eighteen times the time, for the same single write. So the cost tracks
what the shard HOLDS rather than what happened to it, and a store pays more to checkpoint the
larger it gets. It pays it repeatedly, too: the cadence is driven by undumped bytes, so a big shard
does not dump less often to compensate -- it dumps just as often and each one costs more.

The counterpart in the design this follows bounds the work per round instead: it takes a capped
list of dirty slots, dumps that set, advances the dumped-log id, and leaves the rest for the next
round. The cost of one dump is then a function of the round limit rather than of the store.

This adds the measurement, not the fix, alongside the compaction probe that landed just before it.
Between them they say the same thing about two phases of the same cycle: ours are whole-shard where
theirs are bounded-per-round, and both costs grow with the store rather than with the work.

The probe is #[ignore], so it costs nothing on a normal run and can be asked for by name when
someone changes this path.

@bjmeetsfo
bjmeetsfo merged commit 6fc482d into main Sep 8, 2026
@bjmeetsfo
bjmeetsfo deleted the perf/what-a-dump-costs branch September 8, 2026 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants