Skip to content

What a whole-shard compaction costs - #1349

Merged
bjmeetsfo merged 1 commit into
mainfrom
perf/what-a-whole-shard-compaction-costs
Sep 8, 2026
Merged

What a whole-shard compaction costs#1349
bjmeetsfo merged 1 commit into
mainfrom
perf/what-a-whole-shard-compaction-costs

Conversation

@bjmeetsfo

Copy link
Copy Markdown
Collaborator

Compaction holds the whole shard, and nothing bounds how long.

compact_shard_pages takes the shard write lock and relocates EVERY live page of every model onto
a freshly rolled slab. No caller passes a budget and the function does not take one, so the work is
whatever the shard happens to hold. A write cannot proceed while it runs, so its duration is the
stall every reader and writer of that shard sees.

Measured, debug build, so a floor rather than a ceiling:

 1,000 objects        413 ms       413 us per page ref
 5,000 objects      1,876 ms       375 us per page ref
20,000 objects     19,664 ms       983 us per page ref

Twenty seconds at twenty thousand objects, reproduced at 20,162 ms on a second run. The per-ref
cost RISES with the shard -- 2.6x between five and twenty thousand -- so it is worse than linear in
the one thing it is unbounded in. A shard with millions of objects does not have a twenty-second
stall; it has a much longer one.

This adds the measurement, not a fix, and says why the fix is not a one-liner. The counterpart in
the design this follows scans a bounded number of slots per round and drives the rest through
callbacks, which is why its stall does not grow with the store. Simply stopping ours early would
not get there: it rolls a fresh slab at the top of every call, so a bounded round would leave its
own half-filled slab behind and trade one stall for slab proliferation. Bounding it properly means
rolling once and continuing to fill that slab across rounds -- campaign state the function does not
have.

The probe is #[ignore], so it costs nothing on a normal run and can be asked for by name when
someone changes this path. Its suite is 41 passed, 0 failed.

@bjmeetsfo
bjmeetsfo requested a review from superhaiou September 8, 2026 20:06
@bjmeetsfo
bjmeetsfo merged commit 9e30a83 into main Sep 8, 2026
@bjmeetsfo
bjmeetsfo deleted the perf/what-a-whole-shard-compaction-costs branch September 8, 2026 20:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants