Skip to content

Sixteen MiB log segments - #73

Closed
martian56 wants to merge 1 commit into
mainfrom
segments-final
Closed

Sixteen MiB log segments#73
martian56 wants to merge 1 commit into
mainfrom
segments-final

Conversation

@martian56

Copy link
Copy Markdown
Owner

Third attempt at measuring this; the first two were blocked by the harness rather than by the change.

Postgres flushes sixteen MiB log segments and ours were sixty four. Our durable commit costs about 2.5x theirs at the same group size and the same flush count, so if flush cost tracks the file extent rather than the bytes dirtied, this is a direct explanation.

Under the old harness it was the only candidate positive on both runs, linux-x64-full plus 11.4 then plus 35.8 percent, while every other profile disagreed with itself by up to 89 points. With both builds now benched on one runner, and a misbehaving baseline no longer failing the job, it can finally be judged.

Postgres flushes sixteen MiB segments; ours were sixty four. Rotation is
set below the segment size so a segment never has to be extended.
@coderabbitai

coderabbitai Bot commented Jul 25, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@martian56, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 9 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: a40cb49b-9534-4c96-9cc6-f29be1f0634d

📥 Commits

Reviewing files that changed from the base of the PR and between 5718650 and 9a54ce0.

📒 Files selected for processing (2)
  • storage/src/group_commit.rs
  • storage/src/wal.rs
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch segments-final

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

Bench report

⚠️ Worst regression: windows-x64-full writes at -33.9%.

Baseline: main @ same runn. Cells read baseline → this PR. Verdicts need more than ±15% to leave the noise band.

linux-arm64-full

workload ops/s Δ ops/s read p99 ms write p99 ms write p99.9 ms
reads 30655 → 30923 ⚪ +0.9% 0.23 → 0.23 - -
mixed 14527 → 12300 🔴 -15.3% 0.50 → 0.49 11.20 → 19.35 32.47 → 35.89
writes 5471 → 4903 ⚪ -10.4% - 9.07 → 11.02 28.71 → 29.67

linux-x64-1cpu-1gb

workload ops/s Δ ops/s read p99 ms write p99 ms write p99.9 ms
reads 13561 new 0.42 - -
mixed 8171 new 0.62 19.40 28.53
writes 4476 new - 8.21 24.00

linux-x64-2cpu-2gb

workload ops/s Δ ops/s read p99 ms write p99 ms write p99.9 ms
reads 17517 new 0.39 - -
mixed 12099 new 0.64 3.20 14.00
writes 6267 new - 2.28 8.19

linux-x64-full

workload ops/s Δ ops/s read p99 ms write p99 ms write p99.9 ms
reads 45409 → 45507 ⚪ +0.2% 0.15 → 0.16 - -
mixed 10848 → 7626 🔴 -29.7% 0.23 → 0.25 29.69 → 56.73 115.37 → 125.78
writes 3711 → 3606 ⚪ -2.8% - 15.47 → 15.76 59.97 → 82.96

windows-x64-full

workload ops/s Δ ops/s read p99 ms write p99 ms write p99.9 ms
reads 22727 → 22792 ⚪ +0.3% 0.36 → 0.34 - -
mixed 5452 → 4706 ⚪ -13.7% 0.48 → 0.33 30.48 → 31.09 41.87 → 42.08
writes 873 → 577 🔴 -33.9% - 15.61 → 31.76 24.45 → 42.49

Field reference

All engines in identical containers, same driver, measured 2026-07-24 @ ba0ec39. A different harness than the tables above, so read this as standing, not as this PR's delta.

linux-arm64-full

engine reads mixed writes
quantadb 17978 12420 (27ms wp99) 6449 (19ms wp99)
postgres18 15227 14723 (2ms wp99) 9923 (1ms wp99)
mariadb114 4221 4437 (8ms wp99) 6666 (3ms wp99)
mysql84 4258 4394 (9ms wp99) 4274 (4ms wp99)

linux-x64-1cpu-1gb

engine reads mixed writes
quantadb 19419 10854 (29ms wp99) 5170 (39ms wp99)
postgres18 10196 8580 (43ms wp99) 5912 (46ms wp99)
mariadb114 3455 3732 (4ms wp99) 5626 (8ms wp99)
mysql84 3497 3628 (4ms wp99) 3259 (36ms wp99)

linux-x64-2cpu-2gb

engine reads mixed writes
quantadb 22528 16774 (5ms wp99) 8374 (4ms wp99)
postgres18 17060 17006 (2ms wp99) 11404 (1ms wp99)
mariadb114 3735 4006 (3ms wp99) 6741 (3ms wp99)
mysql84 3804 3845 (4ms wp99) 4847 (3ms wp99)

linux-x64-full

engine reads mixed writes
quantadb 18144 15075 (6ms wp99) 8025 (5ms wp99)
postgres18 15402 15154 (2ms wp99) 10872 (1ms wp99)
mariadb114 3437 3699 (4ms wp99) 6138 (3ms wp99)
mysql84 3490 3615 (4ms wp99) 3996 (5ms wp99)

Shared CI runners: treat single-profile swings as suggestive, agreement across profiles as real.

@martian56

Copy link
Copy Markdown
Owner Author

Measured and rejected, this time on a baseline from the same runner.

  • linux-x64-full: minus 2.8 percent, inside the band
  • linux-arm64-full: minus 10.4 percent, inside the band
  • windows-x64-full: minus 33.9 percent, a clear regression
  • the docker profiles have no paired baseline yet

The plus 11.4 and plus 35.8 percent this showed on x64-full under the old harness was an artifact. Paired against the same runner it is neutral to negative everywhere.

Worth stating plainly, because it is the point of the harness work: this is a change I believed in on the strength of two runs agreeing, and it was wrong. Segment size is not why our commit costs about 2.5x a Postgres commit.

@martian56 martian56 closed this Jul 25, 2026
@martian56
martian56 deleted the segments-final branch July 25, 2026 08:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant