From 8590abc1a2121747de06771404f613759bf98806 Mon Sep 17 00:00:00 2001 From: Xe Iaso Date: Wed, 7 Oct 2026 15:55:27 -0400 Subject: [PATCH] perf(storage/tigris): cap the zstd encoder window at 1 MiB The encoder() singleton used the default zstd window, so it kept 64 MiB of match history for the life of the process. After the bounded-memory pack writer, that history was most of the heap a push leaves live. Every input of the singleton is an object of at most inMemoryCap, a probeWindow head, or a block of a .cue or packed-refs. A window of inMemoryCap finds almost all of the same matches. Objects larger than inMemoryCap use streamEncPool, which this does not change. Measured with a real pack through ReadPack over a fake S3: - Live heap after a push: 72 MiB to 16 MiB, for Xe/x and golang/go. - Xe/x peak RSS: 204 MiB to 123 MiB. - golang/go peak RSS: about 476 MiB to 395 MiB. - .bin bytes: identical for both repositories. - .cue bytes: +0.09% for Xe/x, +0.22% for golang/go. Wall time does not change by a measurable amount. The plan records the full table. Assisted-by: Claude Opus 5.5 via Claude Code Signed-off-by: Xe Iaso --- docs/plans/push-memory.md | 35 +++++++++++++++++++++++++---- internal/storage/tigris/compress.go | 9 ++++++++ 2 files changed, 40 insertions(+), 4 deletions(-) diff --git a/docs/plans/push-memory.md b/docs/plans/push-memory.md index a9c2796..5ab523a 100644 --- a/docs/plans/push-memory.md +++ b/docs/plans/push-memory.md @@ -207,6 +207,37 @@ The wall time did not change by a measurable amount, at 25 s to 28 s on a laptop. One `pread` for each inflate, about 1.4 million calls, now takes most of the time. See phase 3. +## A smaller zstd window (done) + +After phase 2, the largest fixed cost of a push was the history of the zstd +encoder. The `encoder()` singleton used the default window, and it kept 64 MiB +of history for the life of the process. + +Each input of the singleton is an object of at most `inMemoryCap` (1 MiB), a +`probeWindow` head, or a block of a `.cue` or packed-refs. A window of +`inMemoryCap` therefore finds almost all of the same matches. Objects larger +than `inMemoryCap` use `streamEncPool`, and this change does not touch them. + +The numbers come from the harness of phase 1, with the same packs. + +| Measurement | Default window | 1 MiB window | +| ------------------------------- | -------------: | -------------: | +| Xe/x, live heap after the push | 72 MiB | 16 MiB | +| Xe/x, resolve peak heap | 213 MiB | 79 MiB | +| Xe/x, peak RSS | 204 MiB | 123 MiB | +| golang/go, live heap after push | 72 MiB | 16 MiB | +| golang/go, resolve peak heap | 450 to 478 MiB | 366 to 372 MiB | +| golang/go, peak RSS | 466 to 487 MiB | 394 to 397 MiB | +| Xe/x, `.bin` bytes | 41,996,083 | 41,996,083 | +| Xe/x, `.cue` bytes | 781,897 | 782,627 | +| golang/go, `.bin` bytes | 811,363,186 | 811,363,186 | +| golang/go, `.cue` bytes | 28,360,904 | 28,422,877 | + +The `.bin` bytes do not change, because each input on that path fits in the +window. A `.cue` block can be larger than 1 MiB, so the `.cue` files grow by +0.09% for Xe/x and by 0.22% for golang/go. The wall time does not change by a +measurable amount. + ## Phase 3: work to examine later - **Index size for very large repositories.** After phase 2, about 110 bytes @@ -219,10 +250,6 @@ of the time. See phase 3. page cache, and cgroup accounting counts them in both cases. - **The `.cue` zstd decoder.** It keeps about 45 MiB of buffers after its first use. A lower decoder concurrency cuts that. -- **The zstd encoder.** It keeps 64 MiB of history for each process. - `EncodeAll` inputs are at most `inMemoryCap` (1 MiB), except `.cue` blocks. - A 1 MiB window cuts that to about 8 MiB. This changes the compressed bytes, - so it needs its own measurement. - **Large bases in their own container.** An object larger than `maxPackBytes` gets a container of its own. A delta against it lands in the next container, so the writer stores that delta whole. This behavior is diff --git a/internal/storage/tigris/compress.go b/internal/storage/tigris/compress.go index 96b1715..edb8a38 100644 --- a/internal/storage/tigris/compress.go +++ b/internal/storage/tigris/compress.go @@ -120,6 +120,14 @@ func SetEncoderConcurrency(n int) { encoderConcurrency = n } +// encoderWindow is the match window of the encoder() singleton. Every input +// it compresses is an object of at most inMemoryCap, a probeWindow head, or a +// block of a .cue or packed-refs, so a window larger than inMemoryCap finds +// almost no more matches. Each encoder state keeps a history buffer sized off +// the window for the life of the process, so the default window cost 64 MiB +// of heap across encoderConcurrency states. +const encoderWindow = inMemoryCap + func encoder() *zstd.Encoder { zstdEncOnce.Do(func() { // Errors here are impossible with a nil writer and valid options; the @@ -127,6 +135,7 @@ func encoder() *zstd.Encoder { zstdEnc, _ = zstd.NewWriter(nil, zstd.WithEncoderLevel(zstd.SpeedDefault), zstd.WithEncoderConcurrency(encoderConcurrency), + zstd.WithWindowSize(encoderWindow), ) }) return zstdEnc