Skip to content

perf(storage/tigris): push any repository in bounded memory - #25

Merged
Xe merged 8 commits into
mainfrom
perf/push-memory
Sep 25, 2026
Merged

perf(storage/tigris): push any repository in bounded memory#25
Xe merged 8 commits into
mainfrom
perf/push-memory

Conversation

@Xe

@Xe Xe commented Sep 25, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • A push of golang/go (465 MiB pack, 708,000 objects) used more than 21 GiB of heap and did not complete. It now peaks at about 364 MiB and takes about 25 s.
  • The cost was in go-git. The scratch filesystem.Storage parses a pack with no storage attached, so the parser keeps every inflated base and every resolved delta until the parse ends. A new small index-pack in indexpack.go replaces it.
  • The pack index is now a list of sorted tables, and not a map. It uses about 60 bytes for each object, where the map used about 210. This helps clones and fetches too: a cold index build of golang/go holds 40 MiB of index, where it held about 180 MiB.
  • This also fixes an existing fault on main: a stored delta chain of exactly 50 links did not read back. Git makes such chains by default.

Details

The full plan, with every measurement, is in docs/plans/push-memory.md.

The new pack reader (indexpack.go). The reader makes two passes:

  1. scanPack reads the pack as it arrives. It stages the bytes, verifies the trailer checksum, and keeps about 60 bytes for each object. It keeps no body.
  2. The resolver walks each delta tree depth first. A body lives only while a delta below it needs it as a base. A body larger than 8 MiB goes to a temp file. When the bodies in memory would pass 64 MiB, the next body goes to a temp file. The delta applier reads its base through io.ReaderAt.

writePack now hands the stream to the storer through an optional ReadPack method. It no longer runs a go-git Scanner, which buffered the full body of each object. ReadPack stops at the trailer, so git:// and SSH still work.

Delta chain depth. Reads now accept a chain of exactly maxDeltaDepth (50) links. A client can send up to 4095 links, so the writer stores an object whole when its stored chain is already 50 links long.

The pack index (packtable.go). Tables never change after they are built. Each registered container adds a table, and more than 16 tables merge into one. The iterator no longer copies every entry or keeps a set of the hashes it has seen. deregister takes a pack id, so packJob no longer holds its cueRecords.

Push-path syscalls. Pack segments now write through a 1 MiB buffer, and a small section is read with one ReadAt of its own length. The wall time did not change measurably, because one pread for each inflate is now the main cost. The plan records a memory map of the staged pack as the next step.

golang/go main This branch
Push, peak heap more than 21 GiB 364 MiB
Push, time did not complete about 25 s
Live heap after the push – 114 MiB
Cold index build, index memory about 180 MiB 40 MiB

For Xe/x (50 MB pack), a push goes from 516 MiB and 14.7 s to about 165 MiB and 1.3 s. Delta reuse did not change: both writers keep 12,016 deltas and store 39 MiB.

Not in this PR:

  • segmentStore is an earlier, unused attempt at the same fix, and only its own tests call it. It needed one flush to keep working, and it can be deleted in a later change.
  • TestSmartHTTPHealsDanglingHEAD fails under -race on main too, in go-git's memory.Storage. This branch does not change it.

Test plan

  • Code compiles with go build ./...
  • Code formatted with npm run format (prettier on the changed docs, gofmt on Go)
  • go test ./... passes. go test -race passes for internal/storage/tigris.
  • New tests push hand-built packs through both ReadPack and Write: every object type, OFS-delta chains, REF-deltas before and after their base, a REF-delta on a base already in the repository, a chain three times deeper than a read walks, a chain of exactly 50 links, and every body on disk. Reject cases: bad signature, bad checksum, truncated pack, bad OFS distance, short entry, wrong delta base size, and missing REF base. Removing the chain cap or the base-size check makes the tests fail.
  • Manual testing: a local harness pushed the full golang/go pack into an S3 fake, then read back all 708,000 objects through a cold Storer and verified each hash. It also passed for Xe/x, a 197-link chain, and a 266 MB blob with a delta, in default mode and with every body on disk.
  • A real push of a large repository to a Tigris bucket with cmd/membench

Xe added 8 commits September 25, 2026 16:28
@Xe
perf(storage/tigris): index pushed packs in bounded memory
b1fae8d 
A push decoded its pack through a scratch go-git filesystem.Storage.
That storage parses with no storage attached, so go-git's parser stays
out of low-memory mode and keeps every inflated base and resolved delta
until the parse ends. A push of golang/go (465 MiB pack) passed 21 GiB
of heap and did not complete.

Replace it with a small index-pack in indexpack.go. One pass stages,
checksums, and indexes the pack as it arrives, with no bodies kept. A
second pass resolves each delta tree depth first, so a body lives only
while a delta below it needs it. Bodies over 8 MiB, or past a 64 MiB
total, go to temp files. writePack hands the stream over through a new
optional ReadPack method, which also removes a go-git Scanner that
buffered every object body in full.

golang/go now peaks at about 521 MiB of heap and takes 23 s. Xe/x goes
from 516 MiB and 14.7 s to about 165 MiB and 1.3 s. Delta reuse is
unchanged.

This also fixes stored delta chains of exactly 50 links, which git
makes by default and which reads refused. Reads now accept a chain of
maxDeltaDepth links, and the writer stores an object whole rather than
extend a stored chain past that, since clients can send 4095.

Assisted-by: Claude Opus 5.5 via Claude Code
Signed-off-by: Xe Iaso <xe@tigrisdata.com>
@Xe
docs: plan bounded push memory and describe the new pack writer
db4bd47 
Record the measurements behind the new pack writer, the delta chain
depth fault it found, and the next phase: a compact pack index to
replace the per-object entries map.

Assisted-by: Claude Opus 5.5 via Claude Code
Signed-off-by: Xe Iaso <xe@tigrisdata.com>
@Xe
perf(storage/tigris): replace the pack index map with sorted tables
b36f565 
The pack index kept a map[plumbing.Hash]packEntry, about 210 bytes for
each object. Every push held one entry for each object it wrote, and
every clone or fetch built one for each object in the repository.

Keep the entries in immutable tables instead: the hashes in one sorted
[]byte and a 40-byte record for each object, about 60 bytes with SHA-1.
A record names its pack by number and its delta base by record number.
Each registered container adds a table, and more than 16 tables merge
into one, so a lookup is at most 16 binary searches. deregister takes
only a pack id, so packJob no longer holds its cueRecords until the
upload ends.

The iterator walks 8-byte record references instead of a copy of every
entry, and yields a hash only from the record the lookup returns, which
removes its set of seen hashes. A snapshot shares the tables with the
index rather than copying them, because the index never edits them in
place.

For golang/go, a cold index build drops from 225 MiB to 113 MiB of live
heap (the index itself from about 180 MiB to 40 MiB). A push peaks at
364 MiB instead of 521 MiB.

Assisted-by: Claude Opus 5.5 via Claude Code
Signed-off-by: Xe Iaso <xe@tigrisdata.com>
@Xe
perf(storage/tigris): cut system calls on the push path
5c59e92 
A CPU profile of a golang/go push spent about 80% of its time in system
calls. packSegment wrote each object to its file with no buffer, which
cost 14.7 s of a 26 s push. Write through a 1 MiB buffer and count the
bytes, so copyCompressed no longer seeks to measure a payload. The
unused segmentStore reads from an open segment, so it flushes first.

Each inflate also read its section through a 32 KiB buffer, so even a
tiny object read 32 KiB. Read a section of 64 KiB or less whole, with
one ReadAt of its own length.

Wall time did not move measurably on a laptop: one pread per inflate
now dominates. The plan records mmap of the staged pack as the next
step.

Assisted-by: Claude Opus 5.5 via Claude Code
Signed-off-by: Xe Iaso <xe@tigrisdata.com>
@Xe
docs: record the compact pack index and its measurements
593dfdb 
Describe the table-based pack index in the storer architecture, and
add the phase 2 results and the remaining push-time work to the plan.

Assisted-by: Claude Opus 5.5 via Claude Code
Signed-off-by: Xe Iaso <xe@tigrisdata.com>
@Xe
style(storage/tigris): gofmt iter.go
b5a5f0b 
Assisted-by: Claude Opus 5.5 via Claude Code
Signed-off-by: Xe Iaso <xe@tigrisdata.com>
@Xe
test(storage/tigris): cover the pack reader with hand-built packs
52358c0 
The existing pack tests reach the new pack reader only through Write,
with packs from the git CLI, so they never send a REF-delta, a delta
whose base is already in the repository, a chain deeper than a read
walks, or a malformed pack.

Build packs by hand and push each one through both ReadPack and Write.
The round-trip cases read every object back through a new Storer and
check the stored delta count and the longest stored chain. The reject
cases check the error and that no PutObject call was made. One more
test pins that ReadPack leaves the bytes after the trailer unread,
which writePack depends on for git:// and SSH.

Removing the chain-depth cap or the delta source-size check makes
these tests fail.

Assisted-by: Claude Opus 5.5 via Claude Code
Signed-off-by: Xe Iaso <xe@tigrisdata.com>
@Xe
docs: format the push memory plan with prettier
e9979b4 
Assisted-by: Claude Opus 5.5 via Claude Code
Signed-off-by: Xe Iaso <xe@tigrisdata.com>
@Xe
Xe merged commit 2bd51ec into main Sep 25, 2026
8 checks passed
Xe pushed a commit that referenced this pull request Sep 25, 2026
@semantic-release-bot
chore(release): 1.13.1
1b1ad2f 
## [1.13.1](v1.13.0...v1.13.1) (2026-09-25)

### Performance Improvements

* **storage/tigris:** push any repository in bounded memory ([#25](#25)) ([2bd51ec](2bd51ec))

Signed-off-by: Tigris Data <social@tigrisdata.com>
@github-actions

github-actions Bot commented Sep 25, 2026

Copy link
Copy Markdown

🎉 This PR is included in version 1.13.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

@Xe Xe mentioned this pull request Oct 7, 2026
Open
4 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

released

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Xe