Repository navigation
feat: erofs snapshots of pushed trees, stored next to the repo - #15
Merged
Merged
Conversation
Adds the design spec, the implementation plan, and the upstream streaming-builder spec for github.com/Xe/erofs. Signed-off-by: Xe Iaso <xe@tigrisdata.com>
Signed-off-by: Xe Iaso <xe@tigrisdata.com>
Signed-off-by: Xe Iaso <xe@tigrisdata.com>
Signed-off-by: Xe Iaso <xe@tigrisdata.com>
Adds GetChecked for ids that are not digests, EvictIdle, and an event observer. Signed-off-by: Xe Iaso <xe@tigrisdata.com>
A hit is now a GetChecked call that returns a file without running a download. Budget evictions are reported after fill closes ready, so the observer never holds up a waiter. The fill failure logs use the same "id" key as the rest of the cache. Signed-off-by: Xe Iaso <xe@tigrisdata.com>
Storer implements snapshot.Store. OpenSnapshot downloads the whole image into the snapshot cache and refuses a body that does not match its erofs-sha256 metadata. Signed-off-by: Xe Iaso <xe@tigrisdata.com>
After a push, runSnapshots builds one image for the tree at each updated branch and tag tip, before hooks run. snapshotRefs now returns tags too, and runHooks keeps running for branches only. Signed-off-by: Xe Iaso <xe@tigrisdata.com>
Adds -erofs-snapshots, -snapshot-timeout, -snapshot-cache-bytes, and -snapshot-cache-max-idle. The snapshot cache lives next to the pack cache, and an errgroup goroutine sweeps idle images out of it. Signed-off-by: Xe Iaso <xe@tigrisdata.com>
Signed-off-by: Xe Iaso <xe@tigrisdata.com>
-erofs-snapshots now defaults to true. membench gains -daemon-erofs-snapshots, off by default, so snapshot cost is not counted as push cost. Signed-off-by: Xe Iaso <xe@tigrisdata.com>
Moves to github.com/Xe/erofs v0.8.0. Ensure now reads only blob sizes during the walk and hands the builder an open function through AddFileFunc, so a build's memory no longer grows with the tree. The two v0.6.1 workarounds (pre-sizing the temp file, adding the root by hand) are gone; v0.8.0 fixes both bugs, and the image bytes are unchanged. A symlink blob's size is now checked before it is read. Signed-off-by: Xe Iaso <xe@tigrisdata.com>
Signed-off-by: Xe Iaso <xe@tigrisdata.com>
…hots-plan Resolves conflicts with the webhook and ssh sh work: snapshots now hang off main's per-request onUpdated updates (filtered to branches and tags), and the startup orphan sweep is keyed on each cache's own directory prefix so the snapshot cache never deletes the live pack cache. Signed-off-by: Xe Iaso <xe@tigrisdata.com>
Xe
pushed a commit
that referenced
this pull request
Sep 23, 2026
# [1.9.0](v1.8.0...v1.9.0) (2026-09-23) ### Features * erofs snapshots of pushed trees, stored next to the repo ([#15](#15)) ([2a31f82](2a31f82)) Signed-off-by: Tigris Data <social@tigrisdata.com>
|
🎉 This PR is included in version 1.9.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
objgitdbuilds a Zstd-compressed erofs image of the tree at every updated branch and tag tip and stores it atsnapshots/erofs/v1/<tree>.erofsunder the repository prefix, on by default (-erofs-snapshots=falsedisables it).snapshot.Opendownloads an image into a local LRU cache next to the pack cache (a secondPackCache, with a byte budget and an idle sweep), for later readers such as a web UI.Details
Images are keyed by tree hash and built reproducibly, so a branch and a tag on one commit share one image. Builds stream blobs through erofs v0.8.0's
AddFileFunc, so memory stays flat (256 MiB of content builds in a 3.3 MiB live heap), and a failed snapshot never fails the push. Addsinternal/snapshot,Storeon*tigris.Storer,GetChecked/EvictIdle/an observer onPackCache, fiveobjgit_snapshot_*metrics, and the design spec, plan, and upstream erofs streaming spec.Test plan
go build ./...npm run format