Repository navigation
Conversation
Retain upload failures across ref updates and log the upload barrier before hooks. Recover stale pack indexes with one coalesced refresh while preserving known-object reads. Exercise the real S3 SDK and Tigris storer over HTTP and SSH with gated and failed uploads. End HTTP shallow-discovery responses at their flush to avoid an extra NAK. Add personal-cluster validation instructions. Refs: #28 Assisted-by: GPT-6 via Codex Signed-off-by: Xe Iaso <xe@tigrisdata.com>
Let long-lived readers retry pack discovery after missing probes and storage errors. Coalesce concurrent refreshes and wait one second after each attempt before allowing another scan. Keep known objects readable. Forward HTTP upload request packets without rebuilding or sorting the wants list. Release the pooled packet buffer after negotiation. Add recovery, cancellation, framing, and allocation benchmarks. Refs: #28 Assisted-by: GPT-6 via Codex Signed-off-by: Xe Iaso <xe@tigrisdata.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Concurrent storage refresh and wire-protocol changes warrant human review, with live-bucket and cluster validation still pending.
0 open findings
What changed in this PR
Hardens ObjGit’s pack publication and reader recovery for issue #28 without claiming to resolve the unconfirmed production cause.
Changes:
- Keeps upload failures fatal and logs the publication barrier before hooks.
- Adds coalesced pack-index refreshes and corrects HTTP shallow-response framing.
- Expands regression coverage and documents operator validation.
| File | Description |
|---|---|
internal/storage/tigris/visibility_test.go |
Tests refresh recovery, concurrency, and cancellation. |
internal/storage/tigris/upload.go |
Preserves the first upload failure. |
internal/storage/tigris/tigris.go |
Supports configured S3 clients. |
internal/storage/tigris/publication_test.go |
Tests upload ordering and failed publication. |
internal/storage/tigris/packindex.go |
Adds coalesced refreshes with cooldowns. |
internal/storage/tigris/objects.go |
Refreshes packs after object misses. |
internal/storage/tigris/livebucket_test.go |
Verifies published objects through fresh readers. |
internal/storage/tigris/client_test.go |
Adds upload gates to the storage fixture. |
docs/usage/pack-publication-check.md |
Provides cluster validation steps. |
docs/plans/2026-10-15-issue-28-pack-visibility.md |
Records investigation, implementation, and validation. |
docs/architecture/transports.md |
Explains HTTP shallow-response handling. |
docs/architecture/tigris-storer.md |
Documents upload failures and refresh behavior. |
docs/architecture/hooks.md |
Clarifies publication-before-hook ordering. |
cmd/objgitd/uploadpack.go |
Adds bounded HTTP shallow-round handling. |
cmd/objgitd/uploadpack_test.go |
Tests packet forwarding and response framing. |
cmd/objgitd/ssh_test.go |
Checks wait-log ordering before SSH hooks. |
cmd/objgitd/s3_publication_test.go |
Exercises publication and fetches through HTTP/SSH. |
cmd/objgitd/receivepack.go |
Logs before the publication barrier. |
cmd/objgitd/publication_test.go |
Tests publication barriers and callback behavior. |
cmd/objgitd/http.go |
Routes fetches through the HTTP adapter. |
cmd/objgitd/http_test.go |
Checks wait-log ordering before HTTP hooks. |
cmd/objgitd/hooks_test.go |
Adds a shared log-order assertion. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A failed pack upload could lose its recorded error on a later flush, allowing a subsequent reference update to publish missing objects. Keep the first upload error fatal for that writer, and log
waiting for packfiles to finish pushingbefore the existing publication barrier and hooks.Readers can now discover packs published after their initial listing. Concurrent misses share a refresh, and a one-second cooldown after each attempt permits recovery from earlier misses or transient errors without a listing for every miss.
Refs #28. The production failure cause remains unconfirmed, so this PR does not close the issue.
Details
docs/usage/pack-publication-check.md.In a local 10,000-want benchmark, HTTP adapter overhead fell from about 1.55 ms and 3.66 MB of allocations to 0.25 ms and 4.3 KB. This measures adapter overhead, not total fetch throughput.
Test plan
go build ./...go test ./...go test -race ./internal/storage/tigrisThe earlier broad daemon race run reported a race in the unchanged
TestSmartHTTPHealsDanglingHEADmemory fixture. Focused race checks pass. Bucket consistency requirements remain unchanged.