Skip to content

Guard file uploads with remote object preconditions#119

Open
AysajanE wants to merge 3 commits into
8thpark:mainfrom
AysajanE:northset/m-1056
Open

Guard file uploads with remote object preconditions#119
AysajanE wants to merge 3 commits into
8thpark:mainfrom
AysajanE:northset/m-1056

Conversation

@AysajanE

@AysajanE AysajanE commented Jul 21, 2026

Copy link
Copy Markdown

Summary

  • use ifMatch for existing remote files after verifying their manifest hash and ETag
  • use ifAbsent for new files and conflict copies
  • safely recognize identical orphaned uploads during retries
  • stop stale plans before they can upload a manifest
  • add deterministic coverage for stale outcomes and identical-orphan retries

Tests

  • npm test
  • npm run lint
  • npm run build

Limitations

  • The integration suite requires a live MinIO service and was not run in the network-off verifier.
  • Remote delete races are outside issue 110 and this patch.

Verification

npm test && npm run lint && npm run build exited 0 on this exact head (583e6c3) in a network-off container.
No workflow or CI files are modified in this change.
Commands, environment, and hashes: receipt M-1056 — checkable in ~30 seconds without trusting us.
Self-run by the contributor, not maintainer verification.

AI-assisted and reviewed by Northset; I take responsibility for this submission.

Greptile Summary

This PR guards file uploads against concurrent remote writes by computing a conditional PUT header (ifMatch for existing objects, ifAbsent for new ones) before every push, and stops a pass from uploading a stale manifest as soon as any file precondition fails.

  • putCondition does a pre-flight GET to obtain the current ETag; if the remote content already matches the local bytes (orphaned upload from a previous pass) it returns done and skips the PUT entirely, making retries cheaper.
  • remoteMatches is called on a 412 from a conditional PUT to distinguish a genuine concurrent writer from an idempotent re-upload, keeping the #87 progress-recording behaviour intact for server errors while still flagging real races.
  • executed.concurrent gate in syncOnce aborts the manifest upload immediately on detection, closing the window where a losing pass could upload a manifest that disagrees with the winning pass's file object (#110).

Confidence Score: 5/5

Safe to merge — the new conditional PUT logic is correctly anchored to the manifest snapshot, the manifest CAS remains the final safety net, and all three changed files have direct test coverage for the new paths.

The putCondition ETag-fetch chain is correctly sequenced. The done path and remoteMatches idempotency check are each exercised end-to-end by the new tests. The concurrent flag propagates correctly through executeAction and executeSyncPlan and gates the manifest upload in syncOnce. The #84 test update is essential and correct. No logical gaps were found in the failure classification or loop-break behaviour.

No files require special attention.

Important Files Changed

Filename Overview
src/sync/execute.ts Core logic: adds putCondition (ETag-based pre-flight GET for existing files, ifAbsent for new ones) and remoteMatches (idempotency check after a 412); concurrent flag propagation and early loop exit are correct; all failure classifications look sound.
src/sync/sync.ts Threads the remote snapshot into executeSyncPlan and adds the early-return gate that prevents a manifest upload when any file precondition already detected a concurrent writer; change is minimal and correct.
src/sync/sync.test.ts Two new end-to-end tests cover the orphaned-upload retry (done path in putCondition) and the #110 regression; the #84 test is correctly updated to supply real content hashes and the remote file object so putCondition can do its ETag fetch.

Reviews (3): Last reviewed commit: "test: cover sync retry outcomes" | Re-trigger Greptile

AysajanE added 2 commits July 21, 2026 18:46
Signed-off-by: Aysajan Eziz <aeziz@northset.ai>
Signed-off-by: Aysajan Eziz <aeziz@northset.ai>
Comment thread src/sync/sync.test.ts
Comment thread src/sync/execute.ts
Comment on lines +334 to +343
if (remoteHash !== expected.hash) {
if (remoteHash === (await hashBytes(bytes))) {
return { ok: true, kind: "done" };
}
return {
ok: false,
concurrent: true,
failure: { path, message: REMOTE_DRIFT_MESSAGE },
};
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 kind:"done" path has no test coverage

The branch where remoteHash !== expected.hash but remoteHash === hashBytes(bytes) — meaning the remote already holds our bytes from a previous orphaned upload — returns { ok: true, kind: "done" } and marks the action as completed. This is the "safely recognize identical orphaned uploads during retries" feature called out in the PR summary. It is not exercised by any test: the new #110 test only exercises the remoteMatches fallback after a failed conditional PUT, not this pre-PUT short-circuit. A targeted test (second-pass retry where the prior pass uploaded the file but lost the manifest CAS) would pin this behaviour against future refactors.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AysajanE Could you address this comment please?

Signed-off-by: Aysajan Eziz <aeziz@northset.ai>
@revett

revett commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

@AysajanE Thanks for the contribution 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants