Skip to content

fix: invalidate cache when S3 object deleted (404), don't serve stale#13

Merged
davekempe merged 1 commit into
sol1:mainfrom
Leonorus:404_fix
Jun 18, 2026
Merged

fix: invalidate cache when S3 object deleted (404), don't serve stale#13
davekempe merged 1 commit into
sol1:mainfrom
Leonorus:404_fix

Conversation

@Leonorus

@Leonorus Leonorus commented Jun 16, 2026

Copy link
Copy Markdown

Issue #12

Problem

proxs3d served a stale cached copy of an object that had been deleted from S3, indefinitely, even when S3 was fully reachable.

HeadObject (internal/s3client/client.go) wrapped every error in a generic fmt.Errorf, so a 404 NotFound was indistinguishable from a transport failure. handleDownload (internal/api/api.go) then treated any HeadObject error as "S3 unreachable" and fell into its serve-stale-cache fallback:

info, err := client.HeadObject(ctx, key)
if err != nil {
    // S3 unreachable — serve stale cache rather than failing
    writeJSON(w, map[string]string{"path": cached})
    return
}

So once an object was deleted on S3, the cached copy was returned on every subsequent /v1/download and never purged. (Object replacement with a new ETag was already handled by IsStale while S3 is reachable — only the deletion/404 path leaked.)

This is the proxs3-side amplifier behind stale cloud-init user-data-<vmid>.iso payloads: a replacement VM reusing a VMID could boot with a previous worker's nocloud identity. It's both an availability and a secret-hygiene problem (bootstrap ISOs carry join tokens / credentials).

Fix

  • internal/s3client/client.go — add ErrNotFound sentinel and isNotFound() (detects types.NotFound/NoSuchKey, smithy APIError codes, and HTTP 404 for S3-compatible backends like MinIO). HeadObject wraps 404 responses with ErrNotFound.
  • internal/api/api.gohandleDownload branches on errors.Is(err, s3client.ErrNotFound): purge the cache and return 404, failing closed instead of serving stale identity-bearing bytes. Uses cache.Remove (not Invalidate) so the chattr +i flag PVE sets on ISOs is cleared and the file is actually deleted. Genuine transport errors keep serving stale cache (graceful degradation preserved for immutable ISOs/templates).
  • internal/api/handler_test.goTestHandleDownload_ObjectDeletedOnS3_RemovesCache (cached entry + 404 → 404 response + cache purged); existing TestHandleDownload_S3Unreachable_ServeStaleCache kept green.

Behavior change

Scenario Before After
Object deleted on S3, cached locally 200 + stale path ("S3 unreachable") 404 + cache removed
S3 network failure, cached locally 200 + stale path 200 + stale path (unchanged)
Fresh / replaced object re-download as needed unchanged

Verification

  • go build ./..., go vet ./... — clean
  • go test ./... — all pass except the pre-existing TestFileInUse_OpenFile (darwin-only fileInUse detection, fails on a clean tree too — unrelated)

handleDownload treated every HeadObject error as "S3 unreachable" and
served the stale cached copy. A deleted object returns a 404, which was
misclassified, so deleted-on-S3 objects were served from cache forever
even when S3 was reachable (CAPI BUG-02 stale cloud-init user-data ISO).

Add s3client.ErrNotFound sentinel; HeadObject now wraps 404 responses
(types.NotFound/NoSuchKey, smithy APIError, or HTTP 404) with it.
handleDownload branches on it: purge the cache via Remove (clears the
immutable flag PVE sets on ISOs) and return 404, failing closed instead
of serving stale identity-bearing bytes. Genuine transport errors still
serve stale cache for graceful degradation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants