Deleted-on-S3 objects served from cache forever — handleDownload misclassifies 404 as "S3 unreachable"

[Leonorus]

Summary

proxs3d serves a stale cached copy of an object that has been deleted from S3, indefinitely, even when S3 is fully reachable. A 404 (object no longer exists) is treated identically to a transport failure, so the daemon falls into its "S3 unreachable → serve stale cache" path instead of invalidating.

Root cause

internal/s3client/client.go — HeadObject wraps every error in a generic fmt.Errorf, so a 404 NotFound is indistinguishable from a network error:

out, err := c.s3.HeadObject(ctx, &s3.HeadObjectInput{...})
if err != nil {
    return nil, fmt.Errorf("heading object %s: %w", key, err)  // 404 looks like any other error
}

internal/api/api.go — handleDownload then treats any HeadObject error as "S3 unreachable":

info, err := client.HeadObject(ctx, key)
if err != nil {
    // S3 unreachable — serve stale cache rather than failing
    log.Printf("download: S3 unreachable for %s/%s, serving cached copy", storageID, key)
    writeJSON(w, map[string]string{"path": cached})
    return
}

Result: once an object is deleted on S3 (out of band, or via a cleanup path that doesn't go through /v1/delete), the cached copy is returned on every subsequent /v1/download and never purged. Object replacement (new ETag) is handled correctly by IsStale as long as S3 is reachable — only the deletion/404 case leaks.

Impact

This is especially dangerous for mutable, identity-bearing artifacts such as cloud-init user-data-<vmid>.iso payloads stored under template/iso/. In a Cluster-API / CAPMOX environment, a replacement VM that reuses a VMID can boot with a previous worker's nocloud identity because proxs3 keeps serving the deleted ISO from cache. (Bootstrap payloads also carry credentials/join tokens, so this is a secret-hygiene issue as well as availability.)

Expected behavior

When HeadObject reports the object no longer exists (404), the daemon should purge the cached file and fail closed (return 404) rather than serving stale bytes. Genuine transport errors should keep serving stale cache (graceful degradation is fine for immutable ISOs/templates).

Steps to reproduce

1. Download an object via /v1/download so it caches locally.
2. Delete that object directly from the S3 bucket (bypassing /v1/delete).
3. Call /v1/download for the same key again with S3 fully reachable.
4. Observed: 200 + stale cached path, logged as "S3 unreachable". Expected: 404 + cache file removed.

Proposed fix

Distinguish 404 from transport errors in HeadObject (sentinel ErrNotFound detecting types.NotFound/NoSuchKey, smithy APIError, and HTTP 404 for S3-compatible backends), and branch on it in handleDownload to cache.Remove(...) (which clears the chattr +i flag PVE sets on ISOs — Invalidate would silently fail) and return 404. Genuine transport errors retain the serve-stale fallback.

[davekempe]

Fixed in v0.7.0 via #13. handleDownload now invalidates the cache on a 404 instead of serving stale data.