fix: avoid stale refills after lost DeleteObjects replies

.claude/rules/distributed-cache.md

@@ -40,17 +40,29 @@ retry the same bytes with a fresh token.
 Meta invalidation/populate ordering is a strategy selected once at construction
 (`cache/coordinator.go`), never per-call if/else:
 
-- **Legacy (default)**: `cache.legacy_coordination: true` — the pre-v1.21
-  tombstone mechanism (plain Get/Put/Delete + timestamp tombstones). Safe for
-  any cluster mix, including nodes upgrading straight from ≤v1.20.
-- **CAS**: `legacy_coordination: false` — the fenced CAS pattern above.
-  Requires every node in the cluster to be CAS-capable (≥v1.21) BEFORE the
-  flip; flip via config + a brisk rolling restart (mixed-mechanism window is
-  bounded by the restart).
-
-The proxy layer is mode-blind: it carries an opaque uint64 token (store version
-in CAS mode, wall-clock stamp in legacy mode) from decision to commit. Never
-branch on the mode outside the coordinator implementations.
+- **Legacy (default)**: `cache.legacy_coordination: true` keeps the v1.20
+  metadata key and eight-byte tombstone protocol for wire-compatible rollouts;
+  it is not a fleet-wide freshness guarantee while v1.20 readers remain.
+  Current TAG also CAS-updates a separate `meta-gen|...` sidecar and records
+  its generation in new metadata; current reads require a match, so old rows
+  without a generation are misses until repopulated. A decision read creates a
+  TTL-bounded publish marker when none exists; eviction before commit makes the
+  captured token fail closed. If a cache owner lacks the sidecar CAS RPC,
+  current TAG fails closed and bypasses that cache entry. Older TAG readers
+  ignore the sidecar and can still serve a late plain metadata Put during a
+  mixed-version rollout. The v1.20 reader reads only the plain metadata key, and
+  its tombstone check precedes a separate plain Put; a writer that passed the
+  check can publish after a later delete. This old check-to-Put limitation is
+  outside the sidecar-aware current-reader guarantee.
+- **CAS**: `legacy_coordination: false` — the fenced CAS pattern above on the
+  metadata key. Requires every node in the cluster to be CAS-capable (≥v1.21)
+  BEFORE the flip; flip via config + a brisk rolling restart (mixed-mechanism
+  window is bounded by the restart).
+
+The proxy layer is mode-blind: it carries an opaque comparable token from
+decision to commit. CAS mode uses the metadata-key version; legacy mode carries
+the generation-sidecar observation and v1.20 tombstone timestamp. Never branch
+on the mode outside the coordinator implementations.
 
 ## Stream Multiplexing > Batching
 

CLAUDE.md

@@ -57,7 +57,7 @@ All dependencies are public, including the Tigris `ocache` modules (`github.com/
 - **Optional Clustering**: Memberlist gossip for discovery, gRPC for cache key routing between nodes
 - **Broadcast/Subscriber Pattern**: Request coalescing via `proxy/broadcast/` - streams chunks to multiple listeners simultaneously
 - **Two-Key Cache Storage**: Objects stored as `meta|bucket|key` (headers/ETag) and `body|bucket|key` (raw bytes)
-- **Meta Coordination Strategy**: Invalidation/populate ordering is a construction-time strategy (`cache/coordinator.go`, `cache.legacy_coordination`). Default is the legacy tombstone mechanism (safe for any cluster mix, including direct ≤v1.20 upgrades); flipping the flag off selects fenced CAS ops (ocache v1.13.0, ocache#267) where every populate commits under a decision-time version token and writes that observed pre-delete state lose atomically. The proxy layer is mode-blind — it carries an opaque token, never branches on the mode
+- **Meta Coordination Strategy**: Invalidation/populate ordering is selected once at construction (`cache/coordinator.go`). The default legacy path keeps the v1.20 tombstone and plain metadata-key protocol for wire-compatible rollouts, not a fleet-wide freshness guarantee while v1.20 readers remain; current TAG tags new metadata with a separate CAS generation sidecar and validates it on reads. Rows without a generation are misses until repopulated. A decision read creates a TTL-bounded publish marker when none exists; if disk-cap eviction removes it before commit, that token refuses the late write. Older readers ignore the sidecar and can still serve a late plain Put during a mixed-version rollout. The v1.20 reader uses only the plain metadata key, and its tombstone check precedes a separate plain Put; a writer that passed the check can publish after a later delete. This check-to-Put limit is outside the sidecar-aware current-reader guarantee. If a cache owner lacks the sidecar CAS RPC, current TAG fails closed and bypasses that entry. Setting `cache.legacy_coordination: false` selects fenced CAS operations on the metadata key (ocache v1.13.0, ocache#267); enable it only after every node supports CAS. The proxy layer stays mode-blind and carries the coordinator's opaque decision token.
 - **Ingress Admission**: `server.max_inflight_requests` (default 1024) bounds concurrently-served S3 requests; excess is shed with 503 SlowDown. Operational endpoints (`/health`, `/metrics`, `/debug/pprof/*`) are exempt. `0`/unset = default, negative = disabled.
 - **Semaphore-Gated Cache Writes**: `cache.max_concurrent_writes` (default 256) bounds concurrent cache-populate operations; when saturated, objects are served from upstream without being cached. `0`/unset = default, negative = disabled.
 

README.md

@@ -102,8 +102,9 @@ When configuring S3 clients, ensure path-style addressing is enabled. See [docs/
 - Among cacheable objects (within `size_threshold`), those at or above `block_size` are cached as fixed-size blocks (block caching, on by default), so a range read fetches only the covering blocks; smaller objects are whole-cached
 - Set `block_caching_enabled: false` to cache whole objects instead — a range request then triggers a background fetch of the full object (if within threshold)
 - PUT/DELETE operations invalidate the cache entry
+- In proxy modes, a lost DeleteObjects reply triggers a second invalidation for each requested key; a cacheable 200 revalidation updates the cache only after a HEAD confirms the same strong ETag and length
 
-See [docs/cache-control.md](docs/cache-control.md) for detailed cache control and revalidation documentation. For tuning and sizing block caching in production, see [docs/deploy.md](docs/deploy.md#block-aligned-caching).
+See [docs/cache-control.md](docs/cache-control.md) for detailed cache control, revalidation, and mixed-version cache behavior. For tuning and sizing block caching in production, see [docs/deploy.md](docs/deploy.md#block-aligned-caching).
 
 ## Configuration
 

cache/block_cache_test.go

@@ -385,7 +385,7 @@ func TestPutMetaIfVersion(t *testing.T) {
 	meta := &CachedObjectMeta{Bucket: bucket, Key: key, ETag: `"v1"`, ContentLength: 100, StatusCode: 200, BlockSize: 4}
 
 	// No tombstone → meta is written.
-	wrote, err := c.PutMetaIfVersion(ctx, bucket, key, meta, 60, VersionAny)
+	wrote, err := c.PutMetaIfVersion(ctx, bucket, key, meta, 60, AnyMetaVersionToken())
 	if err != nil || !wrote {
 		t.Fatalf("PutMetaIfVersion = (%v, %v), want (true, nil)", wrote, err)
 	}