[luthermonson]

Fixes #193 (failed to solve: lease "4igb5uptxddpdjw9lwatd9psk": not found, intermittent, rerun passes).

Root cause — and the issue's hypothesis disproven

The failing ID is a 25-char base36 identity.NewID — the ID space of exactly one lease kind: a BuildKit cache record's own lease (leases.WithID(recordID) + gc.flat, no expiry). The raw unwrapped error comes from containerd's AddResource on a missing lease bucket, reached from the solve path's cached-FROM handling (source/containerimage/pull.go) — matching a trivial busybox build failing instantly on a warm cache.

No ephemerd sweep can cause this. Exhaustively enumerated: the only lease deletions in the repo are CleanupJobNamespace (per-job dind namespaces only) and the cache mirror-lease release. PruneDeadBuildRecords, CleanupJobBuildRecords, imagegc.Collect, SweepBrokenImageChains, and CachePrune delete image records — an image delete cannot remove a lease, and containerd's own GC only deletes expired leases (record leases never carry gc.expire). #187's reaper and #190 are likewise exonerated (runtime namespace only).

What actually happened: a metadata↔lease desync — BuildKit's bbolt store kept a cache record whose containerd lease is gone — the same "two databases describe one store" corruption class as #149's dangling snapshot (heal.go already documents the creators: hard kill between the two stores' writes, ENOSPC during a prune's metadata clear — this fleet's signature failure — or out-of-band lease removal). The desync sits latent until a build cache-hits the record. The rerun passing is the tell: the failed solve's released refs let BuildKit's own GC collect the stale record (record removal tolerates the missing lease), healing the store between runs.

Fix

1. Recognize + repair + retry (the detonation, not one creator): a deliberately narrow matcher — quoted 25-char base36 record-lease ID (optional -view/-variants) + containerd's exact phrasing, so BuildKit's temporary <nanos>-<base64> leases can never trigger it — feeds the dangling-lease signature into the existing fix(buildkit): repair the shared build store instead of failing every job (#149) #150 Healer ladder (prune → rebuild → give-up) with a single retry. The prune rung provably clears the broken state; the incident class becomes one healed solve instead of a red check.
2. Close the adjacent real TOCTOU (the shape the issue hypothesized, found live in DeadBuildRecords): the live-job set is captured before the record listing, so a job starting between the two listings could have its fresh export judged dead. Records younger than a 15-minute grace — or with unknown timestamps, failing safe — are never selected; genuinely dead jobs' records still are, one tick later. Same principle as ReapDeadContainers: never collect a resource whose owning operation could still be live.

Diagnostics

Post-fix, the incident logs build failed on a lease the shared store references but containerd does not have; repairing the shared store → shared build store repaired; retrying the build once with the lease ID in the heal report. On the original node, look for an earlier metadata rebuild, failed prune, or unclean restart adjacent to image-GC activity to date the desync's origin.

Testing

heal_test.go pins the matcher (record IDs match; temporary-lease shapes, other not-founds, and near-misses don't) and the ladder integration; buildkit_cleanup_test.go pins the grace predicate (in-flight/young/unknown-age never selected; dead-past-grace still swept). go vet ./pkg/... clean; dind/buildkit/imagegc/cacheprune/runtime suites green.

[luthermonson]

Adversarial review completed (diagnosis re-verified against vendored containerd v2.2.2 + buildkit v0.25.1 sources — the record-lease ID space, the exact error format, the warm-cache detonation sites, and the prune rung's tolerance of a missing lease all CONFIRMED; the grace predicate's timestamp threading verified populated at both callers, so the fail-safe cannot become fail-dead). Verdict was safe-to-merge with two pre-existing-exposure MAJORs worth closing here rather than in a follow-up — both fixed in f65642a:

• Job-influenced error text could reach the repair ladder. The matchers ran unanchored over the whole error string, and both a hostile registry's HTTP reason phrase and a RUN command's echoed output land in it (the reviewer confirmed matching strings empirically). Repair = shared-cache PruneAll, escalating to a core-stopping Rebuild — a cross-job DoS an attacker-controlled fork PR could drive, which matters with the fork-PR flip on the roadmap. Both regexes are now tail-anchored: genuine containerd not-founds are always the wrap chain's suffix, job-influenced text always gets further wrapping appended. All genuine-signature tests pass unchanged; the rejection set (registry phrase, wrapped RUN output, ref_ history leases, 26/24-char ids, temporary-lease shape) is pinned.
• A canceled ctx could escalate to a Rebuild that bricks the solver. A runner disconnect mid-build fails PruneAll with context.Canceled, which the ladder read as store inconsistency → Rebuild under a dead ctx; a failed Rebuild leaves the shared core stopped (build-dead node until restart, plus a latent double-close on shutdown). applyHealAction now refuses to repair or escalate under a canceled ctx — a genuine desync simply re-fires from the next live build.

Deferred to the 0.2.10 hardening PR (pre-existing, review-identified): Rebuild's failure path leaves s.core pointing at the closed core instead of restoring/nil'ing it, and Server.Close can then double-close it.