Repository navigation
docker shim: intermittent 'failed to solve: lease <id>: not found' on the unprivileged build path #193
Description
Seen on ephpm/ephpm PR CI (job 100114781645, 2026-09-02): the 'Docker build (unprivileged)' check — which builds a trivial busybox image (RUN echo unprivileged-build-ok) through the ephemerd docker shim — failed with:
failed to solve: lease "4igb5uptxddpdjw9lwatd9psk": not found
A plain rerun passed, so it's intermittent. The build itself is trivial and version-independent, so this is shim/containerd-side, not workload-related: a lease is created for the build and is gone by the time the solver references it.
Worth checking whether any ephemerd sweep can remove a lease belonging to an in-flight build — the orphan/snapshot sweeps (#187's reaper, SweepOrphans, and startup CleanOrphans per #188) all operate on containerd resources, and a lease vanishing mid-solve looks like exactly that class of race. If a sweep is the cause, the fix is the same shape as #187: never collect a resource whose owning operation is live.
Activity
luthermonson commentedon Sep 2, 2026
Root-caused — and the sweep hypothesis is disproven: no ephemerd sweep deletes leases at all (they delete image records; the only lease deletions in the tree are per-job dind namespaces and cache mirror-leases). #187/#190 exonerated. The failing ID is a BuildKit cache-record lease, and the mechanism is a latent BuildKit-metadata↔containerd-lease desync (the #149 corruption class) detonating on a warm-cache FROM — the rerun passes because the failed solve lets BuildKit's own GC collect the stale record, healing the store. Fix in the linked PR: the signature now feeds the existing heal ladder (prune → retry once), and a real 15-min-grace TOCTOU found in the dead-record sweep while chasing this is closed too.
luthermonson commentedon Sep 2, 2026
Second occurrence, ~2 hours after the first: ephpm/ephpm PR #446, job 100122419716 (runner
ephemerd-github-ephpm-warm_faraday, 2026-09-02T05:21Z) — sameDocker build (unprivileged)step, same trivial busybox build, failed again. First was job 100114781645 on PR #443.Two hits in one evening on a build that does nothing but
RUN echo unprivileged-build-okmakes this a recurring flake rather than a one-off, and it costs a full PR-check cycle each time (the job is fast, but the rerun re-queues behind the whole fleet backlog).Both times a plain rerun passed. Still points at lease lifetime under the shim rather than anything workload-related — see the original report for the sweep-race hypothesis.
luthermonson commentedon Sep 2, 2026
Third occurrence tonight — ephpm/ephpm PR #448, runner
ephemerd-github-ephpm-clever_bell, 2026-09-02. Same step, same trivial busybox build.Tally: #443 (job 100114781645,
serene_ride), #446 (job 100122419716,warm_faraday), #448 (clever_bell) — three different runners, three different PRs, one evening. Every one passed on rerun.This is now the single most frequent CI failure in the org and it blocks every PR merge behind a rerun cycle. Given three distinct runners, it is not a bad node; it is the shim's lease handling. Raising priority — suggest this jump ahead of the other ephemerd backlog items.