fix(delete): treat podman cgroup race as pod-already-gone (issue #71)

[dndungu]

Summary

Fixes #71: DELETE /api/v1/pods/{name} (and the equivalent req.spark.delete NATS handler) returned a 500 and left a resurrectable phantom store record plus a leaked scheduler/GPU reservation whenever podman pod rm hit cgroup: Unit machine-libpod_pod_<id>.slice not loaded. This is a benign podman race, not a real failure: the pod's containers were already torn down by the preceding stop, and systemd/crun reaped the pod's cgroup slice before rm got to it, so the desired end state (pod gone) was already true — podman just reports the absence of the thing it's cleaning up as a hard error. An earlier occurrence of exactly this left a pod that resurrected across the next upgrade, because the persisted "Running" record was re-adopted at startup.

Changes

• internal/api/pods_mutate.go: added isCgroupCleanupRace(err) (matches the lowercased error text containing both cgroup and not loaded) and isPodAlreadyGone(err) (isNoSuchPod(err) || isCgroupCleanupRace(err)). Every DELETE-path check that used to gate on isNoSuchPod alone (StopPod's error, RemovePod's error, and the issue GPU reservation leaks: scheduler reports 0 GPUs free while the device is physically idle and no live pod holds it #81 podConfirmedGone status-check fallback) now gates on isPodAlreadyGone, so the cgroup race is classified exactly the same way "no such pod" already is: proceed with store/scheduler cleanup instead of aborting.
• internal/bus/handler_delete.go: the identical fix for the NATS handler, using its own private copy of the same helpers (this package already kept its own isNoSuchPod rather than sharing one across the import boundary, per the existing comment there — same pattern continued).
• Bounded retry: both files add removePodWithRetry (a fixed cap of 3 attempts, 20ms apart), retrying the RemovePod call specifically when the error is isCgroupCleanupRace — the issue's own repro showed an immediate manual retry of podman pod rm succeeding outright once the race window passed, so this gives podman a real chance to finish removing the pod rather than only papering over the classification. Retrying "no such pod" or any other error is skipped: it can't change the outcome and would only add latency.
• Left untouched, deliberately: the existing issue GPU reservation leaks: scheduler reports 0 GPUs free while the device is physically idle and no live pod holds it #81 podConfirmedGone/status-check fallback (it stays as the catch-all for genuinely unclassified remove errors — see TestDeletePodRemoveFails / TestDeleteHandler's remove_error case, both still passing), and internal/reconciler/reconciler.go's separate, pre-existing isNoSuchPod copy (different subsystem, out of scope for this issue).

Testing

• go build ./..., go vet ./..., staticcheck ./... — all clean.
• go test ./... -race -timeout 120s -count=1 — all 13 packages pass, no regressions.
• 4 new regression tests, confirmed genuinely red before this change and green after (not just trusted from a runner's exit code — each re-run individually with -v to see the actual --- PASS: line):
  • internal/api/pods_mutate_test.go: TestDeletePodCgroupRaceTreatedAsSuccess (every RemovePod attempt returns the cgroup-race error; retries exhaust; DELETE still returns deleted:true, store record and scheduler reservation gone) and TestDeletePodCgroupRaceRetrySucceeds (errors on the first RemovePod call, succeeds on the retried second call; asserts RemovePod was called exactly twice).
  • internal/bus/handler_delete_test.go: TestDeleteHandler_CgroupRaceTreatedAsSuccess and TestDeleteHandler_CgroupRaceRetrySucceeds, mirroring the two above for the NATS handler.
• Re-ran the pre-existing issue GPU reservation leaks: scheduler reports 0 GPUs free while the device is physically idle and no live pod holds it #81 regression tests specifically (TestDeletePodRemoveFails, TestDeleteHandler's remove_error and remove_error_but_pod_confirmed_gone subtests) to confirm the existing fallback for unrelated remove errors still returns an error/500 and wasn't weakened by the broader isPodAlreadyGone classification.
• Converged via a kazi goal (t4-1-issue71-delete-race, 5 predicates: cap-api-t41, cap-api-t42, cap-bus-t41, cap-bus-t42, guard-suite) — predicates confirmed red at t0 before any code existed, and all 5 pass now (cross-verified with kazi apply --check --json, not just the higher-level convergence report).
• Not run: a live DGX repro against real podman (this is a timing-race triggered by systemd/crun cgroup reaping order, not reliably reproducible on demand outside the fake-executor unit tests above).

Risk & rollback

Low risk: purely additive error classification plus a bounded, short retry — no changes to the existing "no such pod" handling or the issue #81 fallback's own logic. Revert is a straight git revert of the two commits if needed.

Linked issues / tasks

• Fixes DELETE fails (and leaves a resurrectable phantom record) when podman pod rm hits 'cgroup slice not loaded' #71 (DELETE fails and leaves a resurrectable phantom record when podman pod rm hits a cgroup-cleanup-race error).