fix(executor): bound stop/delete's wait against a wedged podman

[dndungu]

Summary

DELETE /api/v1/pods/{name} on a GPU-attached pod hung for ~7 minutes in live verification (self-resolved, not yet root-caused) -- podman pod stop was retried three times, a podman child parented by spark's own PID became a zombie (exited but never reaped), and an unrelated, unfiltered sudo podman pod ps hung host-wide for the same window (fixes #88: "podman pod stop/rm on a GPU-attached pod hangs indefinitely, and the stuck child becomes a zombie under the spark PID"). This bounds StopPod/RemovePod's own wait so a wedged podman invocation can no longer block the calling goroutine (and the HTTP request) indefinitely.

Root cause

Two gaps stacked:

1. No deadline of its own. The DELETE handler (internal/api/pods_mutate.go) passes r.Context() straight into StopPod/RemovePod. That context carries no deadline -- it only ends if the client disconnects. A genuinely wedged podman invocation (blocked inside podman's own storage/CDI locking) has nothing to time it out.
2. A context deadline alone isn't enough, even when one exists. exec.CommandContext's cancellation only signals the direct child. If that child forks a subprocess (podman's own conmon/netavark helpers) that inherits the stdout/stderr pipe and is itself stuck on a lock, Wait()/CombinedOutput() keeps reading for EOF that never arrives -- even after the direct child has already exited to a zombie. That's exactly what ps showed on the DGX during the incident: [podman] <defunct> parented by spark's own PID, while the CombinedOutput() call never returned. This is a documented os/exec gotcha; the stdlib's own fix for it is Cmd.WaitDelay (Go 1.20+), which forcibly closes the pipes after a bound instead of waiting for EOF forever (the default when WaitDelay is left unset, as every call site here did).

Changes

• internal/executor/podman.go: added runPodmanBounded, wrapping a podman invocation with (a) a 20s timeout layered on top of the caller's context (never loosens an earlier deadline -- only adds one where none existed) and (b) a 5s WaitDelay so Wait() can't be held open forever by an orphaned grandchild. StopPod and RemovePod now go through it instead of calling exec.CommandContext(...).CombinedOutput() directly. Scoped to the stop/delete path per the issue's own suggested fix -- PodStatus/ContainerStatuses/ListPods etc. are unchanged.
• internal/executor/podman_wait_test.go (new): real-subprocess tests (no mocks, no podman/DGX dependency) reproducing the exact mechanism via a fake podman script on PATH that backgrounds a long-lived grandchild inheriting its pipe, then exits immediately -- the same shape as podman forking a stuck helper.

Testing

• go build ./..., go vet ./..., staticcheck ./... -- all clean.
• go test ./... -race -timeout 120s -count=1 -- all packages pass, run twice back to back to check for flakiness (the executor package's new tests use real timing; the first draft had a race under parallel -race load and was fixed by giving the pipe/EOF-only sub-test a generous timeout so scheduling jitter can't kill the test's own fixture prematurely).
• New tests, all passing:
  • TestRunPodmanBounded_OrphanedGrandchildHangsWithoutWaitDelay / TestRunPodmanBounded_WaitDelayBoundsTheHang: isolate the pipe/EOF mechanism itself (waitDelay=0 blocks for the grandchild's full sleep; waitDelay=300ms bounds it).
  • TestStopPod_BoundedDespiteOrphanedGrandchild / TestRemovePod_BoundedDespiteOrphanedGrandchild: exercise the real production methods end-to-end via a fake podman on PATH, using the actual podmanStopTimeout/podmanWaitDelay constants.
• Manual red-check (not left in the repo, done to confirm the tests aren't vacuous): reverted podman.go to its pre-fix state and ran StopPod against the same wedged fake podman via a throwaway go run. It took 30.1s, tracking the fake grandchild's sleep exactly, confirming the gap is real in the production code path and that the new tests would have failed against it.
• Not verified: real podman/DGX behavior. Per the task's stop boundary this fix does not touch the DGX -- see the live-verification recipe below for the coordinator to run separately.

Live-verification recipe (for the coordinator -- HIGH RISK, do carefully)

This reproduces issue #88's own trigger conditions on aitopatom-bfc8. Watch host-wide podman pod ps responsiveness throughout; if it stalls, that's expected per the issue (podman's own lock contention, not something this fix can prevent) -- what changed is that Spark's own DELETE call now can't be held open by it.

1. Submit a single throwaway GPU-attached pod (mirroring the issue's own repro): nvidia.com/gpu: 1, --device nvidia.com/gpu=all via CDI, image python:3.12-slim, command sleep 300.
2. Wait 10-20s after it's confirmed running (matches the original timing).
3. DELETE /api/v1/pods/{name} and time the response.
4. Expected: the request now completes (success or a clear timeout-derived error) within roughly 50s worst case (2 * (podmanStopTimeout + podmanWaitDelay) for stop then rm), not an indefinite hang. In the common case it should just succeed quickly, same as any other DELETE.
5. Throughout, in a second shell: timeout 10 sudo podman pod ps a few times. If podman's own storage lock is genuinely contended during teardown, this may still stall during that window (that's podman's own locking, outside Spark's control) -- but it should not stay stuck for minutes, and Spark's own DELETE call must not still be blocked once this resolves.
6. Confirm via journalctl -u spark that at most one retry sequence of podman pod stop/pod rm appears (not the original three), and that ps -eo pid,ppid,stat,etime,cmd shows no lingering zombie [podman] <defunct> parented by spark's PID after the DELETE call returns.

Risk & rollback

Low blast radius: the change only affects the stop/delete path's own internal timeout/WaitDelay handling, not podman's command arguments or lifecycle semantics otherwise. Worst case if the bounds are too tight for a legitimately slow GPU teardown: a DELETE that would have eventually succeeded on its own instead returns an error after ~25s per call (~50s total for stop+rm), which is still a large improvement over an unbounded hang and is a normal, retryable failure mode rather than a stuck goroutine. Revert is a straight git revert of this commit if the live-verification recipe above surfaces a regression.

Linked issues

Fixes #88 ("podman pod stop/rm on a GPU-attached pod hangs indefinitely, and the stuck child becomes a zombie under the spark PID") -- tracked as T2.9 in docs/plan.md's E2 epic.