Pod completes instantly (exitCode=0) with zero container startup — container spec silently dropped

[dndungu]

Summary

POST /api/v1/pods returns success and the pod transitions pending -> running -> completed (exitCode=0) within ~5 seconds, but the actual container defined in spec.containers[] is never created or started. Only the pod's infra/pause container exists. podman logs <pod>-<container> immediately after fails with "no such container" — because it was never created, not because it exited and was cleaned up.

Evidence (from spark's own journald output, --root /var/lib/containers/storage)

INFO pod scheduled pod=sire-dgx-runner-v2
INFO creating pod cmd=podman args="[pod create --name sire-dgx-runner-v2 --network spark-net]"
INFO pod running pod=sire-dgx-runner-v2
INFO started log streaming pod=sire-dgx-runner-v2
INFO log stream goroutine exited pod=sire-dgx-runner-v2
INFO log streaming ended pod=sire-dgx-runner-v2
INFO inspecting pod cmd=podman args="[pod inspect sire-dgx-runner-v2 --format {{.State}}]"
INFO pod exited pod=sire-dgx-runner-v2 exitCode=0
INFO pod completed pod=sire-dgx-runner-v2

Notice there is no podman run/podman container create invocation anywhere in this sequence for the actual workload container — only podman pod create (the pod shell). The pod has nothing running inside it, so it "completes" immediately.

Manifest that triggered this

apiVersion: v1
kind: Pod
metadata:
  name: sire-dgx-runner-v2
spec:
  restartPolicy: Always
  containers:
  - name: runner
    image: ghcr.io/actions/actions-runner:latest
    command: ["/bin/bash", "-c"]
    args: ["./config.sh --url https://github.com/... --token *** --name sire-dgx-runner --labels self-hosted,linux,arm64,dgx --unattended --replace && ./run.sh"]
    resources:
      limits:
        cpu: "2"
        memory: 4Gi

Submitted as YAML with Content-Type: application/yaml.

What's confusing / notable

• The identical shape of manifest (single container, command + args both set) worked correctly earlier in the same session — the container started, registered as a GitHub Actions runner, and processed at least one job before an unrelated network blip took it down. This suggests an intermittent failure in container creation, not a hard manifest-parsing bug.
• A minimal single-container pod with only command (no separate args) -- image: docker.io/library/alpine:latest, command: ["sleep", "60"] -- worked fine and stayed running for the full duration, in the same session, same host, similar resource request.
• A 2-container pod (runner + docker.io/postgres:17-alpine sidecar) hit the exact same "pod exits immediately, zero container startup" symptom.
• Two separate podman storage roots exist on this host (spark's own service runs podman --root /var/lib/containers/storage; an interactive user's rootless podman session is a completely separate storage/namespace) -- this cost me significant debugging time before I found the right one to inspect. Worth documenting prominently in spark's own docs/README for anyone SSHing in to debug a pod, since podman ps/podman images run as a normal user shows nothing relevant.
• GET /api/v1/pods/{name}/logs and GET /api/v1/pods/{name}/events both returned empty/minimal output for these failed pods, making the API itself unhelpful for diagnosing this -- had to go to journald.

Repro rate

Reproduced 3 times in a row (two single-container runner attempts under different pod names, one 2-container runner+postgres attempt), after one earlier successful run of the same shape. Not yet root-caused on the spark side, but consistently pod-create-succeeds / container-create-never-happens / pod reports completed exitCode=0 with no real workload having run.

Environment

• Host: aitopatom-bfc8 (the DGX Spark host referenced in the /spark skill doc), Spark v1.16.2 (per /healthz).
• podman backend, --network-backend netavark, --db-backend sqlite.

[dndungu]

Corroborating data point from a separate incident on the same host, same day (pod name sire-gha-runner-0, not sire-dgx-runner-v2 — different session, likely concurrent with the report above).

Timeline (all times PDT, journalctl -t spark):

• 19:19 — fresh POST (YAML) succeeds normally: container starts, registers as a GitHub Actions runner, and actively processes at least one real job over the next ~4.5h (JobDispatcher] Job request ... processed succeed visible in the runner's own log right before shutdown). So the initial create on this pod name was not affected.
• 23:47:27 — something issues pod stop --time 10 sire-gha-runner-0 via the Spark API (caller not identifiable from journalctl -t spark; no request-identity/IP field logged on that line). The running runner container receives the stop cleanly (graceful shutdown, in-flight job finished first).
• 23:47:39–23:53:15 — restartPolicy: Always triggers three consecutive recreate attempts (pod rm → pod create → pod running → pod exited exitCode=0 → pod completed), each cycle lasting 5–15s. Same exact symptom as this issue: no podman run/container-create log line for the workload container in any of the three attempts, and correspondingly zero application log output from the runner container itself (no config.sh output, nothing) across all three restarts.
• 00:54:00 — housekeeper reaps the dead pod after the 1h completed-pod-ttl. restartPolicy: Always does not resurrect it after the record itself is gone — the pod simply ceases to exist with no alerting.

Net: this suggests the bug isn't limited to POST-triggered fresh creates — it also hits Spark's own internal recreate path on pod stop/restart-always, and once it does, a pod can silently vanish for good (create fails silently → looks completed → reaped → nothing left) rather than being stuck in a visibly broken state. Might be a useful angle for root-causing: whatever differs between the first successful pod create and the subsequent internal recreate (same manifest, same pod name) for this pod.

[dndungu]

Reopening: the fix merged in #90, but per this plan's Operating Procedure, issues close only after DGX live-verification, which hasn't run yet (PR auto-closed this via its Fixes line). Will close again once the release is deployed and live-verified.

[dndungu]

Reopening again: merging the release-please PR (#91) re-triggered GitHub's auto-close for the same reason as #71 (see that issue's comment) -- its changelog body inherited "closes #77" from the original commit trailer. Documented in docs/lore.md. Still queued for the coordinator's centralized live-verify pass.