Skip to content

feat(orchestrator): capture runner Pod post-mortem before cleanup #306

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
chrisleekr merged 3 commits into from
Sep 9, 2026
Merged

feat(orchestrator): capture runner Pod post-mortem before cleanup #306

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5c89e5f
feat(orchestrator): capture runner Pod post-mortem before cleanup
chrisleekr Sep 9, 2026
72828fc
fix(orchestrator): keep the post-mortem writable and wire it to both …
chrisleekr Sep 9, 2026
97711ae
fix(orchestrator): capture a clean exit, and cap the log tail in bytes
chrisleekr Sep 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/operate/deployment.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -313,6 +313,12 @@ rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create", "get", "delete"]
# Post-mortem only. Without it a dead runner's last output is unrecoverable,
# because cleanup deletes the Pod minutes after it dies. The controller
# degrades to recording the terminated container status alone.
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "get", "update", "delete"]
Expand Down
34 changes: 28 additions & 6 deletions docs/operate/observability.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -445,17 +445,39 @@ Structured lifecycle events for `workflow_runs` state transitions, emitted at th

`Workflow runner resources reconciled` (`src/k8s/workflow-runner-spawner.ts#ensureWorkflowRunnerResources`) is emitted once per attempt per reconciler pass, carrying `runId`, `attemptId` and `podName`. Two fields report what kubelet is doing with the Pod, which is how the controller separates a Pod that is still coming up from a runner that will never report in.

| Field | Values | Meaning |
| --------------- | -------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `startupPhase` | `starting` | Pod is not up yet: unscheduled, pulling, initialising volumes, or read back before it reported any status. The pass extends the startup lease, bounded by the attempt deadline and by `WORKFLOW_RUNNER_STARTUP_BUDGET_MS`. |
| `startupPhase` | `running` | Container started or already finished. Heartbeat renewals own the lease from here. |
| `startupPhase` | `stalled` | The attempt is terminalized on this pass rather than waiting out the lease. |
| `startupReason` | `InvalidImageName`, `ErrImageNeverPull`, `PodFailed`, or `not ready within 900s (<blocker>)` | Present only when `startupPhase` is `stalled`. Also appears in the failure comment as `Runner Pod could not start: <reason>`. |
| Field | Values | Meaning |
| --------------- | -------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `startupPhase` | `starting` | Pod is not up yet: unscheduled, pulling, initialising volumes, or read back before it reported any status. The pass extends the startup lease, bounded by the attempt deadline and by `WORKFLOW_RUNNER_STARTUP_BUDGET_MS`. |
| `startupPhase` | `running` | Container started or already finished. Heartbeat renewals own the lease from here. |
| `startupPhase` | `stalled` | The Pod is dead or permanently blocked. Terminalized on this pass **only before the payload is issued**; after that the attempt is left to lease expiry, and in both cases the pass captures a Pod post-mortem first (see below). |
| `startupReason` | `InvalidImageName`, `ErrImageNeverPull`, `PodFailed`, or `not ready within 900s (<blocker>)` | Present only when `startupPhase` is `stalled`. Also appears in the failure comment as `Runner Pod could not start: <reason>`. |

A run of consecutive `startupPhase=starting` lines for one `attemptId` measures total startup delay, not image-pull time alone: scheduling, volume setup and container creation all report the same phase. Before blaming a cold image cache, read the Pod's conditions and events (`kubectl describe pod <podName>`), which separate `Unschedulable` from `ImagePullBackOff` from volume attachment. The `startupReason` on the eventual `stalled` line names the blocker the controller last saw.

`Workflow runner startup lease extension refused` (`src/orchestrator/workflow-runner-reconciler.ts#reconcileActiveResources`) carries `runId` and `attemptId`. It means a `starting` pass did not actually extend the lease, because the payload was issued in the registration race, the lease lapsed between listing and the update, or the attempt deadline was reached. Without it, an attempt dying mid-startup looks identical to one being kept alive: both keep logging `starting`.

## Workflow runner Pod post-mortem

`event: "workflow_runner_pod_died"` (`src/orchestrator/workflow-runner-reconciler.ts#capturePodPostMortem`) is logged at `error` the first time a reconciler pass sees a dead Pod for an attempt, and at most once per attempt. Dead means either a `stalled` startup phase or a Pod that reached `Succeeded`: under `restartPolicy: Never` a runner whose process returns 0 without ever sending a result lands in `Succeeded`, which startup classifies as `running`, and reaching an attempt that is still active means no result was reported. It is the answer to "why did the lease expire", which the lease-expiry notice itself cannot give.

It exists because everything else is gone by the time anyone asks. Cleanup deletes the Pod, and with it the terminated container status and the container log, within seconds of the run being terminalized. A cluster log pipeline is not a dependable second copy: on this deployment the Datadog Logs **aggregate** API counted 547 lines for a dead runner while the **search** API returned zero retrievable events, and no container metrics were collected at all.

| Field | Meaning |
| ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `podName`, `podPhase` | Which Pod, and its phase at capture. `Failed` is a non-zero exit under `restartPolicy: Never`; `Succeeded` is a clean exit that never reported a result; `Pending` is a container that never ran. |
| `reason` | Kubelet's **container** termination reason. `OOMKilled` is the memory limit; `Error` is a non-zero exit. |
| `podReason`, `podMessage` | The **Pod**-level verdict, which is where a node decision lands: `Evicted` for node pressure such as a filled ephemeral-storage limit, `DeadlineExceeded` for `activeDeadlineSeconds`. A container that never ran leaves `reason` null and only these set. |
| `runId`, `attemptId`, `workflowName`, `startupReason` | Attempt identity plus the blocker the reconciler last saw, so the line joins to the `startupPhase` trail above. |
| `exitCode`, `signal` | `137` with `OOMKilled` is the kernel OOM killer. A bare `137` without that reason is an external `SIGKILL`. |
| `message` | The container's termination message, when the runtime set one. |
| `startedAt`, `finishedAt` | Container lifetime. `finishedAt` should match the abnormal WebSocket close, see below. |
| `logTail` | Last 200 lines of runner stdout, secret-stripped and capped at 16 KB, measured in UTF-8 bytes. The cut starts on a character boundary, because a partial character would make the row unwritable as `jsonb`. Empty when `pods/log` RBAC is missing. |
| `logError` | Why `logTail` is not the whole story. A `403` means the Role is missing `pods/log` `get`, see the deployment RBAC. A `transfer ceiling` message means the 200-line window exceeded 256 KB, so the server stopped sending before the runner's final lines. |

The same record is persisted to `workflow_runs.state._runnerPostMortem`, so it survives log rotation and is queryable per run. Both failure comments quote it, the lease/deadline expiry notice and the runner-start-failure notice, since a Pod that dies before registering is terminalized by the second. Each quotes only the kubelet reason (`reason`, falling back to `podReason`) and `exitCode`, and shape-checks the reason against `^[A-Za-z][A-Za-z0-9]{0,63}$` first; `message`, `podMessage` and `logTail` are repository or node content and stay in the controller log. An all-null reading is discarded rather than recorded, so a pass that catches the Pod before kubelet wrote its status does not spend the one-shot slot.

Correlate with the socket: `WebSocket connection closed` carries `kind`, plus `runId` and `attemptId` for a `workflow-runner` socket. Close code `1006` (abnormal, no close frame) at the same second as `finishedAt` is the signature of a runner killed outright rather than one shutting down.

## Workspace lifecycle events

The `workspace.*` family makes the non-success workspace-cleanup paths greppable, complementing the success-path `pipeline.stage stage=workspace.cleanup` row and the startup `workspace.sweep` reaper. Schema pinned by `WorkspaceLogFieldsSchema` (`src/core/workspace-events.ts#WorkspaceLogFieldsSchema`). `workDir` is a process-local temp path and safe to log; the authenticated clone URL embeds the install token and is never logged (clone events carry the `owner/repo` slug and branch only). All `err` fields routed through `redactErrorMessage`.
Expand Down
Loading
Loading