aws-deadline · seant-aws · Aug 7, 2026 · Aug 7, 2026 · github-actions · Aug 7, 2026
@@ -0,0 +1,144 @@
+---
+name: e2e-overnight-run
+description: Run the full worker-agent E2E suite across all four runtime x OS variants (Linux/Windows x Python/Rust session runtimes) unattended against the live Deadline Cloud service. Builds the wheel under test, rebuilds the e2e env, and supervises sequential runs in tmux with automatic retry and a results summary. Use when validating a release candidate, a session-runtime change, or a dependency bump before shipping.
+tags: [deadline-cloud, worker-agent, e2e, session-runtime, rust, python, linux, windows, overnight, release-validation, tmux, hatch, pytest]
+---
+
+# Comprehensive Worker Agent E2E Overnight Run
+
+Run `test/e2e` across all four combinations — **Linux + Python**,
+**Linux + Rust**, **Windows + Python**, **Windows + Rust** — in one unattended,
+tmux-backed, retrying sequence. Built for release validation: kick it off, come
+back later, read one summary table.
+
+The worker agent's `session_runtime` setting selects the OpenJD session backend
+(`python` = v0, `rust` = v1). E2E only pins the runtime on a fresh worker
+config, so validating both backends means running the suite twice per OS. This
+skill automates that matrix. It builds on the E2E setup documented in the
+`worker-agent-testing` skill and `DEVELOPMENT.md`.
+
+## When to use
+
+- Validating a **release candidate** before shipping.
+- A change to the **session runtime** (adapter code, runtime selection, the
+  service `runtimeHint` path) or an **openjd-sessions / openjd-model pin bump**.
+- Any change where "does the agent still run real jobs on both runtimes, on
+  both OSes?" is the question.
+
+For a scoped change (Rust-only), run just the two rust variants — see
+`--variants`.
+
+## Runtime selection reference
+
+| Item | Value |
+|------|-------|
+| Config field | `session_runtime` = `python` \| `rust` \| `service-selected` (config-file / CLI only — no environment variable) |
+| Test-fixtures field | `DeadlineWorkerConfiguration.session_runtime` (recent `deadline-cloud-test-fixtures`) |
+| Hatch scripts | `e2e:test` (agent default = python), `e2e:test-rust` (whole suite pinned to rust) |
+| Runtime selector | `--session-runtime {python,rust,service-selected}` pytest option / `WORKER_AGENT_SESSION_RUNTIME` env var |
+| Canonical runtime proof | worker log line `Selected session runtime: <rt> (hint=...)` |
+
+## Prerequisites
+
+1. **A worker-agent git worktree** with the code under test checked out (or
+   cherry-picked in). The wheel is built from here — test mainline from a
+   mainline checkout, or a PR branch by checking it out first.
+2. **AWS credentials** for your e2e account, valid long enough for the whole
+   run (~6–8h for all four variants). Configure them however you normally do
+   (`AWS_PROFILE` + `AWS_DEFAULT_REGION`, SSO, etc.).
+3. **Testing infrastructure deployed** and the per-OS infra env-var scripts
+   present in the worktree root. See the E2E prerequisites in the
+   `worker-agent-testing` skill / `DEVELOPMENT.md`:
+   ```sh
+   scripts/deploy_e2e_testing_infrastructure.sh
+   ./scripts/get_e2e_test_ids_from_cfn.sh --os Linux   > .e2e_linux_infra.sh
+   ./scripts/get_e2e_test_ids_from_cfn.sh --os Windows > .e2e_windows_infra.sh
+   ```
+4. `tmux`, `hatch`, `git`, `aws` on PATH.
+
+## Run it
+
+```sh
+# All four variants, unattended:
+skills/e2e-overnight-run/scripts/overnight_e2e.sh --repo /path/to/worktree
+
+# Rust-only re-validation after a Rust-path change:
+overnight_e2e.sh --repo /path/to/worktree --variants "linux-rust windows-rust"
+```
+
+The script, in order:
+
+1. Validates every prerequisite (worktree, credentials, infra scripts, tools)
+   and **aborts before provisioning anything** if one is missing.
+2. `hatch build` → exports `WORKER_AGENT_WHL_PATH` to the built wheel.
+3. Rebuilds the `e2e` hatch env so it resolves the pinned test dependencies.
+4. **Verifies `DeadlineWorkerConfiguration` exposes `session_runtime`** — the
+   most common stale-environment failure.
+5. Launches a detached tmux supervisor that runs each variant sequentially,
+   retrying once on failure.
+
+Everything lands in `<repo>/.e2e-overnight/`:
+
+| File | Meaning |
+|------|---------|
+| `<variant>.log` | full pytest output |
+| `<variant>.exit` | exit code (`0` = all passed) |
+| `<variant>-retry.log` | retry attempt, if the first failed |
+| `SUMMARY.md` | one row per attempt with the pytest banner |
+| `SUPERVISOR_DONE` | present once every variant has finished |
+
+## Read the results
+
+```sh
+cat <repo>/.e2e-overnight/SUMMARY.md
+```
+
+- **Exit 0** on a variant = all tests passed on that runtime + OS.
+- **Red on attempt 1, green on retry** = infrastructure flake (EC2 capacity,
+  SSM timeout, bootstrap race), not a code failure.
+- **Red on both** = real failure. Read `<variant>.log`; search for `^FAILED ` /
+  `^ERROR ` lines (not the substring `CREATE_FAILED`, which is benign
+  pre-test-cleanup noise).
+
+Wall time is roughly 6–8h for all four (Linux ~1.5h each, Windows ~2.5h each).
+
+## Verifying a run was genuine (not a silent fallback)
+
+Two traps make a green run meaningless — check both when it matters:
+
+1. **Wrong wheel.** If `WORKER_AGENT_WHL_PATH` is unset, the harness installs
+   the published package instead of your build. `overnight_e2e.sh` always sets
+   it; confirm via the `wheel:` line at the top of each `<variant>.log`.
+2. **Runtime actually used.** The rust variants use `e2e:test-rust` (recorded
+   as the `script:` line in the log). To prove a session used rust, grep the
+   worker's own log line `Selected session runtime: rust`. The
+   `test/e2e/test_session_runtime.py` routing tests assert exactly this, and a
+   negative control (an invalid `session_runtime` value making the worker fail
+   to start) is how the injection path is validated.
+
+## Gotchas
+
+- **Stale e2e env.** A previously built `e2e` env can pin an old
+  `deadline-cloud-test-fixtures` without the `session_runtime` field, so the
+  runtime-routing tests die on a config-injection error. The script rebuilds
+  the env by default; use `--keep-env` only when you know it is current.
+- **Missing binary wheels on older hosts.** If the env build fails compiling a
+  test-only transitive dependency from source, pass a pip constraints file via
+  `--constraints` (see `scripts/e2e-constraints.txt.example`).
+- **Parametrize labels are not the OS.** A test id like
+  `test_...path_mapping[windows]` in the Linux suite is a *parameter label* (a
+  Windows-style storage profile exercised on the Linux worker), not a Windows
+  worker. Real Windows-only tests appear as `SKIPPED` in Linux runs.
+- **Windows `xpassed`/`xfailed`.** Some Windows-only tests are marked
+  `xfail(strict=False)` for known-unreliable environment setup and may xfail or
+  xpass run-to-run without affecting the exit code.
+- **Credential lifetime.** The supervisor cannot refresh credentials; ensure
+  they cover the full run before launching.
+
+## Scripts
+
+| Script | Role |
+|--------|------|
+| `scripts/overnight_e2e.sh` | Entry point — validate, build, prep env, launch supervisor. |
+| `scripts/_supervise.sh` | Internal — sequential runs, retry, `SUMMARY.md`. |
+| `scripts/e2e-constraints.txt.example` | Template for the optional `--constraints` file. |
@@ -0,0 +1,98 @@
+#!/usr/bin/env bash
+#
+# _supervise.sh — internal helper for overnight_e2e.sh. Runs each requested
+# variant to completion, one at a time (never two farms' workers concurrently),
+# retrying once on failure, and records a results table in SUMMARY.md.
+#
+# Expects these from the caller's environment:
+#   REPO                   worker-agent worktree
+#   REGION                 AWS region
+#   VARIANTS               space-separated variant list
+#   WORKER_AGENT_WHL_PATH  wheel under test
+#   PIP_CONSTRAINT         (optional) pip constraints file
+#
+# Not meant to be invoked directly — run overnight_e2e.sh instead.
+set -uo pipefail
+
+OUTDIR="$REPO/.e2e-overnight"
+SUMMARY="$OUTDIR/SUMMARY.md"
+
+export AWS_DEFAULT_REGION="$REGION"
+export WORKER_AGENT_WHL_PATH
+[[ -n "${PIP_CONSTRAINT:-}" ]] && export PIP_CONSTRAINT
+
+cd "$REPO"
+
+variant_infra()  { case "$1" in linux-*) echo ".e2e_linux_infra.sh";; windows-*) echo ".e2e_windows_infra.sh";; esac; }
+variant_script() { case "$1" in *-python) echo "test";; *-rust) echo "test-rust";; esac; }
+
+pytest_banner() {
+    grep -aoE '=+ [0-9]+ (passed|failed)[^=]*=+' "$1" 2>/dev/null | tail -1 \
+        || echo "(no pytest summary — run did not reach a verdict)"
+}
+
+{
+    echo "# Worker Agent E2E — overnight run summary"
+    echo
+    echo "- started: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
+    echo "- repo: $REPO"
+    echo "- commit: $(git log --oneline -1)"
+    echo "- wheel: $WORKER_AGENT_WHL_PATH"
+    echo "- region: $REGION"
+    echo
+    echo "| variant | attempt | exit | pytest summary |"
+    echo "|---------|---------|------|----------------|"
+} > "$SUMMARY"
+
+run_one() {
+    local tag="$1" infra="$2" script="$3" log="$OUTDIR/${1}.log"
+    {
+        echo "=== $tag ==="
+        echo "started:  $(date -u +%Y-%m-%dT%H:%M:%SZ)"
+        echo "infra:    $infra"
+        echo "script:   hatch run e2e:$script"
+        echo "wheel:    $WORKER_AGENT_WHL_PATH"
+        echo "identity: $(aws sts get-caller-identity --query Arn --output text 2>&1)"
+        echo "==============================="
+    } > "$log"
+    # shellcheck disable=SC1090
+    ( source "$REPO/$infra"; echo "OPERATING_SYSTEM=${OPERATING_SYSTEM:-?} FARM_ID=${FARM_ID:-?}" >> "$log"; \
+      hatch run "e2e:$script" >> "$log" 2>&1 )
+    local rc=$?
+    { echo "==============================="; echo "finished: $(date -u +%Y-%m-%dT%H:%M:%SZ)"; echo "EXIT=$rc"; } >> "$log"
+    return $rc
+}
+
+for tag in $VARIANTS; do
+    infra="$(variant_infra "$tag")"
+    script="$(variant_script "$tag")"
+
+    echo "[supervisor] starting $tag ($(date -u +%H:%MZ))"
+    run_one "$tag" "$infra" "$script"; rc=$?
+    echo "$rc" > "$OUTDIR/${tag}.exit"
+    echo "| $tag | 1 | $rc | $(pytest_banner "$OUTDIR/${tag}.log") |" >> "$SUMMARY"
+
+    # One automatic retry. Most non-zero exits here are transient infra flakes
+    # (EC2 capacity, SSM timeouts, bootstrap races); pre_test_cleanup runs at
+    # the start of every attempt. Keep both logs so a real failure (both red)
+    # is distinguishable from a flake (retry green).
+    if [[ "$rc" != "0" ]]; then
+        echo "[supervisor] $tag failed (exit=$rc); retrying once"
+        run_one "${tag}-retry" "$infra" "$script"; rrc=$?
+        echo "$rrc" > "$OUTDIR/${tag}-retry.exit"
+        echo "| $tag | 2 (retry) | $rrc | $(pytest_banner "$OUTDIR/${tag}-retry.log") |" >> "$SUMMARY"
+    fi
+done
+
+{
+    echo
+    echo "- finished: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
+    echo
+    echo "Exit 0 = all tests passed. Non-zero = pytest reported failures or the"
+    echo "run could not start; read the matching .log. A variant that is red on"
+    echo "attempt 1 but green on retry was an infrastructure flake, not a"
+    echo "code failure."
+} >> "$SUMMARY"
+
+touch "$OUTDIR/SUPERVISOR_DONE"
+echo "[supervisor] ALL RUNS COMPLETE — see $SUMMARY"
@@ -0,0 +1,13 @@
+# Optional pip constraints for the e2e env, passed via
+#   overnight_e2e.sh --constraints scripts/e2e-constraints.txt.example
+#
+# Only needed on hosts where a test-only transitive dependency (pulled in by
+# deadline-cloud-test-fixtures) has no compatible pre-built wheel and fails to
+# build from source — for example an older Linux whose glibc predates the
+# manylinux target of a recent pillow/numpy release. On such a host, pin those
+# dependencies below their first incompatible version. CI runners on current
+# base images do not need this file.
+#
+# Example (adjust versions to the packages that fail on your host):
+# pillow < 12.3
+# numpy < 2.3