docs: add e2e-overnight-run skill for full runtime x OS validation - #1041
Conversation
github-actions
Bot
added
the
waiting-on-maintainers
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| cd "$REPO" | ||
|
|
||
| variant_infra() { case "$1" in linux-*) echo ".e2e_linux_infra.sh";; windows-*) echo ".e2e_windows_infra.sh";; esac; } | ||
| variant_script() { case "$1" in *-python) echo "test";; *-rust) echo "test-rust";; esac; } |
There was a problem hiding this comment.
The *-rust variants map to hatch run e2e:test-rust, but no test-rust script exists in hatch.toml — the only script defined under [envs.e2e.scripts] is test (hatch.toml:48). This command will fail immediately with a hatch "unknown script" error, so both rust variants can never run and will be recorded as failures (then retried and failed again).
Looking at how rust is actually exercised in the suite: test/e2e/test_session_runtime.py selects the runtime via the explicit_runtime_worker fixture parametrized over python/rust (and service-selected), driven by DeadlineWorkerConfiguration.session_runtime on a per-worker basis. There is no whole-suite rust pin, no --session-runtime pytest option, and no WORKER_AGENT_SESSION_RUNTIME env var in the repo — so the SKILL.md "Runtime selection reference" table describing e2e:test-rust, --session-runtime, and WORKER_AGENT_SESSION_RUNTIME does not match the actual test harness. As written, linux-rust/windows-rust would run the identical test command as the python variants (once test-rust is fixed), or fail outright (as-is). Please reconcile the supervisor and SKILL.md with the real mechanism.
There was a problem hiding this comment.
sorry claude, the #1040 hasnt merged yet so you cant see the test script
Add a skill that runs the test/e2e suite across all four runtime x OS variants (Linux/Windows x Python/Rust session runtimes) unattended, for release validation. overnight_e2e.sh validates prerequisites, builds the wheel under test, rebuilds the e2e env, verifies the test-fixtures expose the session_runtime field, and launches a detached tmux supervisor. _supervise.sh runs each variant sequentially with a single retry and writes a SUMMARY.md results table. Includes a constraints-file template for hosts lacking pre-built wheels for a test dependency. Signed-off-by: Sean Tang <171081544+seant-aws@users.noreply.github.com>
0180e30 to
8efcee9
Compare
| env -u TMUX tmux kill-session -t "$TMUX_SESSION" 2>/dev/null || true | ||
| env -u TMUX tmux new-session -d -s "$TMUX_SESSION" -x 220 -y 50 | ||
| env -u TMUX tmux send-keys -t "$TMUX_SESSION" \ | ||
| "REPO='$REPO' REGION='$REGION' VARIANTS='$VARIANTS' PIP_CONSTRAINT='${CONSTRAINTS}' WORKER_AGENT_WHL_PATH='$WHL' bash '$SCRIPT_DIR/_supervise.sh' 2>&1 | tee '$OUTDIR/supervisor.log'" \ |
There was a problem hiding this comment.
The supervisor is launched via tmux send-keys into a session on the tmux server, and a child session inherits the server’s environment — not this shell’s — for anything outside tmux’s update-environment list (which is DISPLAY/SSH_* only; AWS_* are not on it). This script explicitly forwards REPO/REGION/VARIANTS/PIP_CONSTRAINT/WORKER_AGENT_WHL_PATH on the send-keys line, but not the AWS credentials it just validated.
Consequences for the common credential setups the SKILL lists:
AWS_PROFILE/ env-var STS creds (AWS_ACCESS_KEY_ID/AWS_SESSION_TOKEN, SSO):overnight_e2e.shvalidates them withaws sts get-caller-identityin its own shell (line 95) and passes, but if a tmux server is already running (e.g. you are inside tmux, or ran this before), the new session gets that pre-existing server’s stale/empty AWS environment. The 6–8h unattended run then authenticates against the wrong account — or failsget-caller-identityinrun_oneand burns both attempts of every variant on an auth error, despite the up-front check having succeeded.- Only the fresh-server case (no tmux server yet) inherits the current env correctly, so this fails intermittently based on prior tmux state — the worst kind of unattended-run flake.
Fix: forward the credential env explicitly on the send-keys line the same way REGION is (e.g. AWS_PROFILE='$AWS_PROFILE' AWS_ACCESS_KEY_ID=... AWS_SESSION_TOKEN=...), or set them in the session via tmux setenv before send-keys. At minimum, document that credentials must be present in the tmux server environment.
|
I wonder whether this could be merged with existing testing skill. |
4 participants
What was the problem/requirement? (What/Why)
Validating a worker-agent release candidate means running the E2E suite across all four runtime x OS combinations (Linux/Windows x Python/Rust session runtimes). Done by hand this is four long, error-prone runs with several easy-to-miss traps (wrong wheel silently tested, stale test-fixtures env, distinguishing infra flakes from real failures).
What was the solution? (How)
Add an
e2e-overnight-runskill underskills/:overnight_e2e.sh— single entry point. Validates prerequisites (worktree, credentials, infra scripts, tools) and aborts before provisioning if any are missing; builds the wheel under test and exportsWORKER_AGENT_WHL_PATH; rebuilds the e2e env; verifies the test-fixtures exposesession_runtime; launches a detached tmux supervisor._supervise.sh— runs each variant sequentially (never two farms concurrently), retries once on failure, writes aSUMMARY.mdresults table with per-attempt pytest banners.e2e-constraints.txt.example— template for the optional--constraintsfile on hosts lacking a pre-built wheel for a test dependency.Complements the existing
worker-agent-testingskill (which documents the one-off E2E setup) by automating the full validation matrix.What is the impact of this change?
Docs/tooling only. No product code touched. Gives a repeatable, one-command way to validate both session runtimes on both OSes.
How was this change tested?
bash -non both scripts;--helpand fail-loud prerequisite validation exercised manually.Was this change documented?
Yes — the skill's
SKILL.mdis the runbook.Is this a breaking change?
No.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.