Fix two chain-killing bugs in the PBS self-scheduling recipe

[OwenPriceSkelly]

Summary

Two latent bugs in scripts/nightly_smoke_test.pbs meant every PBS chain ran exactly one generation, then died silently — generation 2 crashed before its payload and never scheduled generation 3.

Bug 1 — self-path breaks at generation 2. The script located itself via $PBS_O_WORKDIR, which is only the submit directory for the first, human-submitted generation. For a job submitted by another PBS job, PBS_O_WORKDIR is the parent job's cwd — $HOME, since PBS starts jobs there and the script never cds. Generation 2's qsub of the bogus $HOME/... path fails, and set -euo pipefail kills the job before the smoke-test runs. Fix: each generation passes the path down explicitly with qsub -v SELF_PATH=...; the existing ${SELF_PATH:-...} default already honors the override. CLUSTER rides along in the same -v, since PBS doesn't propagate the environment down the chain and shared installs (sophia/polaris) need it on every generation — previously it would have silently reverted to the manifest default from generation 2 on. -v passes only the listed variables, so nothing else about the job environment changes.

Bug 2 — duplicate-chain guard never matched. The guard used qselect ... -s Q, but a job submitted with -a sits in state W (waiting), not Q, so the guard was a no-op. Fix: -s QW.

Also corrects the guidance that went with them, in the script header and docs/nightly-smoke-testing.md:

• Stopping a chain: look for the waiting (W) job, not a queued (Q) one.
• Fast end-to-end test: override RESCHEDULE_AT (via qsub -v) so generation 1 schedules generation 2 a few minutes out — that exercises the actual handoff. The previously suggested qsub -a <time> on the kickoff only delays generation 1's own start and tests nothing about the chain.

The SLURM sibling (nightly_smoke_test.sbatch) is unaffected: it recovers its path from scontrol show job and SLURM propagates the submit environment by default.

Test plan

• bash -n on the edited script.
• The fixed pattern (explicit -v SELF_PATH handoff + QW guard) is lifted from a sibling self-scheduling chain script developed against ALCF; the running Sophia/Polaris chains will be re-kicked-off with per-cluster copies of this fix, which exercises the generation 1 → 2 handoff for real.

🤖 Generated with Claude Code