install-system-deps: hardcoded 300s per-command timeout is un-overridable and fails the post-publish Pages job on a slow apt mirror

[thorwhalen]

install-system-deps applies a hardcoded, un-overridable 300s-per-command timeout. On a slow apt mirror this fails the post-publish Publish GitHub Pages job — the one half of the workflow that cannot be safely re-run as a whole.

What happened

thorwhalen/mixing run 31037584210 (push to main, 2026-08-05). Publish succeeded and uploaded 0.0.32 to PyPI. Publish GitHub Pages then failed at step 3, Install System Dependencies:

✗ Installation failed for ffmpeg: Timeout after 300s
✗ Installed: 0
⊘ Already present: 0
✗ Failed: 1
##[error]Process completed with exit code 1.

This was a transient mirror stall, not a config error. The evidence is unusually clean:

• In the same run, the Validation (3.10) job ran the same action against the same [tool.wads.ops.ffmpeg] block and installed ffmpeg successfully in ~55s (19:03:56 → 19:04:51).
• In the failing job the apt download simply stalled: Get:9 landed at 19:06:47, Get:10 at 19:11:07 — 4m20s on a single 9MB package from azure.archive.ubuntu.com.
• Re-running only that job on the same commit: green in 1m21s.

So the action is not broken and the repo config is not wrong. What the incident exposes is a timeout that is too tight for the workload and cannot be adjusted.

The defect

wads/install_system_deps.py:

def install_dependency(dep_name, install_cmds, timeout: int = 300):
    ...
    result = subprocess.run(cmd, shell=True, capture_output=False, timeout=timeout)
    ...
    except subprocess.TimeoutExpired:
        return False, f"Timeout after {timeout}s"

and the sole call site:

success, error = install_dependency(dep_name, install_cmds)   # timeout never passed

install_system_dependencies(...) takes pyproject_path / platform / skip_check / verbose — no timeout. The argparse CLI has no --timeout. actions/install-system-deps/action.yml has no timeout input. The 300s is therefore unreachable from every layer a consumer can touch: not the action, not pyproject.toml, not the CLI.

apt-get install -y ffmpeg on ubuntu-latest pulls 99 packages / 94.1 MB / 227 MB unpacked. A 300s budget for that is roughly 3–5x a good run and leaves no headroom for a slow mirror.

Why this is worse than an ordinary flake

The job runs needs: [setup, publish], i.e. after the PyPI upload. So the cheap reflex — re-run the whole workflow — is exactly the wrong move: Publish would re-execute against a version already on PyPI. The safe recovery is the non-obvious gh run rerun <run-id> --job <job-id>, which leaves Publish untouched. Anyone reaching for "re-run all jobs" here gets a duplicate-version failure at best.

There is a second, related visibility trap. Publish and Publish GitHub Pages are both default-branch-only, so on the PR check run they show as skipped. A green PR is therefore no evidence at all about this job — and in this incident it caused the post-merge failure to be reported as green.

Blast radius

Seven repos in the local ecosystem declare [tool.wads.ops.*]. Five run the same heavy apt-get install -y ffmpeg on linux — mixing, nw, muvid, reelee, braidio (all in the video_gen federation) — and each is exposed to the same fixed 300s on every default-branch push. audiostream2py (portaudio19-dev) and odbcdol are lighter but share the ceiling.

Suggested fixes (not implemented — this action is the publish gate for ~190 repos)

Roughly in order of value-for-risk:

1. Plumb the timeout through: timeout input on action.yml → --timeout on the CLI → install_system_dependencies(..., timeout=...) → install_dependency(...). Optionally allow a per-dependency timeout key in [tool.wads.ops.<dep>]. Raise the default to something like 900s; the current 300s is not a sane default for an ffmpeg-class install.
2. Retry on timeout (1–2 attempts). Note Acquire::Retries will not help here — the download was progressing, just slowly, so the fix is wall-clock budget, not retry-on-error.
3. Distinguish timeout from genuine failure in the exit status/summary, so a mirror stall is not reported identically to "this package does not exist".
4. Consider whether the pages job needs the full system dep at all — it installs the package only to extract docstrings, and a leaner check (or continue-on-error on the docs path specifically) may be the better structural answer. Flagging, not recommending.

Minor, unrelated, spotted while reading action.yml: the final step does

python3 -m wads.install_system_deps $ARGS
EXIT_CODE=$?
echo "::endgroup::"
exit $EXIT_CODE

GitHub runs shell: bash with -e, so a non-zero exit aborts the step at the python line and ::endgroup:: never closes. Cosmetic only — the step's failure is still reported correctly.

Recovery already applied

thorwhalen/mixing run 31037584210 was recovered with a job-scoped re-run; the run is now green end-to-end and the docs site rebuilt. No code changed in mixing and nothing changed here.