Skip to content

Fix check_deployments never deploying missing workflows - #1319

Open
mvdbeek wants to merge 1 commit into
mainfrom
iwc-missing-deployments
Open

Fix check_deployments never deploying missing workflows#1319
mvdbeek wants to merge 1 commit into
mainfrom
iwc-missing-deployments

Conversation

@mvdbeek

@mvdbeek mvdbeek commented Jul 20, 2026

Copy link
Copy Markdown
Member

Problem

The Check and fix missing deployments workflow can never fix anything.

The deploy job depends on check, but its if lacked always(), so GitHub Actions skips it whenever check fails. Since scripts/check_missing_deployments.py exited 1 precisely when it found missing deployments, deploy could only ever run when there was nothing to deploy. The weekly scheduled run has been going red without redeploying anything.

Most recent failing run: https://github.com/galaxyproject/iwc/actions/runs/29723072857 — 5 missing deployments out of 106 repos.

Changes

  • Add always() to the deploy job's if, so it survives a failing check job.
  • The check script reports through the has-missing output and exits 0 under Actions. --fail-on-missing, and running outside Actions, keep the old non-zero exit.
  • release_exists treated any non-zero gh exit as a missing release, so a rate limit or network error looked like ~100 missing deployments. Only a genuine 404 counts as missing now; other errors are retried 3x and then raised, failing the job loudly instead of triggering a mass redeploy.
  • The deploy loop records per-repo failures and continues instead of aborting on the first one, then exits 1 with a summary, so one bad repository no longer strands the rest. (Note: planemo-ci-action's mode: deploy does || exit 1 per repo, so switching to it would reintroduce this.)
  • missing-repos is passed via the environment rather than interpolated into the shell script.

CI status semantics after this change

Situation Run status
Nothing missing green
Missing, all deploys succeed green (found and fixed)
Missing, a deploy fails red
API error / crash in check red

Verification

  • Extracted the real run: block and executed it against a stub planemo failing for one of three repos: it deployed the other two and still exited 1.
  • Unit-tested release_exists for present / 404 / persistent 403 / transient-then-success.
  • Checked release_exists against the live API: amr_gene_detection v1.1.7 -> True, v1.1.8 -> False, repeatmasking v0.1 -> False.
  • Exit-code matrix verified for all three modes; flake8 clean.

Follow-up, not in this PR

The 5 workflows currently missing deployment need a decision before the deploy is dispatched, because this workflow deploys directly without a test gate:

Workflow Test evidence
gromacs-dctmd v0.1.7 green on current main
functional-annotation-of-sequences v0.3 green on current main
amr_gene_detection v1.1.8 last test job failed after 6h5m (Actions 6-hour ceiling)
clinicalmp-verification v0.2 merged Jan 2025, checks aged out
repeatmasking v0.1 nightly wftest.yml failing 8 of last 8 runs

Separately: workflows/enable_ci_workflows.py calls /orgs/iwc-workflows/repos without pagination, so it only ever sees the first 30 of the org's 112 repos. Repos outside that first page are never re-enabled after disabled_inactivityamr_gene_detection and clinicalmp-verification are both stuck disabled for this reason.

The deploy job depends on the check job, but its `if` lacked always(), so
it was skipped whenever check failed -- which is exactly when there is
something to deploy. Since the check script exited 1 on finding missing
deployments, the deploy job could only run when there was nothing to do.

The script now reports through the has-missing output and exits 0 under
Actions; --fail-on-missing, and running outside Actions, keep the old
non-zero exit. A failing deploy job still turns the run red.

Also:

- release_exists treated any non-zero gh exit as a missing release, so a
  rate limit or network error looked like ~100 missing deployments. Only
  a genuine 404 counts as missing now; other errors are retried and then
  raised.
- The deploy loop records per-repo failures and continues instead of
  aborting on the first one, then exits 1 with a summary, so one bad
  repository no longer strands the rest.
- missing-repos is passed via the environment rather than interpolated
  into the shell script.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant