Skip to content

fix(ci): retry artifact transfers and release uploads; queue releases instead of dropping them - #56

Closed
ryan-morris wants to merge 1 commit into
mainfrom
fix/ci-artifact-retry
Closed

ryan-morris wants to merge 1 commit into
mainfrom
fix/ci-artifact-retry

Conversation

@ryan-morris

Copy link
Copy Markdown
Member

Why

Release run 37631795508 failed on one cell, 9.0.2 osx-x64 (gplv2), even though its build succeeded. actions/upload-artifact died partway through the transfer to blob storage, after 8 MiB, with no error message. Recovering meant re-running the whole hour-long macOS build by hand.

Artifact transfer has no retry of its own. The iOS and SHA256SUMS release uploads had none either.

This PR also fixes a separate silent problem. The default concurrency.queue: single keeps only one pending release:

  • A release that arrives while another is pending cancels and replaces the pending one.
  • So merging several release-triggering PRs during a long release dropped the middle ones.
  • Each run selects only what its own push changed, so nothing ever built the dropped releases.

What

  • Artifact transfers retry once. This covers every build upload (artifact, win-x64 symbols) and every test/release download. The first attempt is continue-on-error; after a 45 s pause the second attempt is the verdict.
    • Upload retries use overwrite: true under the same name, because downstream jobs download by exact name.
    • The retries are inline steps, not a local composite action: the publish jobs have no checkout, and adding one to ~90 cells would introduce a new failure point of the same kind.
  • publish-ios upload and SHA256SUMS now get the same bounded 6-attempt retry the desktop publish upload already had. --clobber keeps both idempotent. A failed manifest generation never uploads, because the upload is gated with &&.
  • release.yml concurrency now uses queue: max. Up to 100 releases can wait, and they run in order.
  • Comments that said the release is public while assets upload are corrected. The release is a draft until finalize, which gates on every upload and on the manifest.

Verification

  • queue: max: actionlint 1.7.12 rejects the key (rhysd/actionlint#654), and an invalid release.yml would stop releases. So I confirmed GitHub accepts it with a throwaway branch and probe workflow, which passed; both are deleted. GitHub's workflow-syntax docs also document it. The repo's CI does not run actionlint.
  • actionlint: no other findings on the four changed workflows.
  • Repo gates: image-pins-test, select-versions-test and matrix-consistency-test pass.
  • Codex: approved the plan with changes; this is the artifact-transfer part of it. Its code review verdict was SHIP with no findings. It checked the retry conditions and outcomes, that diagnostics uploads still fire, the default shell for sleep on every host and container, and the exit paths of both retry loops.

The workflow edits select no FFmpeg versions, so merging this cuts no release. Ordinary green CI also doesn't exercise the retry paths; they only run when a transfer actually fails.

Follow-up (separate PR, design under review):

  • Releases that repair themselves: rebuild any line whose inputs changed since its last published release.
  • A rerun-safe release job.
  • One automatic re-run of failed jobs.

🤖 Generated with Claude Code

… instead of dropping them

A release cell failed after its osx-x64 build succeeded: actions/upload-artifact died mid-transfer
to blob storage (after 8 MiB, no error message), and the only recovery was a manual re-run of the
whole build. Artifact transfer has no retry of its own, and the iOS and SHA256SUMS release uploads
had none either.

- Every build upload (artifact, win-x64 symbols) and every test/release download gets one retry
  after a 45s pause: attempt 1 is continue-on-error, attempt 2 is the verdict. Uploads retry with
  overwrite: true under the SAME name, since downstream jobs download by exact name. Inline steps,
  not a local composite action: the publish jobs have no checkout, and adding one to ~90 cells
  would be a new failure point of the same kind.
- publish-ios upload and SHA256SUMS generation+upload get the same bounded 6-attempt retry the
  desktop publish upload already had (--clobber keeps both idempotent).
- release concurrency: queue: max. With the default queue: single, a release that arrives while
  another is pending cancels and replaces it, so merging several release-triggering PRs during a
  long release silently dropped the middle ones -- and since each run selects only what its own
  push changed, nothing ever built them. max keeps up to 100 pending and runs them in order.
  (actionlint 1.7.12 does not know the key yet -- rhysd/actionlint#654 -- so it was verified on
  GitHub with a throwaway probe workflow.)
- Comments that still said the release is public while assets upload are corrected: it is a
  draft until finalize, which gates on every upload and the manifest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ryan-morris

Copy link
Copy Markdown
Member Author

Closing: #60 moves this repository to ffmpeg-build, whose release workflow already retries every asset upload (a release stays a draft until all are up) and queues releases instead of dropping them.

@ryan-morris
ryan-morris deleted the fix/ci-artifact-retry branch October 14, 2026 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant