Skip to content

ci: size runners to the macOS baseline, self-heal docker cache corruption - #51

Merged
lemon07r merged 1 commit into
masterfrom
ci/balance-to-macos-baseline
Aug 19, 2026
Merged

ci: size runners to the macOS baseline, self-heal docker cache corruption#51
lemon07r merged 1 commit into
masterfrom
ci/balance-to-macos-baseline

Conversation

@lemon07r

@lemon07r lemon07r commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Runner tuning principle

macOS is the fixed floor: the smallest Blacksmith macOS label is 6vcpu (~160s builds), so the release can never finish faster than ~2.7 min. Every other target is sized to land at that baseline — not faster (wasted vcpus), not slower (gates the release).

Target 2vcpu baseline (v1.0.1) New size Est.
linux-gnu 554s 8vcpu ~172s (measured via docker compile job)
musl 633s (incl. ~90s fixed setup) 16vcpu ~3.5 min
aarch64-linux 856s 16vcpu-arm ~3.2 min
windows-msvc 749s 16vcpu ~2.8 min
macOS x2 ~160s @6vcpu unchanged ~2.7 min (anchor)

The 16vcpu labels were smoke-verified (all three picked up and ran). Sizing rationale with the measured baselines lives in a comment above the matrix.

Docker repair-retry

The rocm publish has now failed 3 times with the identical error: failed commit on ref "layer-sha256:4c8a4cb..." unexpected commit digest. Same layer ref across the v1 cache-key, the fresh docker-v2 key, and a solo rerun — so it is a corrupted base-image layer (rocm/dev-ubuntu-24.04:6.4.4-complete) sitting on the persistent builder's disk, not a concurrency race and not fixed by rotating the cache-key.

build-push-action now runs with continue-on-error, and on failure the job prunes the builder (docker buildx prune -af, evicting the corrupt layer) and retries once. Healthy builds never pay for this; sick ones self-heal instead of failing the release.

This PR is also the vehicle for backfilling 1.0.1-rocm: after merge I will re-dispatch the workflow for tag v1.0.1, which exercises the repair path immediately.

Validation

  • actionlint clean; label set in .github/actionlint.yaml pruned to exactly the used labels.
  • Retry logic only restructures when the action runs, not what it publishes (same tags, same context).

View with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is enabled.


Summary by cubic

Sizes release runners to the macOS baseline and adds a self-healing Docker publish retry. Other targets now finish near the ~160s macOS floor, and Docker publishes recover from corrupted base-image layers instead of failing the release.

  • Runner sizing: gnu to blacksmith-8vcpu-ubuntu-2404 (172s measured), musl to blacksmith-16vcpu-ubuntu-2404, aarch64 to blacksmith-16vcpu-ubuntu-2404-arm, windows to blacksmith-16vcpu-windows-2025; macOS unchanged. This removes slow-target gating without overprovisioning faster targets. actionlint.yaml labels pruned to match.
  • Docker repair-retry: the first useblacksmith/build-push-action@v2 run uses continue-on-error; on failure we run docker buildx prune -af and retry once. This fixes the recurring “unexpected commit digest” caused by a corrupt layer in rocm/dev-ubuntu-24.04:6.4.4-complete. Healthy builds keep cache and do not pay the prune cost.
  • Rollout: after merge, re-dispatch the workflow for tag v1.0.1 to backfill 1.0.1-rocm. No tag format or artifact changes.

Written for commit 3c99e4c. Summary will update on new commits.

Review in cubic

Every target now aims for the ~160s macOS floor (smallest mac label is
6vcpu): gnu to 8vcpu (172s measured), musl/arm/windows to 16vcpu
(estimated ~2.8-3.5 min, closest achievable). 16vcpu labels verified via
smoke run. Docker package jobs gain a repair-retry: a corrupted base
layer on the builder's sticky disk (three identical rocm digest-commit
failures) is evicted via buildx prune and rebuilt once; healthy builds
are untouched. actionlint labels pruned to the used set.
Copilot AI lite review requested due to automatic review settings August 19, 2026 12:03
@coderabbitai

coderabbitai Bot commented Aug 19, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@lemon07r, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 22 minutes

Limit details: You’ve used all 3 included reviews currently available.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: bdbe036a-bde2-44e3-b410-5e43e1f63525

📥 Commits

Reviewing files that changed from the base of the PR and between e4f3398 and 3c99e4c.

📒 Files selected for processing (3)
  • .github/actionlint.yaml
  • .github/workflows/docker.yml
  • .github/workflows/release.yml

Comment @coderabbitai help to get the list of available commands.

@lemon07r
lemon07r merged commit f3da185 into master Aug 19, 2026
4 checks passed
@lemon07r
lemon07r deleted the ci/balance-to-macos-baseline branch August 19, 2026 12:05

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the release and Docker publishing GitHub Actions workflows to (1) align runner sizing across targets to a macOS baseline wall-clock and (2) add a one-time self-healing retry path for Docker image publishing when the Buildx cache is corrupted.

Changes:

  • Rebalanced release.yml matrix runner labels to target ~macOS baseline duration (macOS is the fixed floor).
  • Added a failure-only “prune + retry once” repair path around build-push-action in docker.yml to recover from persistent builder cache corruption.
  • Pruned .github/actionlint.yaml runner labels to match the labels actually used by workflows.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
.github/workflows/release.yml Updates build matrix runner sizes and clarifies the sizing rationale in comments.
.github/workflows/docker.yml Adds a continue-on-error first publish step, prunes Buildx cache on failure, and retries once.
.github/actionlint.yaml Removes unused Blacksmith labels and adds the newly used 16vcpu labels to keep actionlint aligned.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants