Skip to content

Commit 8dd0d6e

Browse files
authored
fix(release): a shared storage quota no longer takes down every nightly (#50)
Every scheduled release since 2026-08-11 failed with "Failed to CreateArtifact: Artifact storage quota has been hit". Actions storage is an org-wide quota billed by GB-hour, and this org sits at 2.04 GB against the 2.00 GB its plan includes, so all artifact uploads are blocked account-wide. Two release jobs treated that upload as fatal, and losing them skipped the whole build matrix, the npm publish, and the release itself. Connect was already designed to be optional here, so the tracing config handoff now degrades the way the surrounding code says it should: the upload and the three downloads are best-effort, and the loader tells a missing artifact apart from a Connect-less setup, warning on the former so an untraced production build does not look identical to a healthy one. The config keeps travelling as an artifact rather than a job output because it carries a token the loader has to mask. Separately, the workflow had no concurrency group at all. When Blacksmith had no Ubuntu capacity from 08-03 to 08-06, 23 nightly runs stacked up instead of superseding each other and sat about 480 VM-hours waiting for runners that never arrived, each dying at GitHub's 24h queue cap. Note that timeout-minutes does not help here, since it only counts execution time. Nightlies now share one group; tag releases and manual dispatches key off run_id so they stay unique and are never cancelled. Also drops resource-monitor artifacts from 7-day to 1-day retention, the only release artifact held above the minimum, since it is consumed within the same run. Verified with actionlint: 13 findings before and after, all pre-existing (custom Blacksmith runner labels, run_started_at, SC2129 style). Model: Claude Opus 5. Harness: Claude Code.
1 parent 5ce9308 commit 8dd0d6e

1 file changed

Lines changed: 60 additions & 10 deletions

File tree

.github/workflows/release.yml

Lines changed: 60 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,17 @@ on:
2626
required: false
2727
type: string
2828

29+
# A self-hosted capacity outage does not cancel queued jobs; GitHub kills them
30+
# at its own 24h queue cap. With no group here, the nightlies from 2026-08-03
31+
# to 08-06 stacked up 23 deep and sat ~480 VM-hours waiting for runners that
32+
# never arrived. Nightlies now share one group so a fresh run supersedes a
33+
# stuck predecessor. Tag releases and manual dispatches key off run_id, so they
34+
# are always unique and never cancel one another: losing a real release to a
35+
# scheduling race is far worse than paying for a duplicate build.
36+
concurrency:
37+
group: release-${{ github.event_name == 'schedule' && 'nightly' || github.run_id }}
38+
cancel-in-progress: ${{ github.event_name == 'schedule' }}
39+
2940
permissions:
3041
contents: read
3142
id-token: none
@@ -245,8 +256,16 @@ jobs:
245256
--github-output \
246257
--github-env-file "$RUNNER_TEMP/relay-client-tracing.env"
247258
259+
# Non-fatal on purpose, matching the same reasoning as the upload in
260+
# ci.yml. Actions storage is an org-wide quota that anything in the
261+
# account can exhaust, and when it is full every upload in every repo
262+
# fails. Connect is already optional here and the loader below degrades
263+
# to a build without relay tracing, so a storage condition must not take
264+
# down the release matrix — which is exactly what killed every nightly
265+
# from 2026-08-11 onward.
248266
- name: Upload relay client tracing config
249267
uses: actions/upload-artifact@v7
268+
continue-on-error: true
250269
with:
251270
name: relay-client-tracing-config
252271
path: ${{ runner.temp }}/relay-client-tracing.env
@@ -441,6 +460,7 @@ jobs:
441460

442461
- name: Download relay client tracing config
443462
uses: actions/download-artifact@v8
463+
continue-on-error: true
444464
with:
445465
name: relay-client-tracing-config
446466
path: ${{ runner.temp }}/relay-client-tracing
@@ -449,9 +469,17 @@ jobs:
449469
shell: bash
450470
run: |
451471
config_path="$RUNNER_TEMP/relay-client-tracing/relay-client-tracing.env"
452-
# The artifact is always uploaded so the download stays uniform, but
453-
# it carries no variables when Connect is not configured.
454-
if ! grep -q '^T3CODE_RELAY_CLIENT_OTLP_TRACES_' "$config_path" 2>/dev/null; then
472+
# The upload is best-effort against a shared storage quota, so the
473+
# file can be missing entirely. Building without relay tracing is the
474+
# correct way to degrade, but it is worth surfacing: a silently
475+
# untraced production build should not look identical to a healthy one.
476+
if [[ ! -f "$config_path" ]]; then
477+
echo "::warning::Relay client tracing config artifact unavailable; building without Connect tracing."
478+
exit 0
479+
fi
480+
# Present but carrying no variables when Connect is not configured at
481+
# all, which is a supported setup rather than a problem.
482+
if ! grep -q '^T3CODE_RELAY_CLIENT_OTLP_TRACES_' "$config_path"; then
455483
echo "No relay client tracing config; continuing without it." >&2
456484
exit 0
457485
fi
@@ -736,7 +764,11 @@ jobs:
736764
name: resource-monitor-${{ matrix.resource_key }}
737765
path: resource-monitor-publish/${{ matrix.resource_key }}/*
738766
if-no-files-found: error
739-
retention-days: 7
767+
# Consumed by the CLI publish job in this same run, so it only has to
768+
# outlive the matrix. The old 7-day window was the one release
769+
# artifact held above the minimum, and Actions storage is billed by
770+
# GB-hour against a quota this repo already sits on top of.
771+
retention-days: 1
740772

741773
publish_cli:
742774
name: Publish CLI to npm
@@ -778,6 +810,7 @@ jobs:
778810
779811
- name: Download relay client tracing config
780812
uses: actions/download-artifact@v8
813+
continue-on-error: true
781814
with:
782815
name: relay-client-tracing-config
783816
path: ${{ runner.temp }}/relay-client-tracing
@@ -786,9 +819,17 @@ jobs:
786819
shell: bash
787820
run: |
788821
config_path="$RUNNER_TEMP/relay-client-tracing/relay-client-tracing.env"
789-
# The artifact is always uploaded so the download stays uniform, but
790-
# it carries no variables when Connect is not configured.
791-
if ! grep -q '^T3CODE_RELAY_CLIENT_OTLP_TRACES_' "$config_path" 2>/dev/null; then
822+
# The upload is best-effort against a shared storage quota, so the
823+
# file can be missing entirely. Building without relay tracing is the
824+
# correct way to degrade, but it is worth surfacing: a silently
825+
# untraced production build should not look identical to a healthy one.
826+
if [[ ! -f "$config_path" ]]; then
827+
echo "::warning::Relay client tracing config artifact unavailable; building without Connect tracing."
828+
exit 0
829+
fi
830+
# Present but carrying no variables when Connect is not configured at
831+
# all, which is a supported setup rather than a problem.
832+
if ! grep -q '^T3CODE_RELAY_CLIENT_OTLP_TRACES_' "$config_path"; then
792833
echo "No relay client tracing config; continuing without it." >&2
793834
exit 0
794835
fi
@@ -1060,6 +1101,7 @@ jobs:
10601101
10611102
- name: Download relay client tracing config
10621103
uses: actions/download-artifact@v8
1104+
continue-on-error: true
10631105
with:
10641106
name: relay-client-tracing-config
10651107
path: ${{ runner.temp }}/relay-client-tracing
@@ -1068,9 +1110,17 @@ jobs:
10681110
shell: bash
10691111
run: |
10701112
config_path="$RUNNER_TEMP/relay-client-tracing/relay-client-tracing.env"
1071-
# The artifact is always uploaded so the download stays uniform, but
1072-
# it carries no variables when Connect is not configured.
1073-
if ! grep -q '^T3CODE_RELAY_CLIENT_OTLP_TRACES_' "$config_path" 2>/dev/null; then
1113+
# The upload is best-effort against a shared storage quota, so the
1114+
# file can be missing entirely. Building without relay tracing is the
1115+
# correct way to degrade, but it is worth surfacing: a silently
1116+
# untraced production build should not look identical to a healthy one.
1117+
if [[ ! -f "$config_path" ]]; then
1118+
echo "::warning::Relay client tracing config artifact unavailable; building without Connect tracing."
1119+
exit 0
1120+
fi
1121+
# Present but carrying no variables when Connect is not configured at
1122+
# all, which is a supported setup rather than a problem.
1123+
if ! grep -q '^T3CODE_RELAY_CLIENT_OTLP_TRACES_' "$config_path"; then
10741124
echo "No relay client tracing config; continuing without it." >&2
10751125
exit 0
10761126
fi

0 commit comments

Comments
 (0)