Skip to content

ci: harden Kurtosis install against API rate limits - #1175

Merged
mergify[bot] merged 1 commit into
sigp:unstablefrom
shane-moore:ci/harden-kurtosis-install
Jul 28, 2026
Merged

ci: harden Kurtosis install against API rate limits#1175
mergify[bot] merged 1 commit into
sigp:unstablefrom
shane-moore:ci/harden-kurtosis-install

Conversation

@shane-moore

@shane-moore shane-moore commented Jul 27, 2026

Copy link
Copy Markdown
Member

Problem, Evidence, and Context

  • run-local-testnet fails intermittently in Install Kurtosis, unrelated to the PR under test. Six occurrences since 2026-06-10: runs 30252125901, 29434838333, 29431334751, 29353107664, 29017436052, 27312016351. That is 5% of the 117 workflow runs in that window and 27% of its failures.
  • Every one shows the same signature: Installing Kurtosis version: with an empty value, a 9-byte download, then gzip: stdin: not in gzip format.
  • The version came from an unauthenticated api.github.com call, capped at 60 requests/hour per IP. The shared runner egress IPs exhaust it, so the response carries no tag_name.
  • Nothing stopped the empty value propagating. The step ran without pipefail, so grep finding nothing still exited 0 through sed; curl without -f then wrote the error page to kurtosis.tar.gz. tar was the first command to object, which is why the failure reads as a corrupt archive rather than a failed version lookup.
  • Most recent case: commit b7a3f429 on fix(ssv_types): cap Role::Proposer QBFT rounds at 2 #1173 failed at 09:00 and 09:07, then passed unchanged on re-run. Same tree, opposite results.

Change Overview

  • Resolve the tag from the github.com /releases/latest redirect instead of api.github.com. This is the host the tarball already downloads from, and it is not behind the REST API's unauthenticated quota.
  • Add set -euo pipefail, an explicit guard rejecting an empty or unparsed version, -f on both transfers, and --retry 3 --retry-all-errors.
  • Unchanged: the step still tracks the latest release rather than pinning, and the rest of the workflow is untouched. No token or added permissions: is required, and no new tooling (gh, jq) is assumed on the warp runner image.

Risks, Trade-offs, and Mitigations

  • Blast radius is one CI step. No production code, and nothing outside this workflow.
  • The tag is now parsed from a redirect URL rather than a JSON field. If that redirect shape ever changes, the guard fails the step with a named error instead of passing a bad value downstream.
  • The two endpoints were measured directly rather than assumed: api.github.com returns x-ratelimit-limit: 60, while github.com/.../releases/latest returns a 302 carrying the tag in location and no x-ratelimit-* headers at all. The new path therefore does not consume the REST quota. Retries and fail-fast cover any residual transient failure.

Validation

  • Ran the exact script locally: resolves 1.20.0, downloads kurtosis-cli_1.20.0_linux_amd64.tar.gz, extracts a valid ELF 64-bit LSB executable, x86-64 binary.
  • Confirmed the resolved version matches what api.github.com returns and what the last passing CI run installed (1.20.0).
  • Guard verified against both failure modes: empty string and an unparsed URL both exit 1; 1.20.0 passes.
  • Confirmed -f now catches a bad asset URL at the download with a real HTTP 404 rather than feeding 9 bytes to tar.
  • shellcheck clean on the extracted script. actionlint reports only the two pre-existing warp-ubuntu-latest-x64-16x runner-label warnings, which are present on the unmodified base file.
  • This workflow runs on pull_request, so the updated step executes on this PR itself. On run 30295263760, Install Kurtosis passed with the new code, and the job went on to complete Start Local Testnet with Assertoor, which shells out to kurtosis clean -a and kurtosis run and therefore exercises the installed binary rather than just the download.

Rollback

  • Revert the commit. CI-only, no config, data, or runtime impact.

Additional Info / Next Steps

  • The other failure modes in this workflow (Start Local Testnet with Assertoor, Return Assertoor Test Result) are separate and not addressed here.
  • Note for after merge: GitHub freezes the workflow file at run creation, so re-running an already-failed run will not pick this up. A new run has to be triggered.

Resolve the latest Kurtosis tag from the github.com release redirect
instead of api.github.com. The REST API allows 60 unauthenticated
requests per hour per IP, and the shared runner egress IPs exhaust it,
which returned no tag_name and left KURTOSIS_VERSION empty.

Nothing stopped the empty value propagating: the step ran without
pipefail so the grep/sed pipeline exited 0, and curl without -f wrote
the 9-byte error page to kurtosis.tar.gz. The job then failed in tar
with "not in gzip format", pointing at the archive rather than the
version lookup.

Add set -euo pipefail, an explicit version guard, -f on both transfers,
and retries, so a future lookup failure aborts where it happens with a
message that names the cause.
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.70%. Comparing base (eb138a6) to head (3afc662).

Additional details and impacted files
@@            Coverage Diff            @@
##           unstable    #1175   +/-   ##
=========================================
  Coverage     62.70%   62.70%           
=========================================
  Files           161      161           
  Lines         27000    27000           
=========================================
  Hits          16930    16930           
  Misses        10070    10070           
Flag Coverage Δ
rust 62.70% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@shane-moore

Copy link
Copy Markdown
Member Author

@claude, please take a look

@claude-code-actions-sigp

claude-code-actions-sigp Bot commented Jul 27, 2026

Copy link
Copy Markdown

Claude finished @shane-moore's task in 58s —— View job


Review: PR #1175 — harden Kurtosis install

  • Read the changed workflow file and diff vs origin/unstable
  • Analyze the hardened Kurtosis install step
  • Post review feedback

The diagnosis is precise and the fix matches it. The root cause (unauthenticated api.github.com 60/hr quota → empty tag_name) and why it surfaced as a gzip error rather than a version-lookup failure (no pipefail, no -f) are both correct. The fix addresses each link in that chain.

What holds up:

  • set -euo pipefail closes the silent-failure gap. Previously grep finding nothing exited 0 and propagated an empty value; now the pipeline fails loudly.
  • Redirect-based tag resolution is sound. -fsSLI -o /dev/null -w '%{url_effective}' follows /releases/latest/releases/tag/<ver>, and sed 's#.*/tag/##' extracts the tag. -w emits no trailing newline, so the value is clean for interpolation into the download URL.
  • The guard is well-chosen. ''|*/* catches both failure modes: an empty result, and the case where no redirect occurred (the effective URL would still end in /releases/latest, which contains / and is rejected). The ::error:: annotation makes the failure legible in the run summary.
  • -f on both transfers + --retry 3 --retry-all-errors ensures a bad asset URL fails at download with a real HTTP error instead of writing an error page into kurtosis.tar.gz.

Minor observations (non-blocking):

  • sed's .* is greedy, but there is only one /tag/ in the URL, so extraction is unambiguous. No issue.
  • --retry-all-errors requires curl ≥ 7.71.0; that's comfortably satisfied on current runner images.
  • The step still tracks latest rather than pinning. That's a deliberate choice per the PR description, and it keeps this in line with the existing behavior — reasonable to keep out of scope here. If Kurtosis ever ships a breaking CLI release, the fail-fast guard will at least surface it clearly.

The change is CI-only, correctly scoped to a single step, and the validation (local run resolving 1.20.0, shellcheck clean, guard tested against both failure modes) is thorough. No correctness or security concerns.

LGTM.
· branch ci/harden-kurtosis-install

@mergify

mergify Bot commented Jul 28, 2026

Copy link
Copy Markdown

Queued — the merge queue status continues in this comment ↓.

@mergify

mergify Bot commented Jul 28, 2026

Copy link
Copy Markdown

Merge Queue Status

This pull request spent 29 minutes 7 seconds in the queue, including 27 minutes 8 seconds running CI.

Required conditions to merge
  • check-success=cli-reference-check
  • check-success=run-local-testnet
  • check-success=test-suite-success
  • github-review-approved [🛡 GitHub repository ruleset rule unstable and release-v*] (documentation)
  • any of [🛡 GitHub repository ruleset rule unstable and release-v*]:
    • check-success = license/cla
    • check-neutral = license/cla
    • check-skipped = license/cla

@mergify
mergify Bot merged commit c168c6e into sigp:unstable Jul 28, 2026
26 checks passed
@mergify mergify Bot removed the queued label Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants