Skip to content

fix(running-in-ci): don't let a failed status probe read as no incident - #913

Open
tend-agent wants to merge 2 commits into
mainfrom
hourly/review-31302588340
Open

fix(running-in-ci): don't let a failed status probe read as no incident#913
tend-agent wants to merge 2 commits into
mainfrom
hourly/review-31302588340

Conversation

@tend-agent

Copy link
Copy Markdown
Collaborator

The upstream-incident probe in running-in-ci fails open: when the status endpoint doesn't answer with JSON, the recipe prints nothing, and "nothing" is the same output it prints when there are genuinely no open incidents. The agent reads the empty result as clear and proceeds to file the workaround PR that the check exists to prevent.

Observed live this run. https://www.githubstatus.com/api/v2/incidents/unresolved.json currently answers a GitHub Actions runner with HTTP 405 and a 2,519-byte Human Verification HTML page, reproducibly (3/3 attempts). The documented recipe on that body:

$ curl -s '…/incidents/unresolved.json' | jq '.incidents[] | {created_at, name, impact, components: [.components[].name]}'
jq: parse error: Invalid numeric literal at line 1, column 10
$ echo "stdout=[$OUT]"
stdout=[]

Empty stdout, and the paragraph right below the recipe keys off exactly that: "If the response is non-empty and the components/timing match the symptom … exit without a PR." Empty → no incident → file the PR. Egress is fine generally (pypi.org returns 200 from the same shell), so this is the status endpoint's edge challenging the runner, not a sandbox block.

The fix. Fetch first, parse second, and say so when the fetch fails. A bare curl -f … | jq … || echo does not work — the pipeline's exit status is jq's, and jq exits 0 on empty stdin, so the || branch never fires. Verified both halves against the live endpoint and against a reachable JSON endpoint as a positive control.

One added sentence draws the distinction the recipe can't draw on its own: a probe that errored is unknown, not clear, and unknown resolves the same way an open incident does — record the symptom, skip the workaround PR.

Gate assessment

  • Evidence level / occurrences: the trigger (endpoint unreachable from CI) is occurrence 1 — the evidence gist records 167 prior probes across ~45 legs, every one of them a clean 0 incidents, so this is the first failure. The defect isn't occurrence-counted, though: that the recipe cannot distinguish a failed fetch from a clean result is provable by reading it, and was demonstrated end-to-end above rather than inferred.
  • Structural, not stochastic: no model decision point. Any non-JSON body produces empty stdout every time.
  • Change type: targeted fix — recipe hardening plus one clarifying sentence, no new section.
  • Passes both gates: yes. The failure direction is what tips it — this fails open, toward filing a workaround for a transient, which is the specific outcome the surrounding guidance was written to stop. list-recent-runs.sh already treats the same class of silent-empty as worth failing loud ("refusing to report an empty run list that would read as a false all-clear"); this brings the incident probe onto that footing.

Evidence: https://gist.github.com/dca23a6e6a0d8cae2665944ba31676fb

The upstream-incident recipe piped a possibly-non-JSON body into jq, so a
challenge page or any non-200 produced empty stdout — indistinguishable from
a clean result, and read by the surrounding guidance as licence to file the
workaround PR. Fetch first, parse second, and name a failed probe as unknown.
@tend-agent tend-agent added the claude-behavior Behavioral issues found by review-reviewers label Aug 9, 2026

@tend-agent tend-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix is right about the mechanism — jq swallowing a non-JSON body into empty stdout is indistinguishable from a clean probe, and capturing curl's status separately is the only way to keep the pipeline's exit status from being jq's. Verified the exit codes on this runner: jq returns 5 on an HTML body, 4 on empty stdin, 0 on {"incidents":[]}.

One gap in the same failure mode: the guard keys on the HTTP status alone, so it only closes the half of the challenge that arrives as a non-2xx. The comment right above it names the general case — "an HTML challenge page instead of JSON" — and an edge interstitial served with 200 still lands in $INCIDENTS, still parse-errors to stderr, and still leaves stdout empty. That's the exact reading the PR sets out to prevent, reached by a different status code. A jq -e . validity check on the captured body closes the class rather than the one observed instance; suggestion inline.

(The endpoint answers this runner normally right now — http=200 with valid JSON — so the 405 looks edge- or window-specific, which is an argument for guarding the shape of the body rather than the code it came with.)

Comment thread plugins/tend-ci-runner/skills/running-in-ci/SKILL.md Outdated
…TP status

An edge challenge served with 200 passed `curl -f`, parse-errored in jq, and
left stdout empty — the same reading as a clean probe. Add a `jq -e` validity
check on the captured body so the guard closes the class rather than the one
observed non-2xx instance.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claude-behavior Behavioral issues found by review-reviewers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant