rate-limit preflight: a failed issue create is swallowed, so the spike notice becomes a dangling `#?`

[tend-agent]

When gh issue create fails inside run_issue_create_and_reconcile, the failure is swallowed and the rate-limit preflight emits a literal #? where the issue number belongs. The notice that #874 exists to provide is never filed, and nothing says so.

Mechanism

Bash does not apply set -e inside a command substitution unless inherit_errexit is set, and neither rate-limit-preflight.sh nor lib/run-issue.sh sets it. So a failing gh issue create at run-issue.sh:119 does not abort the subshell. Execution continues to the reconcile, which finds nothing, and the function's last command — printf '%s' "$keep" at run-issue.sh:136 — succeeds. The substitution's exit status is that printf's, so the assignment at rate-limit-preflight.sh:182 sees success and set -e never fires.

PAUSE is left empty, and the next line renders ${PAUSE:-?}:

::error::Rate limit: bot created 47 items today, above the ceiling of 30 ... Refused runs are listed in #?; closing it doubles the ceiling.

Reduced to the shape of the real call:

set -eo pipefail
create_and_reconcile() {
  false                      # stands in for a failing `gh issue create`
  local open keep
  open=""                    # the post-create list finds nothing
  keep=${open%%$'\n'*}
  printf '%s' "$keep"        # last command succeeds -> substitution exits 0
}
PAUSE=$(printf 'body\n' | create_and_reconcile)
echo "reached line after assignment; PAUSE='${PAUSE}'"

$ bash repro.sh
reached line after assignment; PAUSE=''
script exit=0

$ bash -c 'shopt -s inherit_errexit; source repro.sh'
script exit=1

Why it matters here

The step still exits 1, so the run is still refused — this is not a case of the limit failing open. What is lost is the notice. #874's premise is that a bare ::error:: lands on a job nobody opens, which is why the spike trip files an issue at all; when the create fails, the run lands back in exactly that state, plus a dangling #? that reads as corruption. The maintainer has no issue to close, so the documented recovery — closing it to double the ceiling — is unavailable, and the bot stays halted for the rest of the UTC day with no route back.

A failing create is not hypothetical in this path: the preflight runs precisely when the bot is at abnormal volume, which is also when secondary-rate-limit and abuse-detection responses to gh issue create are most likely.

report-failure.sh reaches the same function through a pipeline rather than a command substitution (report-failure.sh:46) and discards stdout, so it is unaffected.

Filed rather than fixed

There is a plausible fix — have the function return non-zero when the create fails and let the caller distinguish "filed" from "could not file" — but it rewrites the body of run_issue_create_and_reconcile, which is exactly what open PR #836 is rewriting (settle-then-list reconcile → primary-key probe). A second PR touching the same function would conflict head-on and force one of them to be redone. Better to land #836 and then fix this on top of whatever that function ends up being.

Worth deciding as part of that: whether shopt -s inherit_errexit belongs at the top of these scripts generally, since the same swallow applies to every other $(...) in them, or whether this one call site should check explicitly. The blanket option is the smaller diff but changes the failure behaviour of substitutions that currently tolerate a non-zero intermediate.

Found by the nightly sweep while resolving merge conflicts on #836; noted there too.

[tend-agent]

Reproduced against the real script, and the deferral to #836 holds — but for a different reason than "the fix would conflict". #836 does not fix this. Its new url=$(gh issue create ...) carries the comment "Keep the create in its own assignment so set -e still sees its exit status", which is true at report-failure.sh's call site (a pipeline, where errexit is inherited) and false at this one (a command substitution, where it is not). Running the preflight with a gh that fails on issue create prints the same #? on main and on #836's branch.

Two things I can add to the open question at the bottom of the issue.

shopt -s inherit_errexit is the wrong lever here. It does remove the #? — but by aborting the script at the failed create, before the ::error:: is ever printed. The step then exits 1 carrying nothing but gh's own stderr line, which is exactly the invisible-failure state #874 exists to end. Verified on both versions.

The fix doesn't have to touch run_issue_create_and_reconcile at all. An empty PAUSE is the "could not file" signal, so branching on it at the call site — one if around the final ::error::, rate-limit-preflight.sh only — turns the dangling #? into an honest annotation. That file isn't in #836's diff, so the shell change conflicts with nothing; the test harness is the only overlap, since asserting it needs the create) arm of FAKE_GH_RATE_LIMIT that #836 rewrites.

And it wants to land after #836, not before. On main, PAUSE is empty whenever the post-create list lags, so the call-site check can't tell a failed create from a create the index hasn't caught up with — it would report "could not be filed" for an issue that was in fact filed, which is the same class of lie in the other direction. #836 makes the check exact: the number comes off the create URL, so empty means the create failed and nothing else.

Reproduction and evidence

Harness: the FAKE_GH_RATE_LIMIT/FAKE_DATE/FAKE_SLEEP shape from generator/tests/test_shared_steps.py, with issue create exiting 1 (gh: HTTP 403 secondary rate limit) and FAKE_TODAY_POSTS=16 against a base limit of 15 to trip the spike gate.

main (5b30d22) — the reported shape:

gh: HTTP 403 secondary rate limit
::error::Rate limit: bot created 16 items today, above the ceiling of 15 ... Refused runs are listed in #?; closing it doubles the ceiling.
exit=1

Same harness, lib/run-issue.sh swapped for #836's (origin/hourly/review-30962483562) — byte-identical ::error::, still #?.

Why, in isolation:

set -eo pipefail
f() { local url; url=$(false); printf '%s' "${url##*/}"; }
X=$(printf body | f);  echo "substitution: X='$X' status=$?"   # -> X='' status=0
printf body | f >/dev/null; echo "pipeline: $?"               # -> never reached; script exits 1

Under #836, mine=${url##*/} is empty, seq $((mine - 1)) -1 $((mine - 10)) starts at -1 so the probe loop breaks on its first [ "$n" -gt 0 ], and the function returns the empty $mine with status 0.

With shopt -s inherit_errexit added after set -eo pipefail, both versions produce no ::error:: line at all — just gh's stderr and exit=1.

Call-site branch I tested (prototype, not committed):

OVER="bot created ${TODAY_POSTS} items today, above the ceiling of ${CEILING} (base limit ${SPIKE_LIMIT}, ${APPROVALS} approval(s), baseline ${PAST_POSTS} over past 6 days)"
if [ -n "$PAUSE" ]; then
  echo "::error::Rate limit: ${OVER}. Refused runs are listed in #${PAUSE}; closing it doubles the ceiling."
else
  echo "::error::Rate limit: ${OVER}. The pause issue could not be filed, so there is nothing to close and the ceiling cannot be lifted before the UTC day rolls over; see the gh error above."
fi

Four combinations run: create-fails and create-succeeds, each against main and #836. All correct except one — main + create-succeeds + lagging post-create list, which takes the "could not be filed" branch for an issue that exists. That case disappears under #836.