Skip to content

fix(report-failure): dedup outage comments per run across matrix legs - #809

Open
tend-agent wants to merge 7 commits into
mainfrom
hourly/review-30783291243
Open

fix(report-failure): dedup outage comments per run across matrix legs#809
tend-agent wants to merge 7 commits into
mainfrom
hourly/review-30783291243

Conversation

@tend-agent

Copy link
Copy Markdown
Collaborator

Problem

When a matrix workflow fails during a bot outage, the tend-outage tracker issue is flooded with one near-identical comment per matrix leg, all citing the same run.

review-reviewers has a 5-leg matrix. During an outage this window (claude -p exiting 1 uniformly, self-resolved by 03:57Z), two failed review-reviewers runs each posted ~5–6 comments to #808:

  • run 30776520686 → 5 comments, all linking .../runs/30776520686
  • run 30779959756 → 6 comments, all linking .../runs/30779959756

Each comment is a one-row table differing only in a jitter-spread timestamp — the same run recorded 5–6 times.

Root cause

shared/steps/report-failure.sh is invoked once per matrix leg, every leg sharing one GITHUB_RUN_ID. The script already handles the concurrent-leg race on the create path — jittered backoff (#586) plus a self-heal reconcile that closes duplicate issues (#744) — but the append path had no dedup: every leg unconditionally gh issue comments its own row.

So the create-create race (duplicate issues) was solved; the comment-append race (duplicate comments) was not. This is the same class of concurrent-matrix-leg noise the maintainer has fixed repeatedly (#586, #744, and #560 which batches enrichment into one comment per issue) — this closes the remaining gap.

Fix

Symmetric to the existing issue reconcile:

  1. Guard — before appending, skip if this run's URL is already recorded on the issue (body or an existing comment). Collapses most of the flood, since jitter (0–30s) already spreads the legs.
  2. Reconcile — the check-then-act still races across jittered legs, so after posting, keep the earliest comment citing this run and delete later duplicates. Convergent: every racing leg sorts identically and computes the same keeper, so deleting an already-deleted comment is a harmless 404.

Net effect: one row per run, regardless of matrix width. Non-matrix workflows (single leg) are unaffected — the guard finds nothing, posts once, reconcile is a no-op.

Comments deleted are the bot's own auto-generated outage rows; no human content is touched.

Gate assessment

  • Class: structural — a shell script with no per-run comment dedup. Replayed 10/10 times, every matrix-workflow failure floods. Not a stochastic model lapse; confirmed by reading the script source and the 11 duplicate comments on Bot temporarily unavailable #808.
  • Evidence: High (structural). 2 distinct failed review-reviewers matrix runs this window, both exhibiting the identical flood (~11 duplicate-run comments total), plus the defect is guaranteed to recur on any future matrix-workflow outage.
  • Magnitude: targeted fix to one shared script (add a guard + a reconcile mirroring the adjacent issue reconcile) — Normal bar, met.
  • Passes both gates. The transient outage that triggered the failures is not itself a code fix (it self-resolved by 03:57Z, tracked by Bot temporarily unavailable #808); this fixes the durable noise the outage exposed.
Window & evidence

A matrix workflow (e.g. review-reviewers, 5 legs) invokes report-failure.sh
once per leg on an outage, every leg sharing one GITHUB_RUN_ID. The existing
reconcile handles the create-create race (duplicate *issues*) but nothing
dedups the append path, so each leg posted its own near-identical row —
flooding the tend-outage issue with 5-6 comments all citing the same run.

Guard the comment on the run not already being recorded (body or existing
comment), then reconcile symmetrically to the issue path: keep the earliest
comment citing this run, delete later duplicates. Convergent across
concurrently-jittered legs.
@tend-agent tend-agent added the claude-behavior Behavioral issues found by review-reviewers label Aug 3, 2026

@tend-agent tend-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dedup logic is sound and the convergence argument holds (stable sort_by over an id-ascending API response makes every leg pick the same keeper). One robustness gap on the delete path: the reconcile selects comments by a bare contains("${RUN_URL}") substring, with no author or row-format scoping. Any comment that merely contains the run URL — a human quoting it while investigating, or a nightly-enrichment comment (#560) that lists this run among others — would be selected and, if it sorts after the earliest row, deleted. That contradicts the PR's "no human content is touched" invariant, since the code doesn't actually restrict deletion to the auto-generated rows.

Matching the full markdown anchor [workflow run](${RUN_URL}) instead scopes both the guard and the delete to the generated rows only, and as a bonus removes a latent prefix-collision false positive (a bare-URL contains also matches a longer run id that has this one as a prefix). The guard and the reconcile must stay consistent, so both suggestions below apply together.

Comment thread shared/steps/report-failure.sh Outdated
Comment thread shared/steps/report-failure.sh Outdated
@tend-agent

Copy link
Copy Markdown
Collaborator Author

New evidence, same defect, roughly 7x the volume. The 2026-08-04 outage (Claude weekly limit, 429 rate_limit, from ~08:51Z until the 00:00Z reset) left #831 with 77 comment rows across 17 distinct runs — 15 review-reviewers runs contributing exactly 5 rows each, one per matrix leg, plus 2 single-leg tend-triage rows. This PR's original evidence was ~11 rows from 2 runs.

Each of those 75 duplicate-leg rows also fired an issue_comment event, so the flood produced 66 tend-mention runs on top of the comments (all correctly gate-skipped, but they are 66 queued jobs). With this PR merged the same outage would have recorded 17 rows instead of 77 — one per run, which is the readable timeline the issue is for.

Evidence log: https://gist.github.com/e08f6e62d6478163cb425a75648eb7e4

@tend-agent

Copy link
Copy Markdown
Collaborator Author

Coordination note from #836, which is now touching the neighbouring branch of this same script.

Review there caught that #836's reconcile carries its row onto the keeper unguarded, so a same-matrix race would post a row duplicating the keeper's seed row — the same flood this PR removes from the EXISTING branch, arriving by a path that bypasses this PR's guard. Fixed in 1c41d39 with the identical four-line check (gh issue view --json body,comments piped to grep -qF "[workflow run](${RUN_URL})").

That means the anchor check now exists twice in the file: once here on the append path, once on the create path. Leaving both copies rather than pre-factoring a helper — the two diffs are textually disjoint as they stand and a helper introduced on either branch would conflict with the other for no benefit until one lands. Whichever of the two merges second should fold them into one helper.

max-sixty pushed a commit that referenced this pull request Aug 5, 2026
…#823)

## Problem

The `Trigger` column of a `tend-outage` row is the only pointer back to
the work a failed run stranded. It goes blank for the one trigger where
that pointer matters most, and prints `#null` when a field is missing.

**`repository_dispatch` is unhandled.** `tend-mention` relays review
events through a secretless job that re-posts them as a
`repository_dispatch`, so the handle job runs on that event and the PR
number arrives as `client_payload.pr` rather than in a `pull_request`
object. The `if`/`elif` chain has no branch for it, so every failure on
the relay path records `Trigger: N/A` — and a relayed review is exactly
the case a maintainer can't recover from the run alone, since
`tend-review` fires only on `pull_request_target` and never retries.
This path is in constant use: `gh api
"repos/max-sixty/tend/actions/runs?event=repository_dispatch"` returns a
steady stream of `tend-mention` runs.

**`workflow_run` names no run.** The ci-fix path hardcodes `REF="CI fix
for workflow run"`, discarding `workflow_run.id` — the id of the CI
failure the job was dispatched to fix.

**Missing fields render as `null`.** `jq -r '.issue.number'` prints the
literal string `null` when the field is absent, so the cell reads
`#null` rather than falling back to `N/A`.

## Solution

Add a `repository_dispatch` branch reading `client_payload.pr`, link the
upstream run id in the `workflow_run` branch, and give every extraction
`// empty` plus a `${VAR:+…}` guard so an absent field leaves the cell
blank (rendered as `N/A` by the existing `${REF:-N/A}`) instead of
`#null`.

## Testing

`shared/steps/` has no shell test harness — shellcheck via pre-commit is
the only automated gate, and it passes on the changed file. So the block
was exercised directly: the REF logic was sliced out of the script
(between the `# Build a one-line reference` comment and `TIMESTAMP=`)
and sourced under crafted `GITHUB_EVENT_PATH` payloads, before and
after.

Before, on `origin/main`:

```
repository_dispatch  -> N/A
workflow_run         -> CI fix for workflow run
issues (no number)   -> #null
```

After:

```
relayed review               -> #815
relay w/o pr                 -> N/A
ci-fix                       -> CI fix for [run 30795510450](https://github.com/max-sixty/tend/actions/runs/30795510450)
ci-fix w/o id                -> CI fix for workflow run
PR event                     -> #821
PR event w/o number          -> N/A
issue comment                -> #808
issue w/o number             -> N/A
schedule                     -> N/A
```

Adding a real harness for `shared/steps/` is worth considering
separately — three of the recent outage-path fixes have all landed in
scripts nothing can test — but that is a bigger change than this fix
warrants, so it is not bundled here.

## Scope

Separate from the other two open changes on this path, and textually
disjoint from both. #818 names the *cause* of a failure (in
`claude/action.yaml`); #809 dedups *rows* across matrix legs (in the
`EXISTING` branch of this same script, lines 49+). This one fixes what
the row *points at*, in the REF block at lines 19–29.

Co-authored-by: tend-agent <270458913+tend-agent@users.noreply.github.com>
…1243

# Conflicts:
#	shared/steps/report-failure.sh

@tend-agent tend-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The guard is correct now — both earlier threads are addressed by 86f39cc, and hoisting the anchor into run_issue_anchor is the right shape: the row and the two dedup matchers can no longer drift apart.

One remaining defect on the reconcile, in the original commit rather than the incremental. The delete path lists comments with ?per_page=100 and no pagination, and GitHub returns issue comments oldest-first — so once a tracker passes 100 comments, the rows this run just posted are not in the response at all and the reconcile silently no-ops. That is the flooded issue the PR exists for: #831 reached 77 rows in a single outage, so crossing 100 on a longer one or a wider matrix is the expected case rather than the edge. The guard is unaffected — gh issue view --json comments does paginate — so only the second line of defence is capped.

--paginate alone will not fix it: gh applies --jq per page, which breaks the cross-page sort_by(.created_at) | .[1:] (each page would keep its own earliest). sort=created&direction=desc is ignored by this endpoint. --paginate --slurp refuses --jq, so the working form is --slurp piped to a downstream jq 'add | …' — which is also what rate-limit-preflight.sh already does one file over for /issues/$PAUSE/events?per_page=100.

Separately, this ships ~40 lines of racy dedup with no test, in a script every adopter runs. generator/tests/test_shared_steps.py already drives rate-limit-preflight.sh and mark-notification-read.sh against a fake gh, and #836 adds a report_failure_env fixture for this exact script — the guard (skip when the anchor is already present) and the reconcile (keep the earliest, delete the rest) both look cheap to cover once whichever of the two lands first.

How the pagination behaviour was verified

Against cli/cli#13840, which has 139 comments:

$ gh issue view 13840 -R cli/cli --json comments --jq '.comments | length'
139
$ gh api "repos/cli/cli/issues/13840/comments?per_page=100" --jq 'length'
100
$ gh api "repos/cli/cli/issues/13840/comments?per_page=100" --jq '[.[0].created_at, .[-1].created_at]'
["2026-07-10T13:54:16Z","2026-08-03T21:08:08Z"]     # oldest-first; newest 39 absent
$ gh api "repos/cli/cli/issues/13840/comments?per_page=100&sort=created&direction=desc" --jq '[.[0].created_at, .[-1].created_at]'
["2026-07-10T13:54:16Z","2026-08-03T21:08:08Z"]     # sort/direction ignored
$ gh api --paginate "repos/cli/cli/issues/13840/comments?per_page=100" --jq 'length'
100
39                                                   # --jq runs per page
$ gh api --paginate --slurp "repos/cli/cli/issues/13840/comments?per_page=100" --jq 'length'
the `--slurp` option is not supported with `--jq` or `--template`
$ gh api --paginate --slurp "repos/cli/cli/issues/13840/comments?per_page=100" | jq -r 'add | length'
139

The empty case needs no extra guard: on a zero-comment issue --paginate --slurp yields [[]], so add gives [] and the filter emits nothing, exit 0.

Comment thread shared/steps/report-failure.sh Outdated
Issue comments come back oldest-first, so the unpaginated `?per_page=100`
read returned only the oldest page. Past 100 comments the rows this run and
its racing sibling just posted are not in the response at all and the
reconcile silently no-ops — on exactly the flooded trackers it exists for
(#831 reached 77 rows in one outage).

`--paginate` alone doesn't fix it: `gh` applies `--jq` per page, so each page
would keep its own earliest comment and `sort_by | .[1:]` would delete the
keeper. `--slurp` refuses `--jq`, so the filter moves downstream with `add`
flattening the pages.

Tests cover both halves of the dedup against a fake `gh`, alongside the
existing `rate-limit-preflight.sh` and `mark-notification-read.sh` suites:
the guard skipping a run already recorded, the reconcile converging two
racing legs to one row, the anchor scoping that keeps a human comment naming
the run out of the delete set, and the pagination case — which is the only
one of the five that fails against the pre-fix script.
@tend-agent

Copy link
Copy Markdown
Collaborator Author

Both points addressed in 834734e.

Pagination — replied inline; took the suggestion as written.

Tests — added five against a fake gh, in the same shape as the existing rate-limit-preflight.sh suite (fixtures in, the script's own filters doing the work). Since neither this nor #836 has landed, this one carries its own report_failure_env rather than waiting on the other:

Test Covers
skips_a_run_already_recorded the guard — a leg whose sibling already posted this run's row posts nothing
appends_a_row_for_an_unrecorded_run the guard doesn't swallow a genuinely new run
reconciles_a_racing_leg two legs that both missed the guard converge to one row
reconciles_past_the_first_page the pagination defect: 139 comments, the racing row at the end
leaves_a_human_comment_naming_the_run the anchor scoping from 86f39cc — a person linking the run isn't a delete candidate

Only reconciles_past_the_first_page fails against the pre-fix script; the other four pass on both, which is the intended split — one regression test for the defect just fixed, four characterising what the original commit already did.

The fake pages the comment list the way the endpoint does (--slurp gets the array of pages, a plain read gets the oldest 100 alone) rather than special-casing the flag the script happens to pass, so that counterfactual is honest rather than a fake built to agree with the fix.

Coordination with #836 — both PRs now add a report_failure_env fixture and a FAKE_GH_REPORT_FAILURE to the end of generator/tests/test_shared_steps.py, so whichever lands second will conflict there. I kept every shared name and env var identical to #836's (REPORT_FAILURE, OUTAGE_TITLE, OUTAGE_LABEL, RUN_LINK, OPEN_ISSUES_JSON, KEEPER_JSON, COMMENT_BODIES, FAKE_NEW_ISSUE) so the resolution is a union rather than a rewrite: take both sets of case arms in the fake — #836's repos/*/issues/N primary-key probe and this one's /comments?per_page=100 pager — and both sets of tests. The two test sets don't overlap: #836's exercise the create path, these the append path.

356 passed  (generator/)
pre-commit: shellcheck, ruff, typos — all pass

@tend-agent tend-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pagination fix is right and the regression test is honest — I re-ran test_report_failure_reconciles_past_the_first_page against 4e93d87's script and it fails there (deleted [], expected ['140']), passes here. gh api --paginate --slurp on a zero-comment issue does return [[]] as the comment claims (checked against #879 in this repo), so add[] → no output, exit 0.

Two gaps in what the new suite pins down, both verified by mutating the script and re-running:

  1. The fake accepts --slurp with --jq; real gh exits 1 on it. That combination is the exact trap the script's comment exists to warn about, and the fake currently blesses it.
  2. The guard's issue-body half is never exercised_seen_by_the_guard hardcodes body: "", so every case reaches the guard through the comments list only. That's the minority path: on the first failed run of an outage one leg creates the issue with its row in the body, and its four siblings arrive at a tracker with no comments at all, matchable only on the body.

Suggestions inline. The second and third apply together — the keyword arg alone is inert, the parametrize alone is a TypeError.

How each gap was confirmed

Two mutations of shared/steps/report-failure.sh, each run against the suite as it stands and against the suite with the suggestions applied:

Mutation As it stands With suggestions
| jq -r "add | …"--jq "add | …" (fold the filter back into gh) 5 passed 4 failed — returncode=1, stderr='the --slurp option is not supported with --jq or --template'
guard's --jq loses .body + "\n" + 5 passed 1 failed — skips_a_run_already_recorded[in-the-issue-body]

And the underlying gh behaviour, on 2.97.0:

$ gh api --paginate --slurp "repos/max-sixty/tend/issues/879/comments?per_page=100"
[[]]
$ gh api --paginate --slurp "repos/max-sixty/tend/issues/879/comments?per_page=100" --jq 'length'
the `--slurp` option is not supported with `--jq` or `--template`
$ echo $?
1

Also checked: --slurp is new to this repo — rate-limit-preflight.sh:114 pairs --paginate with a streaming .[] | select(…) filter, which is safe per-page and so isn't the precedent the earlier review cited. Nothing to fix there; the filter has no cross-page reduction.

Not flagged, for the record: _nwise in the fake is a private jq builtin, but a removal would break the tests loudly rather than silently. #857 also touches this file, but only its header comment.

Comment thread generator/tests/test_shared_steps.py
Comment thread generator/tests/test_shared_steps.py Outdated
Comment thread generator/tests/test_shared_steps.py Outdated
… guard

Two gaps in the suite added a commit ago, both found by mutating the script:

The fake applied `--jq` whatever flags came with it, so `--slurp --jq` was
accepted in the test and rejected in production. That combination is the trap
the reconcile's comment warns about — folding the filter back into `gh --jq`
is the obvious simplification, and it left all five tests green while the
script died under pipefail right after posting its row, never reconciling.
The fake now refuses it the way `gh` does.

The guard matches `.body + "\n" + comments`, but every case reached it through
the comments list, so dropping the body half from the script's `--jq` left the
suite green. That half carries the first failed run of an outage: one leg takes
the `else` branch and seeds the issue with its row in the body, and its
siblings then hit the `EXISTING` branch against a tracker with no comments at
all. Parametrized, at the cost of one extra case.

Confirmed by re-running each mutation: the first now fails 4 tests, the second
fails `skips_a_run_already_recorded[in-the-issue-body]`.

@tend-agent tend-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both gaps from the last review are closed in bb0a92d, re-derived here rather than taken on the commit message's word — I re-applied each mutation to shared/steps/report-failure.sh against this HEAD:

Mutation Result on bb0a92d
| jq -r "add | …"--jq "add | …" 4 failed, returncode=1, stderr='the --slurp option is not supported with --jq or --template'
guard's --jq loses .body + "\n" + 1 failed — skips_a_run_already_recorded[in-the-issue-body] only

The fake's refusal also matches real gh on this runner (2.96.0): gh api --slurp --jq '.' … exits 1 with that message, and it is a parse-time rejection, so placing the check ahead of the subcommand dispatch is the right position for it. The body-half case is faithful too — the else branch seeds the issue through run_issue_create_and_reconcile with $ROW, which carries the anchor, so a sibling leg really does arrive at a tracker whose only match is the body.

357 passed in generator/, ci green on this commit. No further findings; not approving only because the PR is self-authored.

max-sixty pushed a commit that referenced this pull request Aug 7, 2026
…a silent survey (#885)

This hour's Step 2 survey subagent reported **"No visible bot output in
time window"** across 23 successful runs on `max-sixty/tend`. The window
in fact contained 9 reviews (two `APPROVED`), 11 inline review comments,
and 1 conversation comment from `tend-agent` — one of the busiest
windows in the series.

## Evidence

Window 2026-08-07T07:27Z → 08:16Z. The survey's verdict, verbatim:

> **No visible bot output in time window.** All 23 successful runs
(tend-mention, tend-review, tend-review-runs, tend-notifications)
executed and completed but produced zero comments, reviews, or inline
code comments between 2026-08-07T07:27:00Z and 08:16:00Z on tracked
PRs/issues (809, 818, 834, 858, 863, 868, 875, 877, 878, 881, 816, 830).

Two run-independent calls, taking no run ID at all, contradict it
immediately:

```
gh api "repos/max-sixty/tend/issues/comments?since=2026-08-07T07:27:00Z&per_page=100" → 1 tend-agent row
gh api "repos/max-sixty/tend/pulls/comments?since=2026-08-07T07:27:00Z&per_page=100"  → 11 tend-agent rows
```

Plus reviews `4880965990` (COMMENTED), `4881043005` (COMMENTED) and
`4881079314` (APPROVED) on
[#881](#881), `4881113428`
(APPROVED) on [#878](#878), and
five more (all COMMENTED) on
[#809](#809). The survey had
listed 881, 878 and 809 among the PRs it checked, so the numbers were
right and the reads came back empty anyway.

The second-order cost is worse than the omission. Having established
silence, the survey reasoned *from* it: it flagged the window's seven
bot-PR merges as "merged by max-sixty without formal review workflow",
concluding "**direct push/merge bypassing review requirement**, or
**review workflow override via branch protection rule bypass**". Six of
the seven (#818, #834, #858, #868, #875, #877) carry bot `COMMENTED`
reviews predating the window, which is why a `since`-filtered read
missed them; the seventh (#863) is the skill-authorized silence on a
self-authored PR with no concerns. None reads `APPROVED` because GitHub
blocks self-approval — the ordinary shape for a bot PR, not a bypassed
control. Acted on, that summary is a false security finding against the
maintainer.

## Root cause

Every path Step 2 offers is run-keyed: run → `headBranch` → PR →
endpoint. That chain is fine when it works, but it has one failure mode
with no floor — break it anywhere and *every* run returns empty
simultaneously. Uniform absence is exactly what a genuinely quiet hour
looks like, so the summary that comes back is self-consistent and
carries no signal that anything went wrong. The existing sanity-check
line ("note if zero bot activity found across all runs") did fire here,
and the subagent talked itself out of it in the same paragraph — a
prompt to notice absence can't distinguish the two causes, because
nothing in a run-keyed survey can.

## Change

Adds a sweep block to the top of the Step 2 prompt that takes no run ID
— the two `?since=` comment endpoints, bounded on `created_at` at both
ends; a `pr list --search "updated:>"` for the candidate list the review
queries need; and a loop over those candidates counting bot reviews
submitted inside the window, since neither comment endpoint returns
review submissions and an empty-body `APPROVE` is `tend-review`'s most
common output — with instruction to report all four counts and to re-map
from what they found rather than reporting those runs silent. Adds one
sentence at the main-agent review point: an all-quiet report without the
counts isn't usable, and absence isn't a finding to reason from.

This is the check that caught the failure this run. It is four counts
off two run-independent endpoints and one search, and it fails
independently of the mapping it is checking.

## Relation to the other open Step 2 PRs

Distinct problems, non-overlapping edits.
[#864](#864) fixes *who* accepted
(named non-bot actor);
[#869](#869) fixes *which run*
produced an output (confirm from the posting run's log). Both still
start from a candidate PR list reached by run-keyed mapping — neither
makes "no output at all" falsifiable, which is the failure here.

## Gate assessment

- **Evidence level**: High — survey unreliability is recorded in the
evidence gist across prior windows, cumulative **4 → 5** with this one.
High needs 2–3. Prior occurrences were omissions of individual runs and
one mislabelled silence; this is the first categorical zero-output
claim, and the first to produce a fabricated inference from the absence.
- **Structural**: the *specific* empty read is stochastic, but the
skill's exposure is not — Step 2 offers only run-keyed paths, so any
mapping break yields a plausible, uniform, unfalsifiable silence. Replay
it and the summary is equally convincing every time.
- **Change type**: targeted fix (one query block, one sentence) — normal
bar, met.
- **Passes both gates.**

Evidence: https://gist.github.com/e08f6e62d6478163cb425a75648eb7e4

---------

Co-authored-by: tend-agent <270458913+tend-agent@users.noreply.github.com>
max-sixty pushed a commit that referenced this pull request Aug 7, 2026
…nst tend before filing upstream (#891)

## Problem

Two dedup blocks were blind in two different ways, and the cited
duplicate needed both fixed.

**State filter.** `review-runs` Step 5 and `review-reviewers` Step 4
both deduped against PRs with `gh pr list --state open`. A merged PR is
never returned by that query, so a finding whose fix already landed
reads as undeduped and gets filed again. `running-in-ci`'s PR-creation
dedup recheck already gets this right ("with `--state all` so closed and
merged siblings show up"); these two recipes contradicted it.

**Repo scope.** `review-runs` is a generated workflow
([`generator/src/tend/config.py:24`](https://github.com/max-sixty/tend/blob/f65f49f/generator/src/tend/config.py#L24)
lists it in the enabled set), so it runs in each adopter's checkout and
an unqualified `gh pr list` returns *the adopter's* PRs. Step 6 routes
bundled-skill defects upstream to tend, but neither Step 5 nor any of
`running-in-ci`'s dedup recipes — all local-repo — told the agent to
dedup in the target repo before filing there. `--state all` alone does
not close this: the adopter's PR list never contained the upstream fix
at any state.

`review-reviewers` is unaffected by the second half. It runs in
`max-sixty/tend` and files onto tend, so its unqualified `gh pr list`
already resolves to the right repo; only the state filter was wrong
there.

This bites hardest on tend specifically, because of the pinning model:
adopters call `max-sixty/tend/<harness>@X.Y.Z`, so a merged skill fix
stays dormant on their repos until the next release tags. The bug keeps
reproducing after the fix merges — which is exactly the window in which
the analysis legs are looking at it, and exactly when the dedup queries
are blind to the fix.

## What happened

`max-sixty/cargo-affected`'s `tend-review-runs` run
[31160677649](https://github.com/max-sixty/cargo-affected/actions/runs/31160677649)
(08:11:33Z → 08:21:41Z) hit the `| last` evidence-log mis-selection: it
appended ~12 KB of run evidence into the nightly's unrelated comment on
target [#73](max-sixty/cargo-affected#73),
noticed on its post-verify read, restored comment `5188771252`, and
re-appended to the real log `5150650688`. Good recovery. It then filed
[#883](#883) upstream, whose
"Proposed fix" is a `## Run ` heading predicate on the comment selector.

[#875](#875) merged that exact fix
at 07:34:40Z — 46 minutes before the issue was filed — as `test("^## Run
[0-9]")` on the same selector, in the same file. #883 is a duplicate of
a merged PR.

The run made three dedup queries before filing (`gh issue list --state
all --search "tracking issue comment append"`, a broader `gh issue list
--state all` title regex, and a final `gh issue list --state open`
recheck). All three were `gh issue list`, which never returns PRs — and
all three ran against `max-sixty/cargo-affected`. Even had it run Step
5's PR line verbatim, it would not have returned #875, for both reasons:
the state filter excluded merged PRs, and the query's repo was the
adopter's, not tend's.

## The fix

- Both skills: `gh pr list --state open` → `--state all`, projecting
`state,mergedAt` so a merged hit is legible.
- `review-runs` only: add the cross-repo pair (`gh pr list`/`gh issue
list --repo max-sixty/tend --state all`) so a finding heading upstream
under Step 6 is deduped against tend first.
- `review-runs` only: the pinning note is scoped to the upstream repo,
since in that skill the reader is the adopter and the local `gh pr list`
above it has nothing to do with pinned refs. `review-reviewers` keeps
the original wording, where tend is the reader and "on adopters" is the
correct direction.

Both added commands were run against this repo to confirm they parse and
return the expected shape.

## Gate assessment

- **Evidence level**: High. **Occurrences: 1** direct, verified end to
end (session log, both dedup query sets, #875's merge time and diff,
#883's body).
- **Structural, not stochastic.** `gh pr list --state open`
deterministically cannot return a merged PR, and a query scoped to the
adopter's repo deterministically cannot return a tend PR; replayed ten
times it misses #875 ten times. There is no decision point.
- **Change type**: targeted fix — query lines plus one sentence naming
the pinning consequence. It brings both recipes into line with a rule
the same plugin already states in `running-in-ci`, rather than
introducing new policy.
- **Why act at one occurrence**: the six-PR batch merged at 07:34:40Z
([#875](#875),
[#834](#834),
[#868](#868),
[#818](#818),
[#877](#877),
[#858](#858)) is all unreleased,
so six distinct bugs remain observable on every adopter until the next
release. The first analysis leg after that batch produced the first
duplicate. The exposure is six-wide and standing, not one-off.

## Not covered by

No open PR touches either dedup block.
[#845](https://github.com/max-sixty/tend/pull/845)/[#850](https://github.com/max-sixty/tend/pull/850)/[#838](https://github.com/max-sixty/tend/pull/838)
are the run-window cluster;
[#849](#849),
[#856](#856),
[#857](#857),
[#864](#864),
[#869](#869),
[#876](#876),
[#809](#809),
[#837](#837),
[#821](#821),
[#836](#836) are elsewhere.

#883 stays open as a maintainer call — it is redundant with #875 but the
option-2 half of its body (finish the gist migration for `review-runs`)
is not.

Evidence: https://gist.github.com/dca23a6e6a0d8cae2665944ba31676fb

---------

Co-authored-by: tend-agent <270458913+tend-agent@users.noreply.github.com>
…1243

# Conflicts:
#	generator/tests/test_shared_steps.py
@tend-agent

Copy link
Copy Markdown
Collaborator Author

Merged main in during the nightly sweep. The shell halves auto-merged, but generator/tests/test_shared_steps.py conflicted in 9 places — #836 landed as ef7baa1 and rewrote the report-failure suite, so the two sides disagreed about the fixtures, not just which tests to keep. I resolved it as the union this PR's earlier comment prescribed rather than picking a side:

  • Base: main's file wholesale, so all fifteen of its report-failure tests keep the fixtures they were written against.
  • Fake: main's FAKE_GH_REPORT_FAILURE, extended with this branch's --slurp detection and the real-gh refusal of --slurp with --jq; the issue comment arm now also lands the row in ISSUE_COMMENTS_JSON so the reconcile sees what it just posted. The inner dispatch moved from case "$2" to case "$*" — this branch's two calls carry flags (--paginate, -X DELETE) where the path would otherwise sit — with the /comments?per_page=100 and -X DELETE arms ahead of fix(report-failure): reconcile duplicate outage issues by primary-key probe, not a lagging list #836's primary-key probe, since all three URLs contain /issues/.
  • Fixture: report_failure_env gains ISSUE_COMMENTS_JSON and POSTED_AT. It keeps main's empty open-issues.json default, because the create-path tests need it; the five append-path tests here seed it themselves via a new _open_tracker helper.
  • Tests: both sets, unmodified apart from that _open_tracker call. They don't overlap — fix(report-failure): reconcile duplicate outage issues by primary-key probe, not a lagging list #836's exercise the create path, these the append path.

Verified the port still discriminates: with report-failure.sh and lib/run-issue.sh reverted to main in place, skips_a_run_already_recorded (both params), reconciles_a_racing_leg, and reconciles_past_the_first_page fail, while appends_a_row_for_an_unrecorded_run and leaves_a_human_comment_naming_the_run pass — the same split this PR described. Full generator suite: 385 passed. ruff check / ruff format clean.

Worth noting the premise is still live: #905 currently carries exactly the flood this fixes — run 31282882062 in 5 separate comments and 31285087877 in 5 more, each a one-row table differing only in its jitter-spread timestamp, both from review-reviewers matrix runs last night.

Merging main brought #836's create-path dedup alongside this branch's
run_issue_anchor helper, leaving the anchor built twice — once by the
helper the row is written from, once by hand in
run_issue_create_and_reconcile. Change the row's link text and the
create path's guard stops matching it, silently. Route it through the
helper, as this PR's coordination note said the second lander should.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claude-behavior Behavioral issues found by review-reviewers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant