Skip to content

fix(auto-merge): dispatch develop's post-merge gates — a github.token merge fires nothing (58/74 repos confirmed)#10

Merged
ywatanabe1989 merged 2 commits into
mainfrom
fix/dispatch-post-merge-gates
Jul 22, 2026
Merged

fix(auto-merge): dispatch develop's post-merge gates — a github.token merge fires nothing (58/74 repos confirmed)#10
ywatanabe1989 merged 2 commits into
mainfrom
fix/dispatch-post-merge-gates

Conversation

@ywatanabe1989

Copy link
Copy Markdown
Contributor

A github.token merge fires NOTHING on develop

GitHub suppresses workflow triggers for pushes made with the default
github.token (recursive-run protection). The org auto-merge sweep merges
with exactly that token, so every commit it lands arrives on develop with
zero check runs.

This is worse than a red build. A develop-health gate reads "no checks
present" as "no red signal to honour" and keeps merging — green-by-absence.
The hole conceals itself: a repo can sit in this state for months and look
healthy.

Reported by scitex-agent-container (PR #809), confirmed independently here
across the whole org.

Evidence — measured, not inferred (2026-07-22)

Method: gh repo list scitex-ai --limit 100 → 74 repos. For each, the last 50
develop commits, classified bot (author/committer github-actions[bot], or
commit author/committer name/email containing github-actions) vs user. Then
GET /repos/scitex-ai/<repo>/commits/<sha>/check-runstotal_count for up
to 5 heads of each class.

58 of 74 repos have the hole confirmed — at least one bot-merged develop
head with total_count: 0 while user-pushed heads on the same branch carry
checks. Runners were online and idle; not a capacity problem.

repo bot head bot checks bot heads at 0 user head user checks user samples
figrecipe 3ebb01c0 0 1/1 b33a8ed3 19 19,10,10,17,8
scitex-io 49d709be 0 4/4 f9fe3dd5 8 8,16,9,7,9
scitex-app 0e69dfd5 0 5/5 827b75c9 16 16,10,8,16,10
scitex-etc b9ea6919 0 5/5 0a313963 7 7,8,8,5,8
scitex-db 04f9d2c5 0 5/5 26e9a698 8 8,6,6,8,6
scitex-hpc 3a70a0f2 0 5/5 10a7f2a9 6 6,6,6,6,6
scitex-config 189bccf9 0 5/5 d3b6de29 6 6,9,8,6,8
scitex-genai d4af2a0c 0 5/5 0edf9920 7 7,16,10,5,10
scitex-dict ebebd275 0 5/5 7c98db57 6 6,9,5,14,9
scitex-seizure-metrics 25c63fcd 0 5/5 9698ba9e 11 11,15,11,160,15
scitex-events 90027423 0 5/5 675e8e02 6 6,9,6,6,5
scitex-decorators d6339a1e 0 5/5 c00bd67b 6 6,9,6,6,6
scitex-benchmark cea7344e 0 5/5 f29ecb9b 6 6,8,6,6,6
scitex-str 5df90d21 0 5/5 70be5e1a 6 6,9,6,6,5
scitex-pd dc8cc373 0 5/5 86a3a3eb 7 7,9,8,8,7
scitex-ml 1ffdc6df 0 5/5 6e4ff128 8 8,8,7,7,8
scitex-audio 7ce44999 0 5/5 562be6df 12 12,6,5,6,5
crossref-local 81e5cee1 0 4/5 ccdc040c 12 12,12,6,8,5
scitex-sh 0111d65e 0 4/5 9fd829cc 7 7,8,14,9,7
scitex-resource 3c6e3d41 0 4/5 5653afeb 7 7,6,12,6,6
scitex-introspect 967d801f 0 4/5 86baea88 6 6,6,9,8,6
scitex-context 532735bb 0 4/5 cec84651 6 6,6,9,8,6
scitex-gists 9a79841c 0 4/5 8a503882 7 7,6,9,8,8
scitex-plt 9b51766b 0 4/5 5dc477f1 7 7,6,6,6,8
scitex-repro af5e48ad 0 4/5 f7a83e90 8 8,8,15,15,6
scitex-notification bb0c59de 0 4/5 4ab9a302 7 7,7,9,9,7
scitex-parallel 2763f468 0 4/5 a64d8914 6 6,8,7,0,6
scitex-tex 353673f1 0 4/5 4bc44d3f 8 8,7,6,136
scitex-container 53e0b9a6 0 4/5 443cc0a0 9 9,9,9,9,10
openalex-local 29a960cf 0 4/5 2b9a82cb 6 6,5,6,8,15
scitex-notebook e3695870 0 4/5 c0310cc0 6 6,9,6,6,6
scitex-linalg b061dfa3 0 4/5 8dc1f7a7 6 6,6,134,12,43
scitex-path b8080adb 0 4/5 2089a86a 6 6,9,6,6,133
scitex-nn 23c4a20a 0 4/5 0d92be8b 9 9,9,134,44,6
scitex-compat e3f3ff2d 0 4/5 0da701d5 6 6,6,134,12,11
scitex-dsp c468a302 0 4/5 c375b4e6 6 6,8,7,8,8
scitex-ssh 79f4801d 6 4/5 33bf6b45 7 7,6,8,0,0
scitex-session cabdcdc9 6 4/5 fa903eed 5 5,12,6,6,18
scitex-template a031d947 8 4/5 b8ce6d12 6 6,9,6,8,8
scitex-datetime 853e1b69 0 3/5 b9519f21 6 6,8,132,43
scitex-dataset ec9e7115 0 3/5 f2ecf585 12 12,8,6,8,8
scitex-cv b09edee1 0 3/5 e671d1b2 6 6,6,6,12,12
scitex-msword c11f4e39 0 3/5 55c7ba09 6 6,9,6,6,5
scitex-capture 5c5145a7 0 3/5 5b1f52b0 6 6,6,5,5,5
scitex-scholar 45aee5cf 8 3/5 f32d3196 8 8,8,8,8,8
scitex-security 7f9a9b12 6 3/5 1495c34f 8 8,0,12,197,48
scitex-stats 5b168e7a 0 2/5 76f1eb92 8 8,8,8,8,10
scitex-git cf81dc50 0 2/5 e97d1bfd 6 6,6,130,71,5
scitex-browser ccec2587 0 2/5 4fa41640 6 6,9,6,6,129
scitex-clew 0e5a7b0b 0 2/2 df429ea3 8 8,10,8,10,48
scitex-agent-container e9247565 7 2/5 b8b3815f 9 9,9,1,9,1
scitex-types 44de8ca7 99 2/5 90206079 6 6,9,6,6,132
scitex-logging 1efa3c7a 0 1/5 e96f1082 8 8,6,135,12,28
scitex-web 5aafa48e 0 1/5 5fe03745 6 6,6,131,6,69
scitex-ui 5c8087d4 6 1/2 f7090c29 6 6,10,6,10,12
scitex-orochi 8a632179 0 1/1 3ef09b6c 13 13,15,282,30,17
socialia 204bf1a5 0 1/1 1db143fd 5 5,9,8,5,123
newb fc806904 0 1/1 6714da0d 5 5,5,6,5,15

Not confirmed (2) — every sampled bot head carried checks:
scitex-python (2/2 with checks: 20, 22), scitex-math (2/2: 47, 30).

Not measurable for this defect (11) — no bot commits in the last 50
develop commits, so there is nothing to measure. These are NOT clean;
they are unmeasured: scitex-hub, scitex-dev, scitex-writer,
scitex-cards, .github, claude-code-telegrammer, scitex-storage,
scitex-repl, emacs-claude-code, automated-research-demo, scitex-cloud.

Unmeasurable (3): github-action (no develop branch), scitex-ai (no
develop branch), scitex-os (branches API call failed — needs a retry, not
a pass).

The fix

workflow_dispatch IS exempt from the suppression. After a successful merge
the sweep now dispatches develop's post-merge gates explicitly:

  • guarded on merges > 0, after the merge loop — N merges produce ONE
    dispatch round, not N
  • skipped on dry-run (new dry_run input)
  • permissions: actions: write (without it every dispatch 403s)
  • ::error::-loud and exit 1 on any dispatch failure — the merge landed,
    so an un-CI'd develop head must not pass unnoticed
  • an empty gate set is also a loud error, not a pass. "We merged and
    found nothing to dispatch" IS the green-by-absence state.
  • gates dispatched with --ref develop, and auto-discovered by reading
    .github/workflows/* at ref=develop

No PAT, no bot token.

Why gates are auto-discovered rather than hardcoded

Leaf repos genuinely differ: scitex-io ships
scitex-io-quality-audit-on-ubuntu-latest.yml, scitex-writer ships
quality-audit-on-ubuntu-latest.yml, scitex-stats ships neither. A
hardcoded org-wide list would 404 on much of the org and red every sweep.
Discovery filters to active workflows that exist on develop and accept a bare
workflow_dispatch, minus a deny-list (the sweep itself, release/publish,
CLA). The post_merge_gates input overrides it.

The wrong-ref hole is tested

"A dispatch exists but targets the wrong branch" would satisfy a naive
"is there a dispatch?" check while leaving develop just as unverified.
tests/test_auto_merge_dispatch.py pins it from both sides
(test_every_dispatch_targets_develop,
test_no_dispatch_targets_a_non_develop_ref) and pins the discovery ref too
(test_gate_discovery_reads_the_develop_ref).

13 tests, file-only (parse the YAML, assert on the shell body) — no network,
no gh, cannot flake. Run by the new self-test.yml.

Mutation-proven (each mutation applied, tests run, workflow restored):

mutation result
baseline 13 passed
--ref develop--ref main 2 failed (both ref tests)
discovery ref=developref=main 1 failed (discovery ref test)
delete the gh workflow run line 3 failed
drop actions: write 1 failed (token test)
restored 13 passed

THIS FIX IS INERT UNTIL IT REACHES THE DEFAULT BRANCH

GitHub reads schedule- and check_suite-triggered workflow definitions from
the default branch. Merging this to develop alone changes nothing at
runtime.

Both halves are required:

  1. scitex-ai/.githubmain. This repo's default branch is main, and
    workflow_call resolves the reusable workflow from the default branch.
    Until this lands on main, no caller gets the new body.
  2. Each leaf repo's caller stub must live on that repo's own default branch
    (main).
    Already a documented requirement of this file; restated because
    it is the half that costs a day when missed.

Landing this on develop only will look merged and do nothing.

Second, INDEPENDENT mechanism found while measuring — reported separately

Hypothesis handed over by scitex-ui, now confirmed: main-vs-develop
check context-name divergence. Branch protection matches required contexts
by name, so a required context that no workflow emits under that name is
recorded as absent, not red — a second route to green-by-absence.

Measured: 65 of 74 repos have readable develop protection; 60 declare required
contexts. 20 have at least one required context that did not appear on the
develop head.
The pattern is a reusable-caller prefix mismatch, in both
directions:

  • required pytest-matrix / pytest-matrix-on-ubuntu-py3.11, emitted bare
    pytest-matrix-on-ubuntu-py3.11 — figrecipe, scitex-orochi, scitex-app,
    scitex-audio, scitex-dataset, scitex-session, scitex-repl, scitex-math,
    scitex-linalg, scitex-git, scitex-capture, scitex-web, scitex-compat
  • the exact inverse (required bare, emitted prefixed) — scitex-tex,
    scitex-clew, scitex-ml

Five more (scitex-repro, scitex-db, scitex-tex, scitex-logging,
scitex-nn) emitted nothing at all on their develop head — those heads are
bot-merged, so they are casualties of mechanism 1, not evidence for mechanism 2.

Underlying cause is broad workflow drift between main and develop: 59 of
74 repos
differ in workflow files and/or job ids across the two branches. The
most common single divergence is auto-merge-to-develop.yaml itself —
jobs: [automerge] on main vs jobs: [call] on develop, in roughly 20 repos.

This PR does not fix that. It needs its own change (align required contexts
with emitted names, or standardise the caller job id org-wide) and its own
evidence pass.

Also checked: the partial mitigation that hid this

scitex-agent-container's autobump-release-sweep.yaml already dispatches
pytest-matrix-on-ubuntu-py3-11-3-12-3-13.yml on a no-checks develop head
(lines 413-419), and its header already documents that a github.token push
"fires NOTHING". But it is narrow on three axes: it only fires when the repo
variable AUTOBUMP_ENABLED == 'true', it only dispatches the pytest matrix
(its own comment notes quality-audit and import-smoke lack
workflow_dispatch), and it only runs as release-gating recovery.
autobump-release-sweep.yaml exists in exactly one repo org-wide. A
disarmed, single-repo, single-gate partial fix is a plausible reason nobody
noticed the general hole.

🤖 Generated with Claude Code

https://claude.ai/code/session_017sFEMCtcsCYDP1pAszwRwm

… merge fires nothing

GitHub suppresses workflow triggers for pushes made with the default
github.token (recursive-run protection), and the org auto-merge sweep
merges with exactly that token. Every commit it lands therefore arrives
on develop with ZERO check runs.

Measured across scitex-ai on 2026-07-22 (74 repos enumerated): 58 repos
carry at least one bot-merged develop head whose
GET /commits/<sha>/check-runs returns total_count 0, while user-pushed
neighbours on the same branch carry 5-19 each. Runners online and idle.

This is worse than a red build: a develop-health gate reads 'no checks
present' as 'no red signal' and keeps merging. Green-by-absence — the
hole conceals itself.

workflow_dispatch is exempt from the suppression, so after a successful
merge the sweep now dispatches develop's gates explicitly:
  - guarded on merges > 0, after the merge loop (once per call, not per PR)
  - skipped on dry-run (new dry_run input)
  - permissions.actions: write
  - ::error:: + exit 1 on any dispatch failure, and on an EMPTY gate set
  - gates auto-discovered from the DEVELOP ref (post_merge_gates overrides)

Gate discovery is per-repo because leaf repos genuinely differ in
workflow filenames; a hardcoded org list would 404 on half the org.

Pinned by tests/test_auto_merge_dispatch.py (13 tests, file-only, no
network) run by the new self-test.yml. Mutation-proven: --ref main -> 2
red; discovery ref=main -> 1 red; dispatch line deleted -> 3 red;
actions:write dropped -> 1 red; restored -> 13 pass.

Reference: scitex-ai/scitex-agent-container#809
@ywatanabe1989

Copy link
Copy Markdown
Contributor Author

Follow-up measurement: the partial mitigation is confirmed DISARMED

The body says the autobump partial fix "only fires when AUTOBUMP_ENABLED == 'true'". Measured directly rather than left as an inference:

1. It exists in exactly one repo. Scanned all 74 repos' Actions workflow lists for an autobump sweep:

scitex-agent-container: .github/workflows/autobump-release-sweep.yaml

That is the complete result set — one repo.

2. It is not armed there. GET /repos/scitex-ai/scitex-agent-container/actions/variables/AUTOBUMP_ENABLED404 Not Found.

This is not a permissions artifact — the same token lists the repo's variables successfully:

CI_RUNS_ON, CLAUDE_CODE_CREDENTIALS_JSON_SHA256, SAC_ANTHROPIC_API_KEY_SHA256,
SAC_CLAUDE_CODE_CREDENTIALS_JSON_SHA256, SCITEX_CI_APPTAINER, SCITEX_CI_SIF
(total_count: 6)

AUTOBUMP_ENABLED is not among them, so the sweep's own arming gate
([ "${AUTOBUMP_ENABLED:-}" = "true" ], line 154) evaluates false on every tick and the workflow runs report-only.

Net: the only code in the org that knew about this defect was single-repo, single-gate (pytest matrix only), release-gating in scope, and switched off. That combination is a sufficient explanation for the hole surviving unnoticed across 58 repos — and it is why the fix belongs here in the shared reusable workflow, unconditionally, rather than in a leaf.

@ywatanabe1989
ywatanabe1989 merged commit 1ed5f4a into main Jul 22, 2026
1 check passed
@ywatanabe1989
ywatanabe1989 deleted the fix/dispatch-post-merge-gates branch July 22, 2026 08:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant