Skip to content

fix: retry transient GitHub API errors when posting the QC comment - #1067

Merged
edkerk merged 3 commits into
developfrom
fix/qc-comment-transient-retry
Jul 17, 2026
Merged

fix: retry transient GitHub API errors when posting the QC comment#1067
edkerk merged 3 commits into
developfrom
fix/qc-comment-transient-retry

Conversation

@edkerk

@edkerk edkerk commented Jul 17, 2026

Copy link
Copy Markdown
Member

Main improvements in this PR:

The full MEMOTE run does all of its real work — running the suite and committing the scores — hours before it posts the Model QC comment. The final github-script step in the post-qc-comment composite action made unguarded listComments/updateComment/createComment calls, so a transient 503 from the GitHub API on that last call raised an unhandled HttpError and failed the whole run even though the results were already produced and committed (this is what sank run 29538381748).

This PR wraps those three API calls in an exponential-backoff withRetry helper that retries only transient errors (429 and 5xx), backing off 2s → 4s → 8s → 16s (capped at 30s, up to 5 attempts) and logging a core.warning on each retry. Genuine errors (auth, 404, etc.) still fail fast. Because the action is shared, this also protects the incremental model-qc workflow's comment updates.

No model content, tests, or MEMOTE behaviour is changed — only the resilience of the cosmetic comment-posting step.

I hereby confirm that I have:

  • Any removed reactions and metabolites have been moved to the corresponding deprecated identifier lists in data/deprecatedIdentifiers/.
  • This PR has develop as target branch, and will be resolved with a squash-merge.
  • This PR has main as target branch, and will be resolved with a merge commit.

Maintainers: after merge, credit new contributors by commenting @all-contributors please add @username for <type> on this PR or the linked issue (see CONTRIBUTING).

@github-actions

github-actions Bot commented Jul 17, 2026

Copy link
Copy Markdown

Model quality report

⚠️ 6 pre-existing finding(s), no regressions vs develop. Non-blocking.

Each check name links to its explanation in the testResults README.

Model checks

Duplicate keys (model unloadable) and no growth block the merge; every other row is a non-blocking report.

Check Result Δ vs develop
Duplicate !!omap keys 0 0
Growth (biomass producible) 125 0
Reactions with no metabolites 0 0
Model / annotation-table inconsistencies 0 0
Removed reactions or metabolites not deprecated 0 0
Metabolites missing formula 0 0
Metabolites missing charge 0 0
Reaction bound / GPR issues 0 0
Exact-duplicate reaction groups 0 0
Unused metabolites 0 0
Unused genes 0 0
Malformed cross-references 0 0
Cross-refs inconsistent across compartments 3 0 ⚠️

MACAW and mass/charge balance

Check Result Δ vs develop
Reactions flagged by MACAW dead-end test 2510 0 ⚠️
Reactions flagged as MACAW duplicates 377 0 ⚠️
Mass-imbalanced reactions 87 0 ⚠️
Charge-imbalanced reactions 234 0 ⚠️
Structure vs formula/charge inconsistencies 397 0 ⚠️

Model file and metabolic tasks

Check Result
YAML round-trip (cobrapy) pass
YAML round-trip (RAVEN) pass
YAML lint pass
Essential metabolic tasks 57 passed
Verification metabolic tasks 21 passed

MEMOTE

Total score: 63.2% (core subset)   0

Section Score Δ vs base
consistency 42.4% 0
annotation_met 73.0% 0
annotation_rxn 72.7% 0
annotation_gene 46.7% 0
annotation_sbo 81.7% 0
Per-test scores
Section Test Score
Consistency Stoichiometric Consistency 100.0%
Consistency Mass Balance 0.8%
Consistency Charge Balance 2.1%
Consistency Metabolite Connectivity 0.0%
Consistency Unbounded Flux In Default Medium 100.0%
Annotation - Metabolites Presence of Metabolite Annotation 0.0%
Annotation - Metabolites Metabolite Annotations Per Database 62.3%
Annotation - Metabolites Metabolite Annotation Conformity Per Database 45.8%
Annotation - Metabolites Uniform Metabolite Identifier Namespace 0.0%
Annotation - Reactions Presence of Reaction Annotation 0.0%
Annotation - Reactions Reaction Annotations Per Database 75.9%
Annotation - Reactions Reaction Annotation Conformity Per Database 33.3%
Annotation - Reactions Uniform Reaction Identifier Namespace 0.0%
Annotation - Genes Presence of Gene Annotation 0.0%
Annotation - Genes Gene Annotations Per Database 80.0%
Annotation - Genes Gene Annotation Conformity Per Database 80.0%
Annotation - SBO Terms Metabolite General SBO Presence 0.0%
Annotation - SBO Terms Metabolite SBO:0000247 Presence 0.1%
Annotation - SBO Terms Reaction General SBO Presence 0.0%
Annotation - SBO Terms Metabolic Reaction SBO:0000176 Presence 0.0%
Annotation - SBO Terms Transport Reaction SBO:0000185 Presence 0.7%
Annotation - SBO Terms Exchange Reaction SBO:0000627 Presence 0.0%
Annotation - SBO Terms Demand Reaction SBO:0000628 Presence 100.0%
Annotation - SBO Terms Sink Reactions SBO:0000632 Presence 100.0%
Annotation - SBO Terms Gene General SBO Presence 0.0%
Annotation - SBO Terms Gene SBO:0000243 Presence 0.0%
Annotation - SBO Terms Biomass Reactions SBO:0000629 Presence 0.0%

Full suite: 64.2%   0 · from the last /run memote.

The score above is the fast core subset. Comment /run memote to run the full suite on this pull request; the score updates here when it finishes.

Gene essentiality (Hart 2015)

Not run automatically (it takes hours). Comment /run gene-essentiality to run it on this pull request; the result posts as its own comment.

❌ = a count rose vs the target branch (regression) · ⚠️ = a pre-existing non-zero finding (non-blocking) · ⏳ = still running. Counts link to the CSV listing the exact entries.

Full workflow run · this comment is edited as results come in

@edkerk
edkerk force-pushed the fix/qc-comment-transient-retry branch from 8064fc2 to 0ca475a Compare July 17, 2026 06:14
The full MEMOTE run does its real work (running the suite, committing the
scores) hours before it posts the Model QC comment, but the final
github-script step made unguarded listComments/updateComment/createComment
calls. A transient 503 from the GitHub API on that last call raised an
unhandled HttpError and failed the whole run even though the results were
already produced and committed.

Wrap those calls in an exponential-backoff retry that only retries transient
errors (429 and 5xx), so an API blip on the cosmetic comment update can no
longer sink a multi-hour run.
@edkerk
edkerk force-pushed the fix/qc-comment-transient-retry branch from 0ca475a to b393fc1 Compare July 17, 2026 06:19
@edkerk
edkerk force-pushed the fix/qc-comment-transient-retry branch from 3a44dc1 to 66bfdbb Compare July 17, 2026 19:56
The full MEMOTE suite runs for ~40 minutes, during which develop can advance —
a concurrent results run, or (on a re-run) this workflow's own earlier attempt,
may already have pushed a "full MEMOTE result" commit. The "Update local branch
before committing changes" step fetched but never integrated the remote, so it
left local behind origin and the auto-commit push was rejected as
non-fast-forward (which failed run 29538381748 attempt 2).

Reset local to origin/<branch> before committing. memote_score.md — the only
file this workflow commits — is preserved across the reset when the run actually
produced a new score, so it wins over the remote while every other result file
comes from the up-to-date remote. memote_result.json is untracked at the repo
root, so the reset leaves it in place for the artifact upload.
@edkerk
edkerk merged commit 3953134 into develop Jul 17, 2026
1 check failed
@edkerk
edkerk deleted the fix/qc-comment-transient-retry branch July 17, 2026 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant