From 73b4589b091c7db3d065d9d12f8499fbe1d4ebec Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Fri, 17 Jul 2026 00:03:09 +0200 Subject: [PATCH] fix: install raven-toolbox for the full MEMOTE run, and stop hiding crashes A dispatched full-MEMOTE run (29537354961) reported success in 65 seconds having produced nothing. It died 1.4s in: from raven_toolbox.annotation import add_sbo_terms ModuleNotFoundError: No module named 'raven_toolbox' memoteSnapshot began importing annotateGEM, and so raven_toolbox, in #1062. model-qc.yml installs raven-toolbox in a step of its own before MEMOTE, but memote-full.yml installed only cobra, memote and gurobipy, so the script and one of its two callers went out of sync. Install from requirements.txt there too. The run looked green because `timeout ... || echo "::warning::MEMOTE did not finish within the time limit"` swallows every non-zero exit, so an instant crash was reported as a slow run, and with continue-on-error and nothing to commit the job passed. Branch on the exit code instead: 124 stays a tolerated timeout warning, anything else is an error. In memote-full the score is the entire deliverable, so a crash now fails the run (continue-on-error dropped; the publish steps are !cancelled(), so results are still salvaged). In model-qc MEMOTE is one report among many and must not block the other results being committed, so it keeps continue-on-error but now annotates honestly. --- .github/workflows/memote-full.yml | 25 +++++++++++++++++++++---- .github/workflows/model-qc.yml | 18 ++++++++++++++++-- 2 files changed, 37 insertions(+), 6 deletions(-) diff --git a/.github/workflows/memote-full.yml b/.github/workflows/memote-full.yml index 8adaaf4f..9ba711c6 100644 --- a/.github/workflows/memote-full.yml +++ b/.github/workflows/memote-full.yml @@ -37,8 +37,11 @@ jobs: with: python-version: "3.11" + # requirements.txt brings raven-toolbox (plus cobra and pandas): memoteSnapshot + # imports annotateGEM, which needs raven_toolbox for add_sbo_terms and the model + # reader. Without it the run dies instantly with ModuleNotFoundError. - name: Install dependencies - run: pip install cobra memote gurobipy + run: pip install -r code/test/requirements.txt memote gurobipy - name: Set up Gurobi license env: @@ -59,14 +62,28 @@ jobs: echo "BASE_REF=$base" >> "$GITHUB_ENV" echo "BASE_DIR=$RUNNER_TEMP/base" >> "$GITHUB_ENV" + # Producing the score is this workflow's whole purpose, so a crash must fail the + # run. Only a genuine timeout (exit 124) is tolerated. `|| echo` would swallow + # every non-zero exit and report a 1-second crash as "did not finish in time". + # The publish steps below are !cancelled(), so a failure here still salvages + # whatever exists. - name: MEMOTE snapshot (full suite) - continue-on-error: true env: PYTHONUNBUFFERED: "1" MEMOTE_SUBSET: "" run: | - timeout 19800 python code/test/memoteSnapshot.py \ - || echo "::warning::MEMOTE did not finish within the time limit; score unavailable." + set +e + timeout 19800 python code/test/memoteSnapshot.py + rc=$? + set -e + if [ "$rc" -eq 124 ]; then + echo "::warning::MEMOTE did not finish within the time limit; score unavailable." + exit 0 + fi + if [ "$rc" -ne 0 ]; then + echo "::error::MEMOTE failed with exit $rc; score unavailable. See the log above." + exit "$rc" + fi - name: Fetch target-branch results for comparison run: | diff --git a/.github/workflows/model-qc.yml b/.github/workflows/model-qc.yml index 6ce3c4ab..69720e77 100644 --- a/.github/workflows/model-qc.yml +++ b/.github/workflows/model-qc.yml @@ -160,14 +160,28 @@ jobs: # The fast core subset runs on every pull request. The full suite (which does # FVA / a loopless MILP over every reaction and takes hours) is run on demand # only, by commenting /run memote (see pr-command.yml and memote-full.yml). + # continue-on-error: MEMOTE is one report among many here, so its failure must not + # stop the other checks' results being committed. But distinguish a real timeout + # from a crash: `|| echo` reported both as "did not finish", so a MEMOTE that died + # in a second looked like a slow one and the stale score stood. - name: MEMOTE snapshot (fast subset) continue-on-error: true env: PYTHONUNBUFFERED: "1" MEMOTE_SUBSET: "1" run: | - timeout 2400 python code/test/memoteSnapshot.py \ - || echo "::warning::MEMOTE did not finish within 2400s; score unavailable this run." + set +e + timeout 2400 python code/test/memoteSnapshot.py + rc=$? + set -e + if [ "$rc" -eq 124 ]; then + echo "::warning::MEMOTE did not finish within 2400s; score unavailable this run." + exit 0 + fi + if [ "$rc" -ne 0 ]; then + echo "::error::MEMOTE failed with exit $rc; the score shown is the previously committed one." + exit "$rc" + fi # Everything is in: render the final summary so it is part of the commit below, # then post from the committed file - the comment is updated only after the