Skip to content

cmgd_mpa4.2 2.3.0 — fleet health #256

Open
Open
cmgd_mpa4.2 2.3.0 — fleet health#256
Labels
healthautomated fleet-health heartbeatstatus:attentionfleet health: regression signal

Description

@seandavi
seandavi
opened on Oct 10, 2026

cmgd_mpa4.2 2.3.0 — Status: 🔴 ATTENTION · last run 2026-10-19 (14:03Z)

Automated per-version fleet-health heartbeat (read-only). The body is rewritten each run — no notification. A comment is posted only on non-green days and on recovery. Rotates to a new issue when a new version goes active. See the cmgd-triage skill.

date verdict
2026-10-19 🔴 ATTENTION
2026-10-18 🔴 ATTENTION
2026-10-17 🔴 ATTENTION
2026-10-16 🔴 ATTENTION
2026-10-15 🔴 ATTENTION
2026-10-14 🔴 ATTENTION
2026-10-13 🔴 ATTENTION
2026-10-12 🟢 GREEN
2026-10-11 🟢 GREEN
2026-10-10 🟢 GREEN
latest report (2026-10-19) 
cmgd_mpa4.2 health report — 2026-10-19 14:03:27Z
api: https://nf-telemetry.seandavi.workers.dev
============================================================
active workflow: pk=7 version=2.3.0 revision=2.3.3

jobs: 6267/6374 complete (98.32%)  running=50 pending=57 claimed=0 submitted=0 failed=0 dead_letter=57
global dead_letter_unresolved: 1 (all versions; batch's own = 57)
dispatchability: ok (cmgd not stuck)

daemons:
  login07.anvil.rcac.purdue.edu            status=running active_runs=10 last_seen=2026-10-19T14:03:25.284Z
  login-ci4                                status=running active_runs=10 last_seen=2026-10-19T14:03:25.284Z
  br012.ib.bridges2.psc.edu                status=running active_runs=6 last_seen=2026-10-19T14:03:21.299Z

failure signatures (last 24h):
  none

assessment:
============================================================
VERDICT: ATTENTION
  ! daemon login07.anvil.rcac.purdue.edu runs 47a4471; main changed 3 deployed file(s) since (e.g. config/client-bridges2.yaml); run `just deploy-clients`
  ! daemon login-ci4 runs ea8e913; main changed 4 deployed file(s) since (e.g. config/client-anvil.yaml); run `just deploy-clients`

Activity

  1. seandavi
    added
    healthautomated fleet-health heartbeat
    status:greenfleet health: all green
    status:attentionfleet health: regression signal
    and removed
    status:greenfleet health: all green
    on Oct 10, 2026

  2. seandavi commented on Oct 13, 2026

    @seandavi
    seandavi
    on Oct 13, 2026
    OwnerAuthor

    @seandavi 🔴 ATTENTION — cmgd_mpa4.2 2.3.0 — 2026-10-13

    • metaphlan_unknown_viruses_lists_full: 5 OOM failures include retries (attempts [1, 2]) — memory scaling not keeping up
    full report 
    cmgd_mpa4.2 health report — 2026-10-13 14:04:26Z
api: https://nf-telemetry.seandavi.workers.dev
============================================================
active workflow: pk=7 version=2.3.0 revision=793996b

jobs: 125/137 complete (91.24%)  running=12 pending=0 claimed=0 submitted=0 failed=0 dead_letter=0
global dead_letter_unresolved: 0 (all versions; batch's own = 0)
dispatchability: ok (cmgd not stuck)

daemons:
  login-ci4                                status=running active_runs=1 last_seen=2026-10-13T14:04:24.669Z
  login07.anvil.rcac.purdue.edu            status=idle active_runs=0 last_seen=2026-10-13T14:04:22.941Z

failure signatures (last 24h):
  metaphlan_unknown_viruses_lists_full     exit=137    action=RETRY    n=5
  metaphlan_markers_full                   exit=137    action=RETRY    n=2

assessment:

recent task failure rate (last 3h): 2.7% (4/150)
  [expected] daemon login-ci4 reports no commit (1.0.0); redeploy with `just deploy-clients` (#254)
  [expected] metaphlan_markers_full: 2 attempt-1 OOMs, all recover on retry — expected
============================================================
VERDICT: ATTENTION
  ! metaphlan_unknown_viruses_lists_full: 5 OOM failures include retries (attempts [1, 2]) — memory scaling not keeping up

  3. seandavi commented on Oct 14, 2026

    @seandavi
    seandavi
    on Oct 14, 2026
    OwnerAuthor

    @seandavi 🔴 ATTENTION — Traceback None — 2026-10-14

    • (see report)
    full report 
    Traceback (most recent call last):
  File "/home/davsean/.local/share/cmgd-health/health_report.py", line 312, in <module>
    raise SystemExit(main())
  File "/home/davsean/.local/share/cmgd-health/health_report.py", line 295, in main
    r = run(args)
  File "/home/davsean/.local/share/cmgd-health/health_report.py", line 204, in run
    r.say(f"  {row['process']:<40} exit={row['exit_code']:<6} "
TypeError: unsupported format string passed to NoneType.__format__

  4. seandavi commented on Oct 15, 2026

    @seandavi
    seandavi
    on Oct 15, 2026
    OwnerAuthor

    @seandavi 🔴 ATTENTION — cmgd_mpa4.2 2.3.0 — 2026-10-15

    • dead-letter is 8.8% of this batch (12/137) — over 3.0% threshold
    full report 
    cmgd_mpa4.2 health report — 2026-10-15 14:02:03Z
api: https://nf-telemetry.seandavi.workers.dev
============================================================
active workflow: pk=7 version=2.3.0 revision=2.3.1

jobs: 125/137 complete (91.24%)  running=0 pending=0 claimed=0 submitted=0 failed=12 dead_letter=12
global dead_letter_unresolved: 12 (all versions; batch's own = 12)
dispatchability: ok (cmgd not stuck)

daemons:
  login-ci4                                status=running active_runs=1 last_seen=2026-10-15T14:02:02.042Z
  login07.anvil.rcac.purdue.edu            status=idle active_runs=0 last_seen=2026-10-15T14:02:01.523Z
  br012.ib.bridges2.psc.edu                status=running active_runs=0 last_seen=2026-10-15T14:01:59.695Z

failure signatures (last 24h):
  none

assessment:
============================================================
VERDICT: ATTENTION
  ! dead-letter is 8.8% of this batch (12/137) — over 3.0% threshold

  5. seandavi commented on Oct 16, 2026

    @seandavi
    seandavi
    on Oct 16, 2026
    OwnerAuthor

    @seandavi 🔴 ATTENTION — cmgd_mpa4.2 2.3.0 — 2026-10-16

    • daemon login07.anvil.rcac.purdue.edu runs d17a21e; main changed 6 deployed file(s) since (e.g. config/client-alpine.yaml); run just deploy-clients
    • daemon br012.ib.bridges2.psc.edu runs afb6cfc; main changed 9 deployed file(s) since (e.g. config/client-alpine.yaml); run just deploy-clients
    full report 
    cmgd_mpa4.2 health report — 2026-10-16 14:00:42Z
api: https://nf-telemetry.seandavi.workers.dev
============================================================
active workflow: pk=7 version=2.3.0 revision=2.3.2

jobs: 350/607 complete (57.66%)  running=245 pending=0 claimed=0 submitted=0 failed=12 dead_letter=12
global dead_letter_unresolved: 12 (all versions; batch's own = 12)
dispatchability: ok (cmgd not stuck)

daemons:
  login07.anvil.rcac.purdue.edu            status=running active_runs=10 last_seen=2026-10-16T14:00:38.608Z
  br012.ib.bridges2.psc.edu                status=running active_runs=0 last_seen=2026-10-16T14:00:38.482Z
  login-ci4                                status=running active_runs=1 last_seen=2026-10-16T14:00:37.614Z

failure signatures (last 24h):
  fasterq_dump                             exit=1      action=RETRY    n=1123
  fasterq_dump                             exit=1      action=IGNORE   n=533

assessment:

recent task failure rate (last 3h): 0.0% (0/588)
  [expected] fasterq_dump: 1123 download failures (exit 1) — SRA/ENA noise; undownloadable accessions -> DLQ
  [expected] fasterq_dump: 533 download failures (exit 1) — SRA/ENA noise; undownloadable accessions -> DLQ
============================================================
VERDICT: ATTENTION
  ! daemon login07.anvil.rcac.purdue.edu runs d17a21e; main changed 6 deployed file(s) since (e.g. config/client-alpine.yaml); run `just deploy-clients`
  ! daemon br012.ib.bridges2.psc.edu runs afb6cfc; main changed 9 deployed file(s) since (e.g. config/client-alpine.yaml); run `just deploy-clients`

  6. seandavi commented on Oct 17, 2026

    @seandavi
    seandavi
    on Oct 17, 2026
    OwnerAuthor

    @seandavi 🔴 ATTENTION — cmgd_mpa4.2 2.3.0 — 2026-10-17

    • daemon login07.anvil.rcac.purdue.edu runs cf6626e; main changed 2 deployed file(s) since (e.g. config/client-bridges2.yaml); run just deploy-clients
    • daemon login-ci4 runs 83ac8c9; main changed 2 deployed file(s) since (e.g. config/client-bridges2.yaml); run just deploy-clients
    • metaphlan_unknown_viruses_lists_full: 174 OOM failures include retries (attempts [1, 2, 3]) — memory scaling not keeping up
    • metaphlan_unknown_viruses_lists_rarefied: 172 OOM failures include retries (attempts [1, 2, 3]) — memory scaling not keeping up
    • metaphlan_unknown_viruses_lists_full: 86 OOM failures include retries (attempts [1, 2, 3]) — memory scaling not keeping up
    • metaphlan_unknown_viruses_lists_rarefied: 86 OOM failures include retries (attempts [1, 2, 3]) — memory scaling not keeping up
    • metaphlan_unknown_viruses_lists_full: 11 OOM failures include retries (attempts [1, 2, 3]) — memory scaling not keeping up
    • metaphlan_unknown_viruses_lists_rarefied: 3 OOM failures include retries (attempts [1, 2, 3]) — memory scaling not keeping up
    full report 
    cmgd_mpa4.2 health report — 2026-10-17 14:02:46Z
api: https://nf-telemetry.seandavi.workers.dev
============================================================
active workflow: pk=7 version=2.3.0 revision=2.3.2

jobs: 2141/6374 complete (33.59%)  running=467 pending=3727 claimed=0 submitted=0 failed=39 dead_letter=39
global dead_letter_unresolved: 39 (all versions; batch's own = 39)
dispatchability: ok (cmgd not stuck)

daemons:
  login07.anvil.rcac.purdue.edu            status=running active_runs=10 last_seen=2026-10-17T14:02:46.701Z
  br012.ib.bridges2.psc.edu                status=running active_runs=3 last_seen=2026-10-17T14:02:45.245Z
  login-ci4                                status=running active_runs=10 last_seen=2026-10-17T14:02:44.232Z

failure signatures (last 24h):
  metaphlan_unknown_viruses_lists_full     exit=1      action=RETRY    n=174
  metaphlan_unknown_viruses_lists_rarefied exit=1      action=RETRY    n=172
  metaphlan_unknown_viruses_lists_full     exit=1      action=IGNORE   n=86
  metaphlan_unknown_viruses_lists_rarefied exit=1      action=IGNORE   n=86
  fasterq_dump                             exit=3      action=RETRY    n=42
  fasterq_dump                             exit=3      action=IGNORE   n=20
  metaphlan_unknown_viruses_lists_full     exit=137    action=RETRY    n=11
  metaphlan_unknown_viruses_lists_rarefied exit=137    action=RETRY    n=3

assessment:

recent task failure rate (last 3h): 1.0% (97/9794)
  [expected] fasterq_dump: 42 download failures (exit 3) — SRA/ENA noise; undownloadable accessions -> DLQ
  [expected] fasterq_dump: 20 download failures (exit 3) — SRA/ENA noise; undownloadable accessions -> DLQ
============================================================
VERDICT: ATTENTION
  ! daemon login07.anvil.rcac.purdue.edu runs cf6626e; main changed 2 deployed file(s) since (e.g. config/client-bridges2.yaml); run `just deploy-clients`
  ! daemon login-ci4 runs 83ac8c9; main changed 2 deployed file(s) since (e.g. config/client-bridges2.yaml); run `just deploy-clients`
  ! metaphlan_unknown_viruses_lists_full: 174 OOM failures include retries (attempts [1, 2, 3]) — memory scaling not keeping up
  ! metaphlan_unknown_viruses_lists_rarefied: 172 OOM failures include retries (attempts [1, 2, 3]) — memory scaling not keeping up
  ! metaphlan_unknown_viruses_lists_full: 86 OOM failures include retries (attempts [1, 2, 3]) — memory scaling not keeping up
  ! metaphlan_unknown_viruses_lists_rarefied: 86 OOM failures include retries (attempts [1, 2, 3]) — memory scaling not keeping up
  ! metaphlan_unknown_viruses_lists_full: 11 OOM failures include retries (attempts [1, 2, 3]) — memory scaling not keeping up
  ! metaphlan_unknown_viruses_lists_rarefied: 3 OOM failures include retries (attempts [1, 2, 3]) — memory scaling not keeping up

  7. seandavi commented on Oct 18, 2026

    @seandavi
    seandavi
    on Oct 18, 2026
    OwnerAuthor

    @seandavi 🔴 ATTENTION — cmgd_mpa4.2 2.3.0 — 2026-10-18

    • daemon login-ci4 runs ea8e913; main changed 1 deployed file(s) since (e.g. config/client-anvil.yaml); run just deploy-clients
    • metaphlan_unknown_viruses_lists_rarefied: 20 OOM failures include retries (attempts [1, 2, 3]) — memory scaling not keeping up
    • metaphlan_unknown_viruses_lists_full: 18 OOM failures include retries (attempts [1, 2, 3]) — memory scaling not keeping up
    • metaphlan_unknown_viruses_lists_full: 10 OOM failures include retries (attempts [1, 2, 3]) — memory scaling not keeping up
    • metaphlan_unknown_viruses_lists_rarefied: 10 OOM failures include retries (attempts [1, 2, 3]) — memory scaling not keeping up
    • metaphlan_unknown_viruses_lists_full: 8 OOM failures include retries (attempts [1, 2, 3]) — memory scaling not keeping up
    full report 
    cmgd_mpa4.2 health report — 2026-10-18 14:04:24Z
api: https://nf-telemetry.seandavi.workers.dev
============================================================
active workflow: pk=7 version=2.3.0 revision=2.3.3

jobs: 6267/6374 complete (98.32%)  running=50 pending=0 claimed=0 submitted=0 failed=57 dead_letter=57
global dead_letter_unresolved: 162 (all versions; batch's own = 57)
dispatchability: ok (cmgd not stuck)

daemons:
  br012.ib.bridges2.psc.edu                status=running active_runs=3 last_seen=2026-10-18T14:04:24.724Z
  login07.anvil.rcac.purdue.edu            status=running active_runs=10 last_seen=2026-10-18T14:04:24.618Z
  login-ci4                                status=running active_runs=10 last_seen=2026-10-18T14:04:24.157Z

failure signatures (last 24h):
  fasterq_dump                             exit=3      action=RETRY    n=70
  fasterq_dump                             exit=3      action=IGNORE   n=29
  metaphlan_unknown_viruses_lists_rarefied exit=1      action=RETRY    n=20
  metaphlan_unknown_viruses_lists_full     exit=1      action=RETRY    n=18
  metaphlan_unknown_viruses_lists_full     exit=1      action=IGNORE   n=10
  metaphlan_unknown_viruses_lists_rarefied exit=1      action=IGNORE   n=10
  metaphlan_unknown_viruses_lists_full     exit=137    action=RETRY    n=8

assessment:

recent task failure rate (last 3h): 0.3% (6/2005)
  [expected] fasterq_dump: 70 download failures (exit 3) — SRA/ENA noise; undownloadable accessions -> DLQ
  [expected] fasterq_dump: 29 download failures (exit 3) — SRA/ENA noise; undownloadable accessions -> DLQ
============================================================
VERDICT: ATTENTION
  ! daemon login-ci4 runs ea8e913; main changed 1 deployed file(s) since (e.g. config/client-anvil.yaml); run `just deploy-clients`
  ! metaphlan_unknown_viruses_lists_rarefied: 20 OOM failures include retries (attempts [1, 2, 3]) — memory scaling not keeping up
  ! metaphlan_unknown_viruses_lists_full: 18 OOM failures include retries (attempts [1, 2, 3]) — memory scaling not keeping up
  ! metaphlan_unknown_viruses_lists_full: 10 OOM failures include retries (attempts [1, 2, 3]) — memory scaling not keeping up
  ! metaphlan_unknown_viruses_lists_rarefied: 10 OOM failures include retries (attempts [1, 2, 3]) — memory scaling not keeping up
  ! metaphlan_unknown_viruses_lists_full: 8 OOM failures include retries (attempts [1, 2, 3]) — memory scaling not keeping up

  8. seandavi commented on Oct 19, 2026

    @seandavi
    seandavi
    on Oct 19, 2026
    OwnerAuthor

    @seandavi 🔴 ATTENTION — cmgd_mpa4.2 2.3.0 — 2026-10-19

    • daemon login07.anvil.rcac.purdue.edu runs 47a4471; main changed 3 deployed file(s) since (e.g. config/client-bridges2.yaml); run just deploy-clients
    • daemon login-ci4 runs ea8e913; main changed 4 deployed file(s) since (e.g. config/client-anvil.yaml); run just deploy-clients
    full report 
    cmgd_mpa4.2 health report — 2026-10-19 14:03:27Z
api: https://nf-telemetry.seandavi.workers.dev
============================================================
active workflow: pk=7 version=2.3.0 revision=2.3.3

jobs: 6267/6374 complete (98.32%)  running=50 pending=57 claimed=0 submitted=0 failed=0 dead_letter=57
global dead_letter_unresolved: 1 (all versions; batch's own = 57)
dispatchability: ok (cmgd not stuck)

daemons:
  login07.anvil.rcac.purdue.edu            status=running active_runs=10 last_seen=2026-10-19T14:03:25.284Z
  login-ci4                                status=running active_runs=10 last_seen=2026-10-19T14:03:25.284Z
  br012.ib.bridges2.psc.edu                status=running active_runs=6 last_seen=2026-10-19T14:03:21.299Z

failure signatures (last 24h):
  none

assessment:
============================================================
VERDICT: ATTENTION
  ! daemon login07.anvil.rcac.purdue.edu runs 47a4471; main changed 3 deployed file(s) since (e.g. config/client-bridges2.yaml); run `just deploy-clients`
  ! daemon login-ci4 runs ea8e913; main changed 4 deployed file(s) since (e.g. config/client-anvil.yaml); run `just deploy-clients`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    healthautomated fleet-health heartbeatstatus:attentionfleet health: regression signal

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions