cluster/slurm.py:job_status does state = out.strip() on squeue -j <master> -h -o '%T'. When an array has tasks in more than one state (or several running tasks), squeue returns multiple lines, so state becomes e.g. "RUNNING\nPENDING". The daemon then treats it as a non-terminal 'other' transition and writes the multiline value into the DB (daemon.py ~L222); the RUNNING transition (started notification, started_at, log discovery) never fires, and the TUI renders ? RUNNING\nPENDING. Only fully-pending (single aggregated 12345_[0-99] row) or single-remaining-task arrays behave.
Also affects the sacct terminal path (slurm.py:88-93): it returns the first task's state, so a 500-task array where task 0 completed but 37 failed reports COMPLETED and the FAILED notification never fires.
Fix: parse all squeue/sacct lines and aggregate to a canonical status plus per-task counts (running/pending/failed/completed). Primary user submits job arrays as the main workflow, so this is high impact. Needs regression tests (none today for the array status path).
cluster/slurm.py:job_statusdoesstate = out.strip()onsqueue -j <master> -h -o '%T'. When an array has tasks in more than one state (or several running tasks), squeue returns multiple lines, sostatebecomes e.g."RUNNING\nPENDING". The daemon then treats it as a non-terminal 'other' transition and writes the multiline value into the DB (daemon.py ~L222); the RUNNING transition (started notification, started_at, log discovery) never fires, and the TUI renders? RUNNING\nPENDING. Only fully-pending (single aggregated12345_[0-99]row) or single-remaining-task arrays behave.Also affects the sacct terminal path (slurm.py:88-93): it returns the first task's state, so a 500-task array where task 0 completed but 37 failed reports COMPLETED and the FAILED notification never fires.
Fix: parse all squeue/sacct lines and aggregate to a canonical status plus per-task counts (running/pending/failed/completed). Primary user submits job arrays as the main workflow, so this is high impact. Needs regression tests (none today for the array status path).