Nextflow is showing jobs as long-running/submitted when they have actually failed in SLURM, leading to workflow hanging.
4 jobs appeared to be running longer than their allocated CPU time according to nextflow.log, but were actually failed jobs in SLURM that Nextflow was not detecting.
Nextflow Log Output
Jun-18 11:19:11.660 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor slurm > tasks to be completed: 4 -- submitted tasks are shown below
~> TaskHandler[jobId: 5484012_72; id: 774; name: PLINK_RECODE_VCF (cb.strain.set.242_foliated.maggot_0.05); status: SUBMITTED; exit: -; error: -; workDir: /scratch4/eande106/Ryan/2025_briggsae_gwas_strainset_sims_batch_2/95/a850a2bcc4a8c0579a379a71562007 started: -; exited: -; ]
~> TaskHandler[jobId: 5484012_73; id: 775; name: PLINK_RECODE_VCF (cb.strain.set.242_crazy.feline_0.05); status: SUBMITTED; exit: -; error: -; workDir: /scratch4/eande106/Ryan/2025_briggsae_gwas_strainset_sims_batch_2/b5/327cbf412a340d1cd5d8ecc0fdf9fd started: -; exited: -; ]
~> TaskHandler[jobId: 5484012_74; id: 776; name: PLINK_RECODE_VCF (cb.strain.set.242_dissuasive.steer_0.05); status: SUBMITTED; exit: -; error: -; workDir: /scratch4/eande106/Ryan/2025_briggsae_gwas_strainset_sims_batch_2/11/0d2fed45f5f3fcd8056af3972c0da1 started: -; exited: -; ]
~> TaskHandler[jobId: 5484012_75; id: 777; name: PLINK_RECODE_VCF (cb.strain.set.242_conceivable.centipede_0.05); status: SUBMITTED; exit: -; error: -; workDir: /scratch4/eande106/Ryan/2025_briggsae_gwas_strainset_sims_batch_2/de/1cbadecae51eb2395990b4d3f8d7e7 started: -; exited: -; ]
Status of the job according to slurm commands
Check for an active job
$ squeue -j 5484012_72
slurm_load_jobs error: Invalid job id specified
Check for a completed job
$ seff 5484012_72
Job ID: 5489867
Array Job ID: 5484012_72
Cluster: slurm
User/Group: rmckeow1/eande106
State: FAILED (exit code 15)
Nodes: 1
Cores per node: 8
CPU Utilized: 00:00:00
CPU Efficiency: 0.00% of 00:00:00 core-walltime
Job Wall-clock time: 00:00:00
Memory Utilized: 0.00 MB (estimated maximum)
Memory Efficiency: 0.00% of 40.00 GB (40.00 GB/node)
Looks like the job failed right away with an exit code of 15. Even when it failed, Nextflow task monitor never detected it as finished
I resolved the issue by terminating the nextflow run and rerunning the workflow with the -resume flag, but it was challenging to identify the problem. The job hung and repeatedly pinged the scheduler for several hours (possibly indefinitely).
Nextflow is showing jobs as long-running/submitted when they have actually failed in SLURM, leading to workflow hanging.
4 jobs appeared to be running longer than their allocated CPU time according to nextflow.log, but were actually failed jobs in SLURM that Nextflow was not detecting.
Nextflow Log Output
Status of the job according to slurm commands
Check for an active job
Check for a completed job
Looks like the job failed right away with an exit code of 15. Even when it failed, Nextflow task monitor never detected it as finished
I resolved the issue by terminating the nextflow run and rerunning the workflow with the
-resumeflag, but it was challenging to identify the problem. The job hung and repeatedly pinged the scheduler for several hours (possibly indefinitely).