Add a custom gauge tracking jobs that are in RUNNING state but whose
startdate exceeds the maximum wall clock time — indicating the Slurm
plugin has stopped delivering completion callbacks.
mybrc_stuck_running_jobs{partition="savio4"} 3
The gauge can be defined with prometheus_client.Gauge and will be
included in the existing /metrics endpoint via the shared registry.
Alert when this is non-zero.
Add a custom gauge tracking jobs that are in
RUNNINGstate but whosestartdateexceeds the maximum wall clock time — indicating the Slurmplugin has stopped delivering completion callbacks.
The gauge can be defined with
prometheus_client.Gaugeand will beincluded in the existing
/metricsendpoint via the shared registry.Alert when this is non-zero.