Description
AlertManager records targets whose webhook alert subscription failed permanently and then skips them on every later refresh. Nothing ever retries them, so the only recovery is restarting the collector. A target can go days with no alert subscription while /health still reports the collector as healthy, which makes the gap easy to miss.
Two related problems sit in the same path:
- The collector container never received
ALERT_WEBHOOK_BASE_URL or ALERT_ENABLE_WEBHOOK_FALLBACK. docker-compose.yml did not forward them, so webhook fallback ran on built-in defaults and ignored what the operator put in .env.
- Webhook fallback shipped enabled by default. The receiver accepts unauthenticated POSTs and its port is not published, so the default was both insecure and non-functional in a stock deployment.
Steps to reproduce
- Configure a target that falls back to webhooks because SSE is unavailable.
- Make the webhook subscription fail permanently, for example by pointing
ALERT_WEBHOOK_BASE_URL at a host the BMC cannot resolve.
- Watch the collector log the permanent failure and add the target to its permanently-failed set.
- Wait through several subscription refresh cycles.
- Check
/health/detailed and the alerts page. The target is still listed as permanently failed and no new subscription attempt appears in the logs.
- Separately, set
ALERT_ENABLE_WEBHOOK_FALLBACK=false in .env and restart. The collector still behaves as if fallback were enabled, because the variable never reaches the container.
Expected vs. actual behavior
Expected: a permanent webhook failure retries after a bounded cooldown, the operator can see when the next attempt is due, and .env controls whether fallback runs at all.
Actual: the target stays permanently failed until the process restarts, the API exposes only a flat list of failed target IDs with no retry time, and .env has no effect on webhook fallback.
Environment
- GYANAM version / commit (
git rev-parse --short HEAD): 1ea5fe5
- OS / version: Ubuntu 26.04 LTS (WSL2)
- Docker version: podman 5.7.0 via the
docker alias
- Fleet size (approx. GPUs): 247 targets, roughly 2000 GPUs
- Transport (Redfish / SSH-proxy / SSE): SSE with webhook fallback
Description
AlertManager records targets whose webhook alert subscription failed permanently and then skips them on every later refresh. Nothing ever retries them, so the only recovery is restarting the collector. A target can go days with no alert subscription while
/healthstill reports the collector as healthy, which makes the gap easy to miss.Two related problems sit in the same path:
ALERT_WEBHOOK_BASE_URLorALERT_ENABLE_WEBHOOK_FALLBACK.docker-compose.ymldid not forward them, so webhook fallback ran on built-in defaults and ignored what the operator put in.env.Steps to reproduce
ALERT_WEBHOOK_BASE_URLat a host the BMC cannot resolve./health/detailedand the alerts page. The target is still listed as permanently failed and no new subscription attempt appears in the logs.ALERT_ENABLE_WEBHOOK_FALLBACK=falsein.envand restart. The collector still behaves as if fallback were enabled, because the variable never reaches the container.Expected vs. actual behavior
Expected: a permanent webhook failure retries after a bounded cooldown, the operator can see when the next attempt is due, and
.envcontrols whether fallback runs at all.Actual: the target stays permanently failed until the process restarts, the API exposes only a flat list of failed target IDs with no retry time, and
.envhas no effect on webhook fallback.Environment
git rev-parse --short HEAD): 1ea5fe5dockeralias