feat(observability): env-driven Slack/Discord/webhook Alertmanager receivers

[agjs]

The default observability stack already shipped 14 Prometheus alert
rules (API 5xx + latency, Postgres health, disk/memory/CPU pressure,
Traefik down), but Alertmanager's default receiver was an empty
stub — so alerts fired into the Alertmanager UI and nowhere else.
This wires real notification delivery for the three formats people
actually ask for.

Approach

Alertmanager has no native env-var substitution in its config, so a
small entrypoint.sh renders alertmanager.yml from ALERTMANAGER_*
env vars at container boot, then execs the real Alertmanager binary.
The receiver block is built conditionally:

• ALERTMANAGER_SLACK_WEBHOOK_URL set → slack_configs block emitted
with a readable Slack message template (severity + component +
summary + description). Same env handles Discord because Discord
accepts Slack-format payloads at the /slack suffix — paste a
Discord webhook URL with /slack appended and it just works.
• ALERTMANAGER_WEBHOOK_URL set → webhook_configs block emitted
for operators who want Alertmanager's native JSON (custom bridges,
n8n, PagerDuty events-v2 translators, etc.).
• Both can be set; alerts fan out to both.
• Neither set → receiver has no notifiers; alerts surface in the
Alertmanager UI at :9093 only. Still a valid config — verified
with amtool check-config.

Routing already supports severity=page (1h re-notify, urgent) and
severity=warn (12h re-notify, no pager-spam) via the existing rule
labels — the new alertmanager.yml header preserves that.

Files

• compose/alertmanager/entrypoint.sh (new) — renders + execs.
• compose/alertmanager/alertmanager.yml (deleted) — static file
  superseded by the env-driven render.
• compose/docker-compose.observability.yml — alertmanager service now
  uses entrypoint, declares ALERTMANAGER_* env vars, mounts the
  script in place of the static config.
• compose/.env.example — documents the new env vars + the Discord
  /slack-suffix trick.
• infra/compose/docs/alerts.md (new) — full walkthrough: which rules
  ship, Slack setup, Discord setup, generic webhook setup, tuning
  thresholds, adding rules, end-to-end ping test.
• infra/compose/docs/observability.md — old "Adding alert rules"
  paragraph replaced with an Alerts section pointing at alerts.md.

Verification

• STACK=dev ./dev.sh config --quiet exits 0 with the new compose.
• shellcheck -x -S warning clean on entrypoint.sh.
• amtool check-config passes against the rendered config in both the
  both-URLs case (slack_configs + webhook_configs both emitted) and
  the no-URLs case (bare - name: default receiver, alerts UI-only).

Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com