Route alerts by severity, and prove where they landed

[sethbergman]

Alertmanager had a single catch-all route and a receiver configured as
webhook_configs: [] — one that parses, validates, and delivers
nowhere. Every alert took the same path, and the tests could only
assert alerts reached Alertmanager's API, never that they reached anyone.

There are still no vendor receivers, deliberately. A config full of
fake PagerDuty keys proves nothing a reader could reuse. What transfers
is everything above the vendor:

	Critical	Warning
Receiver	page	ticket
group_wait	0s	30s
repeat_interval	15m	12h

group_wait: 0s is the point of the split — at the default a page waits
to see whether a second alert joins its group, which is fine for a ticket
and bad for a cluster that has lost quorum. And a critical that notifies
once then goes quiet is indistinguishable from one nobody sent.

Grouping is by alertname + cluster, not instance: three nodes
sealing at once is one incident, and per-instance grouping would page
three times for it.

Two inhibit rules so an outage pages for the cause, not each consequence:
no scrape targets at all explains every node being down, and lost quorum
explains there being no active node.

Proving it

Receivers post to alert-sink, which records each delivery. Not a
stand-in for a pager — it exists because Alertmanager's API cannot
answer the question worth asking: not "did this alert arrive" but "which
receiver did it reach". Without a receiving end, a misrouted alert looks
exactly like a correctly routed one.

The integration suite now reads it and asserts a critical alert reached
the pager path and was not also filed as a ticket.

tests/alert-routing covers the tree. Where the answer belongs to
Alertmanager — which route matches a label set — the question goes to
amtool from the pinned image rather than a reimplementation of its
matching rules, because a test that reimplements what it tests agrees
with itself and nothing else. Those cases need Docker and skip without it
(I have no Docker locally, so the four amtool assertions are unverified
until CI runs them — everything else passes here).

The check worth reading

Every alert in the rule file must carry a severity that has a route of
its own. One with a misspelled severity matches no route, falls to the
catch-all, and is never paged for — nothing else here would notice. The
suite asserts that failure mode directly by asking amtool where
severity=crticial goes.

Mutations

Nine, each caught by the assertion added for it:

Mutation	Caught by
an alert loses its severity label	every alert carries a severity
critical waits to be grouped	critical has no group_wait
critical stops repeating	critical repeats until someone acts
warning repeats every 15m	warning does not repeat through the night
grouping goes per-instance	grouping is not per-instance
inhibit names a typo'd alert	every inhibit rule references a real alert
a route points at an undefined receiver	no route names an undefined receiver
webhook_configs: [] (the previous state)	every receiver actually delivers somewhere
the warning route is deleted	every severity has a route of its own

Still not covered

Whether your PagerDuty key is correct. And that two alerts firing at once
really does suppress the second — inhibition is checked structurally so a
typo cannot leave an inert rule, but arranging two specific alerts to
overlap on a live cluster is a fixture problem rather than a monitoring
one. docs/monitoring.md says both.

[sethbergman]

Verified in CI, not just green

The four amtool assertions skip when Docker is absent, and I have no
Docker locally — so "23/23 green" could have meant they never ran. They
did:

passed: 16   failed: 0   skipped: 0

skipped: 0 is the line that matters. All four routing questions were
put to amtool from the pinned prom/alertmanager:v0.27.0 image,
including the one that documents the failure mode:

PASS  a misspelled severity is silently not paged

The integration suite confirms delivery end to end — 82 passed (80
before), the two new ones being:

PASS  and the routing tree delivers it to the pager receiver
PASS  and does not also file it as a ticket

That second one is worth keeping. A route that matches twice pages and
files, and the duplicate is exactly the noise that trains people to
ignore the pager.

The regression this guards

The previous config was:

receivers:
  - name: default
    webhook_configs: []

That parses. It validates. amtool check-config is happy with it. And it
delivers nowhere. One of the nine mutations restores precisely that
state, so the assertion that catches it is pinned against the actual bug
rather than a hypothetical one.

Still not covered, deliberately

• Whether your PagerDuty key is correct. No vendor receiver exists
  here and none should; swapping the sink for a real receiver is one line
  per receiver, and the tree above it is what transfers.
• That inhibition actually suppresses at runtime. The rules are
  checked structurally — each names alerts that exist, so a typo cannot
  leave an inert rule that reads in review as though the noise problem
  were handled. Arranging two specific alerts to overlap on a live
  cluster is a fixture problem rather than a monitoring one.

Both are stated in docs/monitoring.md rather than left for someone to
discover.