A multi-window multi-burn-rate SLO alerting engine over OpenTelemetry-style spans, with an honest measured comparison against a naive threshold: the naive alert detects a severe regression a few minutes sooner, but raises three false pages on healthy spiky traffic where multi-window raises zero. The trade-off is precision, not speed, and the benchmark proves it both ways.
-informational){kind=icon}
- SLO alerts must catch real regressions without paging on every transient spike, and a single threshold cannot do both; the measured result is that a short-window threshold false-pages three times on healthy spiky traffic.
- Multi-window multi-burn-rate alerting uses a fast and a slow window together, so a spike never sustains the slow window and a real burn clears both; it raises zero false pages on the same trace.
- The usual pitch for burn-rate alerting is "it detects faster", and this repo measures that claim honestly and finds it false for a well-tuned threshold; the real, defensible value is false-alarm suppression.
Every team that adopts SLOs writes an alert, and the first version is almost always "page me when the error rate is over 5% for 15 minutes". It works until the on-call gets paged at 3 a.m. for a two-minute blip that self-resolved, learns to ignore the alert, and then misses the real regression a month later. The single threshold is trying to serve two goals with one knob: shorten the window and it flaps, lengthen it and it is slow. Alert fatigue, not detection latency, is what actually kills SLO programs.
Multi-window multi-burn-rate alerting (from the Google SRE workbook) resolves the tension by requiring a fast window and a slow window to both exceed a burn-rate threshold before firing. The burn rate is computed from good/bad event counts, not averaged latency (ADR-002): a request is bad if it errored or crossed the latency threshold, and burn rate is the bad fraction divided by the error budget, so burn 14.4 means a 30-day budget is spent in about two days. Two tiers page on fast burns (14.4x on 5-minute and 1-hour windows) and ticket on slow burns (6x on 30-minute and 6-hour windows). The slow window is the false-positive filter; the fast window is the responsiveness.
The engine replays a seeded span trace minute by minute and evaluates both strategies, and the comparison is deliberately run across three regimes so the result is honest rather than cherry-picked: a severe regression, a slow ramp, and healthy-but-spiky traffic. The headline number is not detection speed (the naive threshold wins that); it is that multi-window raises zero false pages where the naive alert raises three.
Left: a severe regression at minute 120, both alerts fire almost immediately (naive one minute sooner). Right: healthy traffic with periodic two-minute spikes and no real regression, the naive threshold pages three times (red), multi-window never fires. Raw numbers in benchmark/results/comparison.json.
| Technology | Role in this project | Why chosen here |
|---|---|---|
| Python 3.11 + NumPy | Burn-rate math over span windows | Vectorized window aggregation; the full 6-hour replay runs in under a second |
| Good/bad event model | The SLI | Catches tail failures an averaged latency would hide (ADR-002) |
| Multi-window burn-rate | The alerting policy | The precision the benchmark measures; two tiers separate page from ticket |
| FastAPI | Span ingest + live /burn |
Same evaluator behind a service the way a collector sink would use it |
| Seeded 3-regime trace | Honest benchmark | Severe, slow, and spiky, so the comparison cannot be cherry-picked |
| pytest + pytest-cov | Suite | 10 tests including the false-alarm trade-off and the honest "naive is not slower" finding; 89 percent measured |
| GitHub Actions | CI | Lint, tests, and a replay that re-asserts the false-alarm invariant |
Prerequisites: Python 3.11+, git.
git clone https://github.com/<you>/otel-slo-burnmeter.git
cd otel-slo-burnmeter
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest --cov=burnmeter # test suite
for s in severe spikes slow; do python demo/generate_trace.py --seed 5 --scenario $s --out demo/trace_$s.json; done
python -c "
from burnmeter.config import Settings
from burnmeter.replay import load_trace, replay
for s in ('severe','spikes','slow'):
r = replay(load_trace(f'demo/trace_{s}.json'), Settings(_env_file=None))
print(s, 'mw_false', r['multiwindow_false_alarms'], 'naive_false', r['naive_false_alarms'])"Measured on the seeded traces (benchmark/results/comparison.json), SLO 99.5%, latency threshold 300ms:
| Regime | Multi-window detection lag | Naive detection lag | Multi-window false pages | Naive false pages |
|---|---|---|---|---|
| Severe regression | 8 min | 1 min | 0 | 0 |
| Slow ramp | 89 min | 52 min | 0 | 0 |
| Healthy spikes (no regression) | n/a (never fires) | n/a | 0 | 3 |
The honest reading: the naive threshold detects real regressions as fast or faster. Multi-window's win is the last row, three false pages eliminated. On a real on-call rotation, that is the difference between an alert people trust and one they mute.
ADR-001: multi-window over a single threshold, with the measured trade-off (precision bought at a few minutes of latency) stated plainly. ADR-002: burn rate from good/bad event counts rather than averaged latency, and why averages hide tail failures.
- A real OTLP receiver. The engine consumes span records; wiring an OpenTelemetry collector exporter to
/spansis mechanical and additive. - Alert routing (PagerDuty, Slack). The engine emits a tier decision; routing is a downstream concern.
- Adaptive thresholds. Burn thresholds are the workbook's fixed values; auto-tuning them per service is future work that needs historical incident labels.
Spans carry latency and an error flag, no payloads or user data. SLO parameters (target, threshold, window) are environment config and printed in the burn state. The engine is read-only over its ingested spans.
| Failure | Detection | Behaviour | Recovery |
|---|---|---|---|
| Transient spike on healthy traffic | Slow window stays under threshold | No page (the whole point) | Tested: 0 false pages |
| Severe sustained burn | Both windows clear threshold fast | Page tier fires | Tested; detection lag measured |
| Slow ramp regression | Ticket tier eventually fires | Ticket, not page (correct: slow burn) | Measured lag; naive is faster here and the README says so |
| Sparse traffic (empty window) | burn_rate returns 0 on no data | No spurious alert on a quiet minute | Guarded and tested |
| Latency threshold misconfigured | Burn rate shifts | Threshold printed in /burn |
Config knob; a product decision |
I set out to prove the standard claim that burn-rate alerting detects regressions earlier than a naive threshold, built the trace, ran the replay, and measured the opposite: the naive 5%-over-15-minutes alert fired at minute 172 on my slow ramp, while multi-window did not page until 209. My first instinct was that my windows were misconfigured, and I spent a while convinced there was a bug. There was not. A short averaging window genuinely detects a rising error rate faster than a burn-rate alert that has to accumulate signal in a one-hour window; that is arithmetic, not a defect.
The real bug was in my framing, not my code. I had picked the one scenario and the one naive tuning where the marketing claim happens to be defensible, and even there it was not true. The honest move was to test the regimes where the two strategies actually differ, which meant building three traces (severe, slow, spiky) and measuring both detection lag and false-alarm rate. That surfaced the finding that matters: on healthy spiky traffic the naive threshold raises three false pages and multi-window raises none, and a test now pins exactly that (test_spikes_multiwindow_suppresses_false_alarms), alongside a test that pins the uncomfortable truth (test_severe_naive_not_slower_than_multiwindow). The lesson I keep from it: when your measurement contradicts the claim you set out to prove, the measurement is usually right, and the interesting result is the one you did not expect. A benchmark that only tests the flattering case is marketing; testing the cases where your approach loses is what makes the case where it wins believable.
- OTLP receiver wiring so a real collector feeds
/spans. - Per-service burn thresholds learned from historical incident labels.
- A third alerting strategy (EWMA) in the same replay for a three-way comparison.
- Alert routing with dedup so the two tiers map to page vs ticket destinations.
- First metric to watch in adoption: pages-per-week and their acknowledged-actionable rate. The whole point is that the second number stays near 100 percent.
MIT