Skip to content

Commit 912f9fc

Browse files
agjsclaude
agjs
and
claude
committed
docs: update boringstack docs to reflect new observability surface
Catches the operator-facing docs site up with all the infra and correlation work that landed across PRs #45, #46, #47, and #48 — defaults flip + dashboards + structured logging + user/trace correlation + env-driven alert receivers. topics/observability.mdx - "What ships" — five dashboards (was one); Pino pipeline detail; Alertmanager wired with env-driven receivers (was "you wire it yourself"). - "Default dashboards" — full per-dashboard breakdown (API, API logs, UI logs, Postgres, Host) with what each is best at and a pointer to the click-through data links on trace_id / userId / requestId. - "Design choices" — added "on by default", trace_id flow, and env-driven Alertmanager entries; dropped the now-misleading "Prometheus + Loki, not OpenTelemetry" claim (the Sentry tracing side does carry traces). - "Querying" — LogQL examples now use the `level` label (was the broken `level=ERROR` substring match) and include trace_id / userId pivot queries. - "Adding an alert" — points at the single bundled rules.yml and the new Alerts topic. topics/error-tracking.mdx - New "Correlation: GlitchTip ↔ Loki ↔ requestId" section with a mermaid diagram of the trace_id flow (browser → API → Loki + GlitchTip), a description of what flows automatically (trace_id / span_id / userId / requestId), and the three click-through paths (Grafana → GlitchTip via dashboard data links, Grafana → Grafana via the requestId link, GlitchTip → Grafana via the operator-side one-time tag link template). - Design-choices block updated with on-by-default GlitchTip and the user.id / trace_id tagging. topics/alerts.mdx (new) - New topic page promoting the alerts walkthrough to the user-facing docs site. Covers what the 14 default rules fire on, the three receiver formats (Slack, Discord-via-/slack-suffix, generic webhook), an end-to-end curl one-liner to ping a fake alert, how to add rules, and the cost-of-pager-fatigue note. reference/env-vars.mdx - New entries for ALERTMANAGER_SLACK_WEBHOOK_URL, ALERTMANAGER_SLACK_CHANNEL, ALERTMANAGER_WEBHOOK_URL. reference/commands.mdx - "Hot-reload Prometheus rules" — `curl POST /-/reload`. - "Ping a fake alert (verify receiver wiring)" — the verification one-liner from the new Alerts topic. astro.config.mjs - Wire the new Alerts topic into the Topics sidebar between Observability and Provisioning with OpenTofu. Verification: `bun run build` clean, 66 pages built (was 65, now +1 for Alerts), pagefind index built without errors. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1 parent 1804744 commit 912f9fc

6 files changed

Lines changed: 371 additions & 47 deletions

File tree

‎apps/docs/astro.config.mjs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -463,6 +463,7 @@ export default defineConfig({
463463
{ label: "Cloudflare Email", link: "/topics/cloudflare-email/" },
464464
{ label: "Error tracking", link: "/topics/error-tracking/" },
465465
{ label: "Observability", link: "/topics/observability/" },
466+
{ label: "Alerts", link: "/topics/alerts/" },
466467
{
467468
label: "Provisioning with OpenTofu",
468469
link: "/topics/provisioning-with-tofu/",

‎apps/docs/src/content/docs/reference/commands.mdx‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,15 @@ commands run from `infra/compose/compose`; operational helper scripts live in
9494
<FaqItem title="Backup db (offsite)">
9595
`BACKUP_DRY_RUN=1 ./scripts/backup-wrapper.example.sh` then drop the flag. Data preserved.
9696
</FaqItem>
97+
<FaqItem title="Hot-reload Prometheus rules">
98+
`curl -X POST http://localhost:9090/-/reload`. Re-reads `prometheus.yml` + `rules.yml` without restarting the container.
99+
</FaqItem>
100+
<FaqItem title="Ping a fake alert (verify receiver wiring)">
101+
```bash
102+
curl -XPOST http://localhost:9093/api/v2/alerts -H 'Content-Type: application/json' -d '[{"labels":{"alertname":"TestPing","severity":"warn","component":"manual-test"},"annotations":{"summary":"Manual ping"}}]'
103+
```
104+
Auto-resolves after 5min. See [Alerts](/topics/alerts/).
105+
</FaqItem>
97106
</FaqGroup>
98107

99108
Minimal opt-out: `WITH_OBSERVABILITY=0 WITH_GLITCHTIP=0 ./scripts/compose-up.sh` boots just Postgres + Valkey + your apps.

‎apps/docs/src/content/docs/reference/env-vars.mdx‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -331,6 +331,15 @@ import FaqItem from "../../../components/FaqItem.tsx";
331331
<FaqItem title="GRAFANA_ADMIN_USER / GRAFANA_ADMIN_PASSWORD">
332332
**Repo:** infra. **Required:** rotate before exposing Grafana beyond localhost. Observability is on by default.
333333
</FaqItem>
334+
<FaqItem title="ALERTMANAGER_SLACK_WEBHOOK_URL">
335+
**Repo:** infra. **Required:** no. Slack-format webhook URL (or a Discord webhook URL with `/slack` appended). When set, Alertmanager delivers rule fires to it. Unset → alerts surface in the UI at `:9093` only. See [Alerts](/topics/alerts/).
336+
</FaqItem>
337+
<FaqItem title="ALERTMANAGER_SLACK_CHANNEL">
338+
**Repo:** infra. **Required:** no. Default `#alerts`. Ignored by Discord (channel is fixed when the webhook is created).
339+
</FaqItem>
340+
<FaqItem title="ALERTMANAGER_WEBHOOK_URL">
341+
**Repo:** infra. **Required:** no. Generic HTTP endpoint that receives Alertmanager's native JSON payload — for custom bridges (n8n, alerter services, PagerDuty translators). Can be set alongside the Slack URL; alerts fan out to both.
342+
</FaqItem>
334343
<FaqItem title="*_LIMITS_CPUS / *_LIMITS_MEMORY (and reservations)">
335344
**Repo:** infra. **Required:** no. Per-service resource caps; see [Resource
336345
limits](/infra/resource-limits/).

‎apps/docs/src/content/docs/topics/alerts.mdx‎

Lines changed: 188 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,188 @@
1+
---
2+
title: Alerts
3+
description: 14 default Prometheus rules + env-driven Slack, Discord, and webhook receivers in Alertmanager. Set one env var to get pager output, none to keep alerts in the UI.
4+
---
5+
6+
import PageIntro from "../../../components/docs-kit/PageIntro";
7+
import FaqGroup from "../../../components/FaqGroup.tsx";
8+
import FaqItem from "../../../components/FaqItem.tsx";
9+
import DocCallout from "../../../components/DocCallout.tsx";
10+
11+
<PageIntro
12+
eyebrow="Alerts"
13+
actions={[
14+
{ label: "What fires", href: "#what-rules-ship" },
15+
{ label: "Receivers", href: "#wiring-a-receiver" },
16+
]}
17+
facts={[
18+
{ value: "14", label: "default rules" },
19+
{ value: "3", label: "receiver formats" },
20+
{ value: "1", label: "env var to wire" },
21+
]}
22+
>
23+
Prometheus alert rules + Alertmanager receivers ship with the
24+
observability stack. Set `ALERTMANAGER_SLACK_WEBHOOK_URL` in
25+
`compose/.env` and alerts deliver to Slack, Discord, or any
26+
Slack-compatible endpoint. Without it, alerts surface in the
27+
Alertmanager UI at `:9093` only — useful for confirming rules fire
28+
before committing to a pager destination.
29+
</PageIntro>
30+
31+
## What rules ship
32+
33+
`compose/prometheus/rules.yml` defines four rule groups covering the
34+
obvious failure modes. Severity labels drive Alertmanager's re-notify
35+
cadence: `severity=page` repeats hourly, `severity=warn` every 12
36+
hours.
37+
38+
<FaqGroup>
39+
<FaqItem title="boringstack-api" open>
40+
`ApiServerErrorsHigh` (page, 5xx > 1% for 5min),
41+
`ApiServerErrorsCritical` (page, 5xx > 5% for 2min),
42+
`ApiLatencyP95High` (warn, p95 > 1s for 10min),
43+
`ApiUnreachable` (page, no API traffic for 5min).
44+
</FaqItem>
45+
<FaqItem title="boringstack-database">
46+
`PostgresDown` (page, exporter can't reach pg for 2min),
47+
`PostgresConnectionsHigh` (warn, > 80% of `max_connections` for 5min),
48+
`PostgresReplicationLagHigh` (warn, lag > 60s for 5min).
49+
</FaqItem>
50+
<FaqItem title="boringstack-host">
51+
`DiskSpaceLow` (warn, < 10% free for 5min),
52+
`DiskSpaceCritical` (page, < 5% free for 2min),
53+
`MemoryPressureHigh` (warn, available < 10% for 10min),
54+
`HostCpuSaturated` (warn, load5/cores > 1.5 for 15min),
55+
`NodeExporterDown` (warn, host metrics stale for 3min).
56+
</FaqItem>
57+
<FaqItem title="boringstack-edge">
58+
`TraefikDown` (page, Traefik metrics endpoint unreachable for 2min) —
59+
Traefik is the only path into the stack in prod; this firing means
60+
the site is down.
61+
</FaqItem>
62+
</FaqGroup>
63+
64+
Thresholds are conservative defaults for a single-host BoringStack
65+
deployment. Tune per workload — a busy app may legitimately push
66+
4xx/5xx absolute counts above what a quiet baseline catches.
67+
68+
## Wiring a receiver
69+
70+
Alertmanager has no native env-var substitution in its config, so the
71+
stack ships a small `entrypoint.sh` that renders alertmanager.yml
72+
from `ALERTMANAGER_*` env vars at container boot. Empty env → that
73+
block is omitted entirely, so `amtool check-config` stays happy.
74+
75+
<FaqGroup>
76+
<FaqItem title="Slack" open>
77+
Create a Slack app + Incoming Webhook
78+
([guide](https://api.slack.com/messaging/webhooks)). Copy the URL,
79+
then in `compose/.env`:
80+
```env
81+
ALERTMANAGER_SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T.../B.../...
82+
ALERTMANAGER_SLACK_CHANNEL=#alerts
83+
```
84+
Restart: `docker compose restart alertmanager`. Default message
85+
template renders status, severity, component, summary, description.
86+
</FaqItem>
87+
<FaqItem title="Discord">
88+
Discord accepts Slack-format payloads at a `/slack` suffix on its
89+
webhook URL — so the **same env var** delivers there:
90+
1. In Discord: `Server Settings → Integrations → Webhooks → New
91+
Webhook`. Copy the URL.
92+
2. **Append `/slack`**: `https://discord.com/api/webhooks/.../slack`.
93+
3. Set `ALERTMANAGER_SLACK_WEBHOOK_URL` to that value in
94+
`compose/.env`. `ALERTMANAGER_SLACK_CHANNEL` is ignored — the
95+
channel is fixed when the webhook is created.
96+
4. Restart Alertmanager.
97+
</FaqItem>
98+
<FaqItem title="Generic webhook (Alertmanager JSON)">
99+
For custom bridges (n8n, your own alerter, PagerDuty events-v2
100+
translators), use the generic receiver. It POSTs Alertmanager's
101+
native JSON — different from Slack's format.
102+
```env
103+
ALERTMANAGER_WEBHOOK_URL=https://your-bridge.example.com/alerts
104+
```
105+
Can be set in addition to the Slack one; alerts fan out to both.
106+
</FaqItem>
107+
<FaqItem title="None (UI only)">
108+
Leave both env vars unset. The 14 rules still fire — they land in
109+
the Alertmanager UI at `http://localhost:9093`. Useful for seeing
110+
what alerts look like before deciding where to pager them.
111+
</FaqItem>
112+
</FaqGroup>
113+
114+
## Verifying receivers work
115+
116+
Curl a fake firing alert into Alertmanager's API directly, no waiting
117+
for a real incident:
118+
119+
```bash
120+
curl -XPOST http://localhost:9093/api/v2/alerts \
121+
-H 'Content-Type: application/json' \
122+
-d '[{
123+
"labels": {
124+
"alertname": "TestPing",
125+
"severity": "warn",
126+
"component": "manual-test"
127+
},
128+
"annotations": {
129+
"summary": "Manual ping from the alerts docs",
130+
"description": "If this lands in your Slack/Discord/webhook, the receiver is wired correctly."
131+
}
132+
}]'
133+
```
134+
135+
The alert auto-resolves after `resolve_timeout` (5 minutes) since no
136+
follow-up ping keeps it firing.
137+
138+
## Adding rules
139+
140+
Drop new entries into `compose/prometheus/rules.yml` under an
141+
existing group (or create a new group). Required fields per alert:
142+
`alert`, `expr`, `for` (optional debounce duration), `labels.severity`
143+
(`page` or `warn`), `annotations.summary`. The 14 bundled rules are
144+
worked examples.
145+
146+
Hot-reload Prometheus without a restart:
147+
148+
```bash
149+
curl -X POST http://localhost:9090/-/reload
150+
```
151+
152+
## Adding routes
153+
154+
Out of the box every alert lands in the single `default` receiver. To
155+
route specific labels to different receivers (e.g. `component=database`
156+
to a DBA channel), edit `compose/alertmanager/entrypoint.sh` directly —
157+
it's a small shell script with `cat >> "$CFG" <<EOF` blocks for each
158+
receiver. Append your own routes block + matching receivers.
159+
160+
If the routing grows beyond a handful of cases, graduate to a static
161+
`alertmanager.yml` you maintain yourself and remove the entrypoint
162+
script.
163+
164+
<DocCallout
165+
variant="tip"
166+
title="Pager fatigue is a real cost"
167+
eyebrow="Tuning thresholds"
168+
>
169+
The default `for:` durations are deliberately generous to avoid
170+
3am pages on transient blips. If you find you're regularly silencing
171+
the same alert, the threshold is wrong — either raise the duration,
172+
the rate threshold, or both. An alert that fires and nobody acts on
173+
is worse than no alert.
174+
</DocCallout>
175+
176+
## Source
177+
178+
- [`compose/prometheus/rules.yml`](https://github.com/boringstack-xyz/boringstack/blob/main/infra/compose/compose/prometheus/rules.yml) — the 14 rules.
179+
- [`compose/alertmanager/entrypoint.sh`](https://github.com/boringstack-xyz/boringstack/blob/main/infra/compose/compose/alertmanager/entrypoint.sh) — env-driven config renderer.
180+
- [`infra/compose/docs/alerts.md`](https://github.com/boringstack-xyz/boringstack/blob/main/infra/compose/docs/alerts.md) — the operator-side runbook.
181+
182+
## Related
183+
184+
- [Observability](/topics/observability/) — the metrics + logs stack
185+
these alerts run inside of.
186+
- [Error tracking](/topics/error-tracking/) — Sentry/GlitchTip catches
187+
exceptions; Alertmanager catches "metrics-shaped" failures (rates,
188+
resource pressure, unreachability). Different surfaces, same goal.

‎apps/docs/src/content/docs/topics/error-tracking.mdx‎

Lines changed: 60 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -39,21 +39,77 @@ The SDKs don't know which one they're talking to. Choosing is a one-env-var chan
3939
<FaqItem title="Sentry SDK on both sides (not a custom shim)" open>
4040
Same SDK interface and error format across both services.
4141
</FaqItem>
42-
<FaqItem title="GlitchTip as the self-host option">
43-
Wire-compatible with Sentry; runs on the same Postgres + Valkey the app
44-
already uses.
42+
<FaqItem title="GlitchTip as the self-host option, on by default">
43+
Wire-compatible with Sentry; runs on the same Postgres + Valkey the
44+
app already uses. `WITH_GLITCHTIP=0` opts out.
4545
</FaqItem>
4646
<FaqItem title="Empty DSN: SDK is a no-op">
47-
Dev stays clean; tests do not ship error reports.
47+
Tests do not ship error reports. Once you copy a project DSN from
48+
the GlitchTip UI into `SENTRY_DSN` / `VITE_SENTRY_DSN`, error
49+
capture starts.
4850
</FaqItem>
4951
<FaqItem title="Replay-on-error on, full-session replays off">
5052
Captures the broken flow without storing healthy sessions.
5153
</FaqItem>
5254
<FaqItem title="Single env var per side: SENTRY_DSN (API), VITE_SENTRY_DSN (UI)">
5355
Same protocol regardless of backend; swap is one redeploy.
5456
</FaqItem>
57+
<FaqItem title="Events carry user.id + trace_id automatically">
58+
The API's auth plugin calls `Sentry.setUser({id, email})` once it
59+
resolves a session; the UI's `SentryUserSync` does the same after
60+
every `useMe` change. Errors land in GlitchTip tagged with the
61+
user that hit them. See "Correlation" below for the full pivot
62+
surface.
63+
</FaqItem>
5564
</FaqGroup>
5665

66+
## Correlation: GlitchTip ↔ Loki ↔ requestId
67+
68+
Errors are useful in proportion to how easily you can reach the
69+
context around them. The stack wires a single chain of IDs across
70+
browser, API, log store, and error tracker so any one of them
71+
takes you to the other three.
72+
73+
```mermaid
74+
flowchart LR
75+
user["Browser action"] -->|sentry-trace header| api["API request handler"]
76+
api -->|Pino mixin| logs["Loki log lines<br/>(trace_id, requestId, userId)"]
77+
api -->|on error| gt["GlitchTip event<br/>(tagged: trace_id, user.id)"]
78+
logs -.->|click trace_id link| gt
79+
gt -.->|click trace_id tag| logs
80+
```
81+
82+
**What flows automatically:**
83+
84+
- `trace_id` + `span_id`: the UI's Sentry SDK adds `sentry-trace` /
85+
`traceparent` headers to every `/api/*` fetch via
86+
`browserTracingIntegration`; the API's Sentry init reads them; the
87+
Pino logger's mixin injects them on every log record; Promtail
88+
promotes them to Loki structured metadata.
89+
- `userId`: the API's auth plugin calls `Sentry.setUser({id, email})`
90+
on every authenticated request. The Pino mixin reads `userId` from
91+
the Sentry scope and emits it on each log line. The UI's
92+
`SentryUserSync` provider mirrors the same call after login, MFA
93+
verify, account switch, page reload — anywhere `useMe` resolves.
94+
- `requestId`: a UUID assigned by the API's request-logger middleware,
95+
returned as the `x-request-id` header and present on every log line
96+
for that request.
97+
98+
**Where the click-through lives:**
99+
100+
- **Grafana → GlitchTip** — the "BoringStack — API logs" dashboard
101+
defines clickable data links on `trace_id` and `userId` in the live
102+
log panel. Expand any log line, click the field → opens GlitchTip
103+
filtered to that trace / user in a new tab. The dashboard variables
104+
`$glitchtip_url` and `$glitchtip_org` configure the target (defaults
105+
fit the bundled GlitchTip; edit once per environment).
106+
- **Grafana → Grafana** — `requestId` link opens Explore with a Loki
107+
query pre-filtered to that one request's lines.
108+
- **GlitchTip → Grafana** — configure once in GlitchTip's project
109+
settings (UI, not code): a `tags.trace_id` external link template
110+
pointing at Grafana Explore with `{compose_service="api-dev"} | json | trace_id="<value>"`.
111+
Each event detail then shows a "View logs" link.
112+
57113
## How it's wired
58114

59115
```mermaid

0 commit comments

Comments
 (0)