Skip to content

feat(fleet-report): per-site error budgets from drosera's SLO queries - #290

Open
lentago-claude-runner[bot] wants to merge 1 commit into
mainfrom
agent/fleet-report-error-budgets
Open

lentago-claude-runner[bot] wants to merge 1 commit into
mainfrom
agent/fleet-report-error-budgets

Conversation

@lentago-claude-runner

Copy link
Copy Markdown
Contributor

Closes lentago/drosera#199

What it adds

A Site error budgets — last 30 days section in fleet-reports/fleet-report.md, between Activity and the Code census. The weekly fleet-reports.yml run produces it. There is one row for each SLO site in drosera ADR-0008 (lentago.dev, icecreamtofightwith.com, pondviewlane.com), with:

  • the SLO target (99.9%)
  • 30-day measured availability
  • budget used, as a percentage
  • budget left, in minutes (marked when overspent)
  • burn alerts in the window: how many separate stretches the page rule's condition held (14.4× over 1h ∧ 5m) and how many the ticket rule's held (3× over 24h ∧ 2h), with the date of the most recent

The epics and incidents reports are untouched.

Data path

  • Queries. No new PromQL. The section sends drosera's own expressions:
    • the three stat-panel expressions from dashboards/sites-slo-error-budget.json (attainment, budget remaining, budget minutes), as instant queries at the generation time;
    • the burn-rate expression from terraform/alerts.tf ("Site SLO burn rate"), as 30-day range queries over the long and short windows. The generator then applies the rules' own A > t ∧ B > t per step.
  • Endpoint. The queries go to https://lentago.grafana.net/api/datasources/proxy/uid/grafanacloud-prom/api/v1/{query,query_range}. That is the stack's Prometheus-compatible API, reached through Grafana's data-source proxy, on the same grafanacloud-prom data source the alert rules name.
  • Burn-alert approximation. The rules are re-evaluated at 5-minute (page) and 1-hour (ticket) steps rather than the rules' 60 s, to keep the range queries cheap: about 0.5M and 1M samples per site. Their for delays are not applied, so the column counts stretches where the condition held, not notifications sent. The report states this under the table. Grafana's alert state-history API would be exact, but it needs per-rule UIDs and its Cloud backend could not be checked from here, so I left it out.
  • Failure handling. With no token, the section is one line, Not configured: the FLEET_REPORTS_GRAFANA_TOKEN secret is unset…, and the run stays green. A token that is set but fails (401, unreachable, rejected query) renders Unavailable this run: HTTP 401… plus a ::warning:: in the run, and the rest of the report publishes as normal. Error text never includes the request, so the token cannot reach the CI log.
  • New budget check in ci/validate.py. It pins the expressions verbatim against drosera's text for lentago.dev, tests the burn-stretch counter (including gt-not-ge), and tests the one-line no-token path. I confirmed it can fail by breaking each of the three in turn; each run went red, then green again once restored.

Secret the maintainer must create

FLEET_REPORTS_GRAFANA_TOKEN is a Grafana service-account token (glsa_…) under an account with the built-in Viewer role. Viewer can query data sources and change nothing. This follows ADR-0011 rule 4 and drosera's service_accounts.tf pattern: Terraform holds the slot, and the value is minted by hand. The full steps are in the header of .github/workflows/fleet-reports.yml:

  1. In lentago/drosera, add grafana_service_account "fleet_reports_viewer" (name = "fleet-reports-viewer", role = "Viewer") to terraform/service_accounts.tf, plus its row in terraform/README.md § Service accounts. Merge so CI applies it. That drosera change is not in this PR.
  2. In the stack UI, open the account, choose Add service account token, and copy the value.
  3. Run gh secret set FLEET_REPORTS_GRAFANA_TOKEN --repo lentago/.github and paste the value at the prompt. If you set an expiry, record it in the credential register (Credential register: every fleet credential's store, scope, issuer and expiry (names only) #243).
  4. Run the workflow by hand (workflow_dispatch) and check that the refresh PR shows the table, not the note (ADR-0011 rule 5).

The secret goes only to the "Regenerate reports" step.

Testing

  • python3 ci/validate.py: all checks pass.
  • I ran the section against a local fake of the Prometheus API. It rendered correctly with no token, with a 401, with an unreachable host, with no-data sites, with an overspent budget, and with multiple series per job (the worst series is reported).
  • Not tested against live Grafana Cloud. No token is available to this runner, so step 4 above is the first real run.

🤖 Generated with Claude Code

Adds a "Site error budgets — last 30 days" section to fleet-report.md for
the three SLO sites in drosera ADR-0008: target, 30-day availability,
budget used, budget left in minutes, and burn-rule stretches in the
window. The PromQL is drosera's SLO board and burn rules verbatim, sent
to the stack's grafanacloud-prom data source through Grafana's
data-source proxy.

The token is an optional new repo secret, FLEET_REPORTS_GRAFANA_TOKEN
(Viewer service account). Unset, the section is one "not configured"
line and the run stays green; a failing token renders an error note and
a workflow warning. ci/validate.py gains a `budget` check pinning the
expressions, the burn-stretch counter and the no-token path.

Refs lentago/drosera#199

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Review (haiku, claude-review.yml)

Review of PR #290: per-site error budgets in the fleet report

  1. Minor. PR body, Closes lentago/drosera#199. Merging auto-closes drosera#199, but the PR body says the required drosera change (the fleet_reports_viewer service account) is "not in this PR." The issue would close while the work it tracks is still open. The closing reference is also cross-repo, not the in-repo Closes #<n> form the fleet convention uses. Consider linking drosera#199 without a closing keyword, or closing it once the drosera change lands.

  2. Minor. metrics/generate-fleet-reports.py:498. The section says "The probes run from our own building." The same file's header (line 530) states the report includes no on-prem-internal detail and calls out "the machines in our own building." Naming the probe vantage point publishes the on-prem detail the report says it leaves out. Either drop the sentence or reconcile it with the stated policy.

  3. Minor. metrics/generate-fleet-reports.py:41 and :169. FLEET_REPORTS_GRAFANA_TOKEN is set on the whole "Regenerate reports" step, so every subprocess.run (git, cloc) inherits it through os.environ. Only prom_get needs the token. Passing a scrubbed environment to the child processes would keep the Grafana token away from the tools that clone and count repos.

Not verified in this review: python3 ci/validate.py could not be run here, and the drosera PromQL and burn rules could not be fetched to confirm the "verbatim" claim. The budget check's pinned expressions and burn-episode test cases were checked by hand against the generator and are consistent with it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Error-budget monthly section in the fleet report

0 participants