diff --git a/README.md b/README.md index 83978a7..df30239 100644 --- a/README.md +++ b/README.md @@ -53,9 +53,19 @@ path. See [`docs/reference/cli.md`](docs/reference/cli.md) for `promforecast val which CI users can run against a config file before merging. See [`docs/`](docs/) for configuration, model selection, and operational -guidance. For a real-world deployment — multi-group config, HA values, and -ready-to-apply ArgoCD/Flux manifests with a reference architecture diagram -— see [`examples/production/`](examples/production/). +guidance. Before flipping production traffic, walk +[`docs/production-checklist.md`](docs/production-checklist.md) — sizing +tables, reference architectures, and the boxes worth ticking off +before going live. For a real-world deployment — multi-group config, +HA values, and ready-to-apply ArgoCD/Flux manifests with a reference +architecture diagram — see [`examples/production/`](examples/production/). + +New to forecasting Prometheus metrics? The +[`docs/cookbook/`](docs/cookbook/) collection covers the common +metric families (disk-fill, request rate, Kafka lag, JVM GC, AWS +spend, ...) with copy-pasteable configs sized for a small fleet and +companion alert snippets — the fastest path from "I want to forecast +X" to "X is on a dashboard with an alert". ## Status diff --git a/charts/promforecast-stack/templates/prometheusrule.yaml b/charts/promforecast-stack/templates/prometheusrule.yaml index b5ba0bf..d06ea03 100644 --- a/charts/promforecast-stack/templates/prometheusrule.yaml +++ b/charts/promforecast-stack/templates/prometheusrule.yaml @@ -25,6 +25,22 @@ spec: {{`{{ $labels.group }}`}} in over 2 hours. Check pod logs and datasource availability. + - alert: ForecastSeriesStale + expr: forecast_data_staleness_seconds > 14400 + for: 15m + labels: + severity: warning + annotations: + summary: "Forecast {{`{{ $labels.id }}`}} model {{`{{ $labels.model }}`}} on {{`{{ $labels.instance }}`}} hasn't been refit for >4h" + description: | + The (series, model) tuple for {{`{{ $labels.id }}`}} on group + {{`{{ $labels.group }}`}} hasn't produced a successful forecast + for over four hours. The group-level ForecastStale alert stays + quiet because other series in the group are still being fit + successfully; this rule catches the silent per-series failure + (model crash, persistent fit timeout, data gap that drops the + series under min_points). + - alert: ForecastModelDegraded expr: forecast_accuracy_mape > 30 for: 24h @@ -49,7 +65,7 @@ spec: the forecast is doing no better than a seasonal naive baseline. - alert: ForecastFallbackPersistent - expr: count by (group) (forecast_quality_score{model_fallback="true"}) > 0 + expr: count by (group) (forecast_quality_score{model_fallback="true", horizon=""}) > 0 for: 6h labels: severity: info @@ -61,7 +77,7 @@ spec: - alert: ForecastQualityLow expr: | - avg by (group) (forecast_quality_score{cold_start!="true"}) < 0.4 + avg by (group) (forecast_quality_score{cold_start!="true", horizon=""}) < 0.4 for: 6h labels: severity: info @@ -73,7 +89,7 @@ spec: downstream forecast-based alerts with caution. - alert: ForecastQualityZero - expr: forecast_quality_score{cold_start!="true"} == 0 + expr: forecast_quality_score{cold_start!="true", horizon=""} == 0 for: 1h labels: severity: info @@ -103,7 +119,7 @@ spec: - alert: ForecastDeviation expr: | forecast_deviation_outside_band == 1 - and on (id, group, model) forecast_quality_score > 0.6 + and on (id, group, model) forecast_quality_score{horizon=""} > 0.6 for: 15m labels: severity: warning @@ -117,7 +133,7 @@ spec: - alert: ForecastDeviationSevere expr: | abs(forecast_deviation_ratio) > 2 - and on (id, group, model) forecast_quality_score > 0.6 + and on (id, group, model) forecast_quality_score{horizon=""} > 0.6 for: 5m labels: severity: critical diff --git a/charts/promforecast-tuner/Chart.yaml b/charts/promforecast-tuner/Chart.yaml new file mode 100644 index 0000000..a3a948d --- /dev/null +++ b/charts/promforecast-tuner/Chart.yaml @@ -0,0 +1,22 @@ +apiVersion: v2 +name: promforecast-tuner +description: Weekly CronJob that surfaces hyperparameter recommendations for promforecast configs. +type: application +version: 0.1.0 +appVersion: "0.1.0" +kubeVersion: ">=1.27.0-0" +home: https://github.com/esops-dev/promforecast +sources: + - https://github.com/esops-dev/promforecast +keywords: + - prometheus + - forecasting + - hyperparameter-tuning + - cronjob +maintainers: + - name: esops-dev + url: https://github.com/esops-dev +icon: https://raw.githubusercontent.com/esops-dev/promforecast/main/docs/assets/logo.png +annotations: + artifacthub.io/license: Apache-2.0 + artifacthub.io/category: monitoring diff --git a/charts/promforecast-tuner/README.md b/charts/promforecast-tuner/README.md new file mode 100644 index 0000000..94ee3f8 --- /dev/null +++ b/charts/promforecast-tuner/README.md @@ -0,0 +1,31 @@ +# promforecast-tuner + +Weekly Kubernetes `CronJob` that surfaces hyperparameter recommendations for the configured promforecast groups. Pushes one `forecast_recommended_param` sample per candidate that beats the configured baseline by `--min-improvement-pct` (default 5%). + +The tuner reuses the live forecaster's container image (`promforecast tune ...` is a one-shot subcommand) and reads the same YAML config the live runner uses, so the baseline `(model, season_length)` values match exactly. + +## Install + +```bash +helm install promforecast-tuner \ + oci://ghcr.io/esops-dev/promforecast/charts/promforecast-tuner \ + --namespace promforecast \ + --set configMap.name=promforecast-config +``` + +## Values + +See `values.yaml` for the full set of knobs and their defaults. The common ones: + +| Key | Default | Description | +|---|---|---| +| `schedule` | `0 2 * * 0` | Cron expression. Weekly Sunday 02:00 UTC by default. | +| `configMap.name` | `promforecast` | Existing ConfigMap carrying the forecaster YAML. Defaults to the ConfigMap emitted by `helm install promforecast charts/promforecast`; override for other install shapes (umbrella: `-promforecast`). | +| `sinkUrlOverride` | `""` | Override `sink.remote_write.url`. Inherits from the config when empty. | +| `minImprovementPct` | `5.0` | Minimum MAPE improvement to surface a candidate. | +| `lookback` | `null` | Prometheus duration override (e.g. `14d`). | +| `groups` | `[]` | Subset of groups to tune. Empty = all. | + +## How to read the recommendations + +See [`docs/operations/auto-tuning.md`](../../docs/operations/auto-tuning.md) for the "should I apply this recommendation?" decision guide. diff --git a/charts/promforecast-tuner/templates/_helpers.tpl b/charts/promforecast-tuner/templates/_helpers.tpl new file mode 100644 index 0000000..baf4eba --- /dev/null +++ b/charts/promforecast-tuner/templates/_helpers.tpl @@ -0,0 +1,51 @@ +{{/* +Common helpers for the promforecast-tuner chart. +*/}} + +{{- define "promforecast-tuner.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} +{{- end -}} + +{{- define "promforecast-tuner.fullname" -}} +{{- if .Values.fullnameOverride -}} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}} +{{- else -}} +{{- $name := default .Chart.Name .Values.nameOverride -}} +{{- if contains $name .Release.Name -}} +{{- .Release.Name | trunc 63 | trimSuffix "-" -}} +{{- else -}} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} +{{- end -}} +{{- end -}} +{{- end -}} + +{{- define "promforecast-tuner.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}} +{{- end -}} + +{{- define "promforecast-tuner.labels" -}} +helm.sh/chart: {{ include "promforecast-tuner.chart" . }} +{{ include "promforecast-tuner.selectorLabels" . }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +app.kubernetes.io/part-of: promforecast +app.kubernetes.io/component: tuner +{{- end -}} + +{{- define "promforecast-tuner.selectorLabels" -}} +app.kubernetes.io/name: {{ include "promforecast-tuner.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end -}} + +{{- define "promforecast-tuner.serviceAccountName" -}} +{{- if .Values.serviceAccount.create -}} +{{- default (include "promforecast-tuner.fullname" .) .Values.serviceAccount.name -}} +{{- else -}} +{{- default "default" .Values.serviceAccount.name -}} +{{- end -}} +{{- end -}} + +{{- define "promforecast-tuner.image" -}} +{{- $tag := default .Chart.AppVersion .Values.image.tag -}} +{{- printf "%s:%s" .Values.image.repository $tag -}} +{{- end -}} diff --git a/charts/promforecast-tuner/templates/cronjob.yaml b/charts/promforecast-tuner/templates/cronjob.yaml new file mode 100644 index 0000000..de6c874 --- /dev/null +++ b/charts/promforecast-tuner/templates/cronjob.yaml @@ -0,0 +1,85 @@ +{{- if .Values.enabled }} +apiVersion: batch/v1 +kind: CronJob +metadata: + name: {{ include "promforecast-tuner.fullname" . }} + labels: + {{- include "promforecast-tuner.labels" . | nindent 4 }} +spec: + schedule: {{ .Values.schedule | quote }} + concurrencyPolicy: {{ .Values.concurrencyPolicy }} + successfulJobsHistoryLimit: {{ .Values.successfulJobsHistoryLimit }} + failedJobsHistoryLimit: {{ .Values.failedJobsHistoryLimit }} + {{- if .Values.startingDeadlineSeconds }} + startingDeadlineSeconds: {{ .Values.startingDeadlineSeconds }} + {{- end }} + jobTemplate: + spec: + activeDeadlineSeconds: {{ .Values.activeDeadlineSeconds }} + backoffLimit: {{ .Values.backoffLimit }} + template: + metadata: + labels: + {{- include "promforecast-tuner.selectorLabels" . | nindent 12 }} + spec: + serviceAccountName: {{ include "promforecast-tuner.serviceAccountName" . }} + # The tuner needs no K8s API access — it reads from VM via PromQL + # and pushes to the configured sink. Disabling the API token mount + # keeps the principle of least privilege in shops where the default + # ServiceAccount has cluster-wide read by accident. + automountServiceAccountToken: false + restartPolicy: {{ .Values.restartPolicy }} + {{- with .Values.securityContext }} + securityContext: + {{- toYaml . | nindent 12 }} + {{- end }} + {{- with .Values.nodeSelector }} + nodeSelector: + {{- toYaml . | nindent 12 }} + {{- end }} + {{- with .Values.tolerations }} + tolerations: + {{- toYaml . | nindent 12 }} + {{- end }} + {{- with .Values.affinity }} + affinity: + {{- toYaml . | nindent 12 }} + {{- end }} + containers: + - name: tuner + image: {{ include "promforecast-tuner.image" . | quote }} + imagePullPolicy: {{ .Values.image.pullPolicy }} + args: + - "tune" + - "--config=/etc/promforecast/{{ .Values.configMap.key }}" + - "--min-improvement-pct={{ .Values.minImprovementPct }}" + {{- if .Values.sinkUrlOverride }} + - "--sink-url={{ .Values.sinkUrlOverride }}" + {{- end }} + {{- if .Values.lookback }} + - "--lookback={{ .Values.lookback }}" + {{- end }} + {{- with .Values.groups }} + - "--groups" + {{- range . }} + - {{ . | quote }} + {{- end }} + {{- end }} + {{- with .Values.extraEnv }} + env: + {{- toYaml . | nindent 16 }} + {{- end }} + volumeMounts: + - name: config + mountPath: /etc/promforecast + readOnly: true + resources: + {{- toYaml .Values.resources | nindent 16 }} + volumes: + - name: config + configMap: + name: {{ .Values.configMap.name | quote }} + items: + - key: {{ .Values.configMap.key | quote }} + path: {{ .Values.configMap.key | quote }} +{{- end }} diff --git a/charts/promforecast-tuner/templates/serviceaccount.yaml b/charts/promforecast-tuner/templates/serviceaccount.yaml new file mode 100644 index 0000000..701946a --- /dev/null +++ b/charts/promforecast-tuner/templates/serviceaccount.yaml @@ -0,0 +1,12 @@ +{{- if .Values.serviceAccount.create }} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ include "promforecast-tuner.serviceAccountName" . }} + labels: + {{- include "promforecast-tuner.labels" . | nindent 4 }} + {{- with .Values.serviceAccount.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +{{- end }} diff --git a/charts/promforecast-tuner/values.schema.json b/charts/promforecast-tuner/values.schema.json new file mode 100644 index 0000000..2299d56 --- /dev/null +++ b/charts/promforecast-tuner/values.schema.json @@ -0,0 +1,84 @@ +{ + "$schema": "https://json-schema.org/draft-07/schema#", + "$id": "https://github.com/esops-dev/promforecast/charts/promforecast-tuner/v1", + "title": "promforecast-tuner Helm values", + "description": "Schema for the promforecast-tuner sub-chart values. Surfaces the knobs that drive the weekly hyperparameter-recommendation CronJob; the tuner reuses the live forecaster's container image so image / resource shapes mirror the main chart.", + "type": "object", + "additionalProperties": true, + "properties": { + "image": { + "type": "object", + "additionalProperties": false, + "properties": { + "repository": { "type": "string", "minLength": 1 }, + "tag": { "type": ["string", "null"] }, + "pullPolicy": { "type": "string", "enum": ["Always", "IfNotPresent", "Never"] } + }, + "required": ["repository"] + }, + "enabled": { "type": "boolean" }, + "schedule": { + "type": "string", + "description": "Cron expression for the CronJob. Defaults to weekly (Sunday 02:00 UTC)." + }, + "configMap": { + "type": "object", + "additionalProperties": false, + "properties": { + "name": { "type": "string", "minLength": 1 }, + "key": { "type": "string", "minLength": 1 } + }, + "required": ["name", "key"] + }, + "sinkUrlOverride": { "type": "string" }, + "minImprovementPct": { "type": "number", "minimum": 0 }, + "lookback": { + "type": ["string", "null"], + "description": "Prometheus duration override for the lookback (e.g. ``14d``)." + }, + "groups": { + "type": "array", + "items": { "type": "string", "minLength": 1 } + }, + "resources": { "type": "object" }, + "concurrencyPolicy": { + "type": "string", + "enum": ["Allow", "Forbid", "Replace"] + }, + "startingDeadlineSeconds": { "type": "integer", "minimum": 0 }, + "successfulJobsHistoryLimit": { "type": "integer", "minimum": 0 }, + "failedJobsHistoryLimit": { "type": "integer", "minimum": 0 }, + "activeDeadlineSeconds": { "type": "integer", "minimum": 1 }, + "backoffLimit": { "type": "integer", "minimum": 0 }, + "restartPolicy": { + "type": "string", + "enum": ["OnFailure", "Never"] + }, + "serviceAccount": { + "type": "object", + "additionalProperties": false, + "properties": { + "create": { "type": "boolean" }, + "name": { "type": "string" }, + "annotations": { "type": "object" } + }, + "required": ["create"] + }, + "securityContext": { "type": "object" }, + "nameOverride": { "type": "string" }, + "fullnameOverride": { "type": "string" }, + "nodeSelector": { "type": "object" }, + "tolerations": { "type": "array" }, + "affinity": { "type": "object" }, + "extraEnv": { + "type": "array", + "items": { + "type": "object", + "properties": { + "name": { "type": "string", "minLength": 1 } + }, + "required": ["name"] + } + } + } +} diff --git a/charts/promforecast-tuner/values.yaml b/charts/promforecast-tuner/values.yaml new file mode 100644 index 0000000..fe40486 --- /dev/null +++ b/charts/promforecast-tuner/values.yaml @@ -0,0 +1,131 @@ +# promforecast-tuner — Helm values. +# +# The tuner is a one-shot binary (``promforecast tune ...``) shipped +# in the same container image as the live forecaster. This sub-chart +# wires it into a CronJob that runs on a configurable cadence +# (weekly by default) and pushes recommendation rows +# (``forecast_recommended_param``) to the configured ``remote_write`` +# sink. +# +# Operators read the recommendations from a dedicated Grafana panel +# and apply them by editing the forecaster's config — there is no +# auto-apply path. + +image: + repository: ghcr.io/esops-dev/promforecast + # tag: "" # Defaults to the chart's appVersion when empty. + pullPolicy: IfNotPresent + +# Cron schedule for the recommendation walk. Weekly (Sunday 02:00 UTC) +# is a sensible default — it covers a full weekly seasonality cycle of +# the past 14 days without contending with the typical batch-job +# window. Switch to a daily cadence for very dynamic workloads or to a +# monthly one for stable infrastructure metrics. +schedule: "0 2 * * 0" + +# Whether the CronJob is enabled. Disabling lets operators install the +# chart for dashboards-only (the Grafana panel referenced in +# ``docs/operations/auto-tuning.md`` works against an empty metric set +# without errors). +enabled: true + +# Existing ConfigMap carrying the forecaster YAML config. The tuner +# loads the same config the live forecaster runs against so the +# baseline (model, season_length) values match exactly. +# +# The default ``promforecast`` matches the ConfigMap the standalone +# ``promforecast`` chart emits when installed under that release +# name (``helm install promforecast charts/promforecast``). Other +# install shapes require overriding: +# +# * Umbrella ``promforecast-stack`` released as ``stack`` → +# ConfigMap is ``stack-promforecast``. +# * Standalone released as ``pf`` → +# ConfigMap is ``pf-promforecast``. +# * ``existingConfigMap`` set on the forecaster install → use that +# ConfigMap name verbatim here. +configMap: + name: "promforecast" + key: "config.yaml" + +# Override ``sink.remote_write.url`` for the tuner's recommendation +# push. Leave empty to inherit from the config; useful when running +# the tuner against a dedicated TSDB endpoint (e.g. a relaxed +# ``futureRetention`` for the recommendation gauges). +sinkUrlOverride: "" + +# Minimum percentage MAPE improvement a candidate must show over the +# configured baseline to be emitted as a recommendation. Tune higher +# for noisy data (avoids small / noisy improvements polluting the +# panel) and lower when small wins matter. +minImprovementPct: 5.0 + +# Lookback window for the grid search. Falls back to +# ``defaults.lookback`` from the config when null. 14d is the typical +# choice — covers two weekly cycles without making the per-fit cost +# explode. +lookback: null + +# Optional list of group names to tune. Empty / null tunes every +# configured group. Useful for splitting a heavy tuner walk across +# multiple CronJobs with different schedules. +groups: [] + +# Per-pod resource requests / limits. The tuner runs ``N_groups * +# N_queries * (1 + 2 + (M-1))`` fits per run (baseline + season_length +# grid + alternative models). Sized for a small fleet by default; +# bump for deployments with hundreds of queries or expensive +# models (AutoARIMA on long lookbacks). +resources: + requests: + cpu: 200m + memory: 512Mi + limits: + cpu: 1 + memory: 1Gi + +# CronJob concurrency. ``Forbid`` ensures a long-running walk never +# overlaps with the next schedule — the tuner is best-effort and a +# missed run is fine; double-running would just push the same +# recommendations twice. ``startingDeadlineSeconds`` matches the +# default schedule cadence so a brief cluster outage doesn't fire a +# stale catch-up run. +concurrencyPolicy: Forbid +startingDeadlineSeconds: 3600 +successfulJobsHistoryLimit: 1 +failedJobsHistoryLimit: 3 + +# Activation deadline for a single tuner run. A walk that takes +# longer than this is killed by the kubelet; the next schedule fires +# normally. Sized comfortably above the typical run time (single +# digits of minutes per group). +activeDeadlineSeconds: 1800 +backoffLimit: 2 +restartPolicy: Never + +serviceAccount: + create: true + name: "" + annotations: {} + +# Pod security context. The tuner doesn't need anything special — +# runs as a non-root user, no privileged operations. +securityContext: + runAsNonRoot: true + runAsUser: 65532 + runAsGroup: 65532 + fsGroup: 65532 + +nameOverride: "" +fullnameOverride: "" + +nodeSelector: {} +tolerations: [] +affinity: {} + +# Extra environment variables to inject into the CronJob pod. Useful +# for passing through proxy settings or auth tokens that the tuner's +# datasource / sink need. +extraEnv: [] + # - name: HTTP_PROXY + # value: "http://proxy.internal:3128" diff --git a/charts/promforecast/values.yaml b/charts/promforecast/values.yaml index 5d1b0ea..ff65400 100644 --- a/charts/promforecast/values.yaml +++ b/charts/promforecast/values.yaml @@ -1,6 +1,13 @@ # Default values for promforecast. # Single-replica by default. Multi-replica HA mode requires # `highAvailability.enabled: true` and a Redis backend. +# +# Before going live, walk the production-readiness checklist: +# https://github.com/esops-dev/promforecast/blob/main/docs/production-checklist.md +# — covers sizing tables, three reference architectures, and the +# pre-flight items (sink + scrape-drop, alerts, dashboards, backfill, +# cost estimate, quality-score gate on predictive autoscaling) that +# turn a fresh install into a production-ready one. image: repository: ghcr.io/esops-dev/promforecast @@ -69,6 +76,15 @@ config: # starve scheduled fits. Same network-level access-control model as # /metrics and /-/reload. expose_what_if_endpoint: false + # Off by default. Flip to true to expose `GET /forecast/warmup` which + # reports per-(group, query) "what is still loading?" status — useful + # immediately after a fresh install (or a restart that wiped the + # TSDB volume) so the operator can answer "why are my forecasts not + # showing in Grafana yet?" without log scraping. Each call costs one + # cheap PromQL probe per configured query; safe for ad-hoc debugging, + # wasteful for any monitoring system polling it on cadence. The + # companion CLI is `promforecast warmup --url `. + expose_warmup_endpoint: false safety: max_series_per_query: 500 diff --git a/dashboards/grafana/promforecast-capacity.json b/dashboards/grafana/promforecast-capacity.json index 20fe2ae..30782f0 100644 --- a/dashboards/grafana/promforecast-capacity.json +++ b/dashboards/grafana/promforecast-capacity.json @@ -402,6 +402,57 @@ } ] } + }, + { + "type": "table", + "id": 9, + "title": "Capacity ETAs ranked by cost of action", + "description": "Joins ``forecast_time_to_threshold_seconds`` with ``forecast_threshold_action_cost`` on (id, group, name) and sorts by cost descending — so 'soon-and-expensive' rows surface above 'soon-and-cheap' ones. Only thresholds with an ``action_cost`` annotation contribute. ``type`` is operator free text (usd, eur, engineer_hours, ...); pin the panel to a single ``type`` if your fleet mixes units.", + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "gridPos": { "h": 9, "w": 24, "x": 0, "y": 48 }, + "targets": [ + { + "refId": "A", + "expr": "forecast_threshold_action_cost{group=~\"$group\",id=~\"$id\"}", + "legendFormat": "{{id}} / {{name}}", + "format": "table", + "instant": true + }, + { + "refId": "B", + "expr": "max by (id, group, name) (forecast_time_to_threshold_seconds{group=~\"$group\",id=~\"$id\"})", + "legendFormat": "{{id}} / {{name}}", + "format": "table", + "instant": true + } + ], + "transformations": [ + { "id": "merge", "options": {} }, + { + "id": "organize", + "options": { + "excludeByName": { "Time": true, "__name__": true, "job": true, "instance": true }, + "renameByName": { + "id": "metric", + "group": "group", + "name": "threshold", + "type": "cost type", + "Value #A": "action cost", + "Value #B": "ETA (s)" + } + } + }, + { "id": "sortBy", "options": { "fields": {}, "sort": [{ "field": "action cost", "desc": true }] } } + ], + "fieldConfig": { + "defaults": { "custom": { "align": "left" } }, + "overrides": [ + { + "matcher": { "id": "byName", "options": "ETA (s)" }, + "properties": [ { "id": "unit", "value": "s" } ] + } + ] + } } ] } diff --git a/dashboards/grafana/promforecast-is-it-working.json b/dashboards/grafana/promforecast-is-it-working.json index 3210dd3..1dbdba5 100644 --- a/dashboards/grafana/promforecast-is-it-working.json +++ b/dashboards/grafana/promforecast-is-it-working.json @@ -327,6 +327,87 @@ } ] } + }, + { + "type": "stat", + "id": 40, + "title": "Service forecast health (per bucket)", + "description": "Aggregated forecast_quality_score per (service, environment, ...) bucket. Off until the operator configures governance.health_score; the value is the mean trustworthiness of the forecasts for that bucket. Red < 0.6 = bucket fails the alert-gate threshold; green > 0.8 = trustworthy. Hover a tile to see the series_count documentation label.", + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "gridPos": { "h": 6, "w": 24, "x": 0, "y": 36 }, + "targets": [ + { "refId": "A", "expr": "forecast_service_health_score", "legendFormat": "{{service}} / {{environment}}" } + ], + "options": { + "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, + "orientation": "horizontal", + "colorMode": "background", + "graphMode": "none", + "textMode": "value_and_name" + }, + "fieldConfig": { + "defaults": { + "unit": "none", + "min": 0, + "max": 1, + "decimals": 2, + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "red", "value": null }, + { "color": "yellow", "value": 0.6 }, + { "color": "green", "value": 0.8 } + ] + } + }, + "overrides": [] + } + }, + { + "type": "table", + "id": 41, + "title": "Right-sizing suggestions (ranked by savings)", + "description": "Per-workload recommendations from recommendations.resource_sizing. Sorted by savings %. Positive savings = safe to downsize; negative = upsize recommended. Gated by quality_floor so untrustworthy forecasts produce no row. ``confidence`` label categorises the underlying forecast_quality_score.", + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "gridPos": { "h": 9, "w": 24, "x": 0, "y": 42 }, + "targets": [ + { "refId": "A", "expr": "forecast_recommended_resource_size", "legendFormat": "{{workload}} / {{kind}}", "format": "table", "instant": true }, + { "refId": "B", "expr": "forecast_recommended_resource_current", "legendFormat": "{{workload}} / {{kind}}", "format": "table", "instant": true }, + { "refId": "C", "expr": "forecast_recommended_resource_savings_pct", "legendFormat": "{{workload}} / {{kind}}", "format": "table", "instant": true } + ], + "transformations": [ + { "id": "merge", "options": {} }, + { + "id": "organize", + "options": { + "excludeByName": { "Time": true, "__name__": true, "job": true, "instance": true }, + "renameByName": { + "workload": "workload", + "kind": "kind", + "confidence": "confidence", + "Value #A": "recommended", + "Value #B": "current", + "Value #C": "savings %" + } + } + }, + { "id": "sortBy", "options": { "fields": {}, "sort": [{ "field": "savings %", "desc": true }] } } + ], + "fieldConfig": { + "defaults": { "custom": { "align": "left" } }, + "overrides": [ + { + "matcher": { "id": "byName", "options": "savings %" }, + "properties": [ + { "id": "unit", "value": "percent" }, + { "id": "custom.cellOptions", "value": { "type": "color-text" } }, + { "id": "thresholds", "value": { "mode": "absolute", "steps": [ + { "color": "red", "value": null }, { "color": "yellow", "value": 0 }, { "color": "green", "value": 10 } + ]}} + ] + } + ] + } } ] } diff --git a/docker/dev-config.yaml b/docker/dev-config.yaml index bafc400..2649611 100644 --- a/docker/dev-config.yaml +++ b/docker/dev-config.yaml @@ -40,6 +40,12 @@ server: # Earlier 1m caused the per-group lock to back up runs to ~10 min effective # cadence on http_demo and node_storage. refresh_interval: 5m + # Expose the warm-up status endpoint so an operator can verify the + # fresh-install behaviour with + # docker compose exec pf-forecaster promforecast warmup --url http://localhost:9091 + # from the dev stack without re-rendering the chart. Safe in dev because the + # forecaster's metrics port isn't exposed beyond the docker-compose network. + expose_warmup_endpoint: true # Tight caps to exercise overflow handling end-to-end in dev. safety: diff --git a/docs/README.md b/docs/README.md index 4ed020d..589b6ef 100644 --- a/docs/README.md +++ b/docs/README.md @@ -54,13 +54,39 @@ scheduling, and serving telemetry. - [Scheduling](operations/scheduling.md) — per-query refresh intervals, max-concurrent-fits cap - [High availability](operations/ha.md) — Lease-based leader election, Redis snapshot cache - [Query cache](operations/cache.md) — in-memory LRU + optional Redis dedup for overlapping PromQL -- [Telemetry](operations/telemetry.md) — internal metrics catalogue +- [Telemetry](operations/telemetry.md) — internal metrics catalogue + per-series staleness + self-observability (fit success ratio, regressor stability, per-group CPU/memory) +- [Warm-up status](operations/warmup.md) — fresh-install "what is still loading?" endpoint and CLI +- [Forecast inspector](operations/inspect.md) — debug a single weird forecast end-to-end, with optional per-fold backtest exposure +- [Historical backfill](operations/backfill.md) — replay the fit pipeline at past timestamps so SLO dashboards are meaningful from day one +- [Quality score](operations/quality-score.md) — composite reliability gauge, per-horizon segmentation, and which horizon to gate alerts on +- [Forecast stability](operations/forecast-stability.md) — `forecast_revision_magnitude` and `forecast_model_disagreement` for catching a thrashing model or a configured model set that can't agree on the future +- [Validate CLI](operations/validate.md) — `promforecast validate` modes: cardinality estimate, `--strict-cardinality` CI gate, `--estimate-cost` runtime budget, `--dry-run` sample fit +- [Degraded-mode operations](operations/degraded-modes.md) — failure-mode matrix covering datasource read failure, sink write failure, partial group failure, sustained fit timeout, and stale snapshot +- [Hyperparameter auto-tuning](operations/auto-tuning.md) — weekly recommendation CronJob that surfaces alternative `season_length` / model choices when the current config has drifted +- [Right-sizing recommendations](operations/right-sizing.md) — per-workload limit/request recommendations from the forecast peak, gated by quality score, optional savings-pct sibling +- [Drift change-correlation explainer](operations/drift-explained.md) — structured log + per-(group, query, primary_suspect) counter that turns a drift alert into "start your investigation on this regressor" + +## Governance + +Aggregations and reporting views over the forecaster's own output. + +- [Service-level health composite](governance/forecast-health.md) — per-(service, environment) aggregate of `forecast_quality_score` for executive dashboards + +## Cookbook + +Copy-pasteable recipes for the common Prometheus metric families. +Each recipe ships a working YAML snippet, a recommended model + rationale, +the cardinality footprint, and example alerts. + +- [Cookbook index](cookbook/README.md) — index of all recipes ## Guides Cross-cutting explanations that don't fit cleanly under one feature. - [Reading the numbers](guides/reading-the-numbers.md) — how to interpret quality and accuracy in practice (mean vs p95, horizon paradox, "wait three cycles", when to actually worry) +- [Production-readiness checklist](production-checklist.md) — sizing table, three reference architectures, and the boxes worth ticking off before flipping production traffic to a new install +- [Multi-cluster reference architectures](architecture/multi-cluster.md) — three patterns for deployments spanning multiple Kubernetes clusters, with sample chart values and cardinality / network trade-offs ## Contributing diff --git a/docs/architecture/multi-cluster.md b/docs/architecture/multi-cluster.md new file mode 100644 index 0000000..7667cdf --- /dev/null +++ b/docs/architecture/multi-cluster.md @@ -0,0 +1,118 @@ +# Multi-cluster reference architectures + +The chart defaults assume a single forecaster reading a single VictoriaMetrics (or Mimir/Thanos) endpoint. Multi-cluster deployments require additional choices: + +- Where the forecaster runs (per cluster or centrally). +- Where its reads and writes land. +- How the `cluster` label is propagated for dashboards. + +Three reference patterns cover the common cases. + +## Pattern comparison + +| Pattern | Autonomy | Cardinality | Network | When to use | +|---------|----------|-------------|---------|-------------| +| **A. Per-cluster forecaster + per-cluster VM → central aggregation** (recommended default) | High (cluster failure doesn't affect others) | Per-cluster (sized locally) | Low (local reads, one outbound push per cluster) | Any deployment with ≥2 autonomous clusters | +| **B. Central forecaster + federated read** | Low (single point of failure) | Union of all clusters | Higher (cross-cluster reads on every fit) | Small fleets (<5 clusters, <5000 series total) | +| **C. Per-cluster forecaster + per-cluster VM, Grafana mixed-datasource** | Highest (full isolation) | Per-cluster | Zero cross-cluster | Strict regulatory/compliance isolation | + +## A. Per-cluster forecaster, per-cluster VM, central aggregation (recommended) + +``` +Cluster A: prometheus(A) ──remote_write──> VM(A) <── forecaster(A) ──remote_write──> VM(central) +Cluster B: prometheus(B) ──remote_write──> VM(B) <── forecaster(B) ──remote_write──> VM(central) +``` + +Each cluster runs its own forecaster, reads from its local VM, and pushes forecasts to a shared central VM. Central tier is read-only aggregation for dashboards. + +**Why this is preferred** +- High autonomy: one cluster’s failure never blocks others. +- Local reads keep fit timing predictable. +- Cardinality is sized per cluster, not the union. + +**Sample values (per cluster)** + +```yaml +config: + datasource: + url: http://victoriametrics.monitoring.svc.cluster.local:8428 + sink: + remote_write: + enabled: true + url: https://vm-central.example.com/api/v1/import/prometheus + headers: + Authorization: "Bearer ${VM_CENTRAL_TOKEN}" +``` + +The `cluster` label flows automatically from each cluster’s Prometheus `external_labels`. + +## B. Central forecaster, federated VM read + +For small fleets that prefer one forecaster instance: + +``` +Cluster A: prometheus(A) ──remote_write──> VM(A) ─┐ +Cluster B: prometheus(B) ──remote_write──> VM(B) ─┼──> VM(federated read) <── forecaster(central) ──remote_write──> VM(central) +``` + +One forecaster reads through a federated endpoint (e.g. VM’s `vmselect`). + +**Trade-offs** +- Simpler ops, native cross-cluster correlations. +- Higher read latency and single point of failure. +- Cardinality sized for the union of all clusters. + +**Sample values** + +```yaml +config: + datasource: + url: http://vm-federated.monitoring.svc.cluster.local:8481/select/0/prometheus + timeout: 60s + sink: + remote_write: + enabled: true + url: http://victoriametrics.monitoring.svc.cluster.local:8428/api/v1/import/prometheus + safety: + max_total_series: 50000 # sized for union +``` + +## C. Per-cluster forecaster + per-cluster VM, Grafana mixed-datasource + +Strict isolation (no central aggregation): + +``` +Cluster A: prometheus(A) ──remote_write──> VM(A) <── forecaster(A) +Cluster B: prometheus(B) ──remote_write──> VM(B) <── forecaster(B) +``` + +Grafana uses multiple datasources and mixed-datasource queries for cross-cluster views. + +**Trade-offs** +- Maximum isolation and compliance. +- Cross-cluster dashboards and alerting are more complex. +- No cross-cluster forecasts possible. + +**Sample values** (identical per cluster, sink points to local VM) + +```yaml +config: + datasource: + url: http://victoriametrics.monitoring.svc.cluster.local:8428 + sink: + remote_write: + enabled: true + url: http://victoriametrics.monitoring.svc.cluster.local:8428/api/v1/import/prometheus +``` + +## Label alignment + +The `cluster` label flows automatically from each cluster’s Prometheus `external_labels` through `remote_write` into the forecaster’s source queries and back out via the sink. No forecaster-side configuration is needed. + +The forecaster’s operational metrics (`forecast_*`) describe its own state and are scoped by scrape labels (`job`, `instance`, `namespace`) rather than `cluster`. + +## Cross-references + +- [Production-readiness checklist](../production-checklist.md) +- [Emission paths](../reference/emission-paths.md) — labelset preservation details +- [High availability](../operations/ha.md) — pairs cleanly with Pattern A diff --git a/docs/cookbook/README.md b/docs/cookbook/README.md new file mode 100644 index 0000000..cf4af79 --- /dev/null +++ b/docs/cookbook/README.md @@ -0,0 +1,36 @@ +```markdown +# Cookbook + +Recipes that turn *"I want to forecast X"* into *"here's the config, here's the alert, here's the dashboard"*. + +Each page is anchored to a single common Prometheus metric family, picks a sensible model + alert shape, and is exercised by CI (schema validation + cardinality estimator). If a recipe lands on your `/metrics` and behaves nothing like the dashboard panel described here, the test is the first place to look. + +## Capacity & infrastructure + +* [`node_filesystem_avail_bytes`](node-filesystem-avail-bytes.md) — predict disk-fill time, alert before the pager fires. +* [`redis_connected_clients`](redis-connected-clients.md) — see Redis connection-pool exhaustion hours before `maxclients` trips. +* [`kafka_consumer_lag`](kafka-consumer-lag.md) — capacity *and* drain forecasts on the same backlog. + +## Workloads & application metrics + +* [`http_requests_total`](http-requests-total.md) — service traffic forecast, SLO error-budget projections. +* [`kube_deployment_status_replicas`](kube-deployment-status-replicas.md) — predict autoscaler ramp before it surprises capacity. +* [`jvm_gc_pause_seconds`](jvm-gc-pause-seconds.md) — long GC pauses, drift-based anomaly detection. + +## Cost & financial + +* [`aws_cloudwatch_estimated_charges_usd`](aws-cloudwatch-estimated-charges-usd.md) — month-end bill projection. + +Each recipe ships a copy-pasteable config in [`examples/configs/cookbook/`](../../examples/configs/cookbook/). Use that as the starting point and tune PromQL, thresholds, and cost framings to your environment. + +## How a recipe is structured + +Every page covers: + +1. **The metric** — what it measures and why you'd forecast it. +2. **The config** — a copy-pasteable YAML snippet sized for a small fleet. +3. **The recommended model** — which `defaults.models` to pick and why. +4. **The expected forecast shape** — what the curve typically looks like in Grafana. +5. **The cardinality footprint** — what `promforecast validate` reports for the snippet. +6. **Example alerts** — one or two `PrometheusRule` snippets paired with the recipe. +``` \ No newline at end of file diff --git a/docs/cookbook/aws-cloudwatch-estimated-charges-usd.md b/docs/cookbook/aws-cloudwatch-estimated-charges-usd.md new file mode 100644 index 0000000..90702c1 --- /dev/null +++ b/docs/cookbook/aws-cloudwatch-estimated-charges-usd.md @@ -0,0 +1,68 @@ +# Recipe: `aws_cloudwatch_estimated_charges_usd` + +> Month-end AWS bill projection that finance can read alongside engineering. + +## The metric + +`aws_cloudwatch_estimated_charges_usd` is the AWS billing estimate published by the CloudWatch exporter. It updates roughly every 6 hours and is denominated in USD. + +## The config + +```yaml +groups: + - name: cloud_cost + queries: + - id: aws_estimated_charges_usd + promql: | + aws_cloudwatch_estimated_charges_usd{service_name="Total"} + step: 6h + thresholds: + - name: monthly_budget + value: 50000 + comparator: gt + alert_within: 14d + severity: warning + action_cost: + type: usd + value: 50000 +``` + +`step: 6h` matches CloudWatch’s publish cadence — finer resolution wastes TSDB CPU on duplicate samples. + +## Recommended model + +* **`AutoETSDamped`** with `damping_parameter: 0.95` (more aggressive than the default 0.98) because billing curves are fundamentally bounded by service contracts — linear extrapolation overstates. +* `SeasonalNaive` is included as a safety net; the weekly cycle is the dominant signal and often wins auto-select. + +## Expected forecast shape + +- Smooth upward trend within a billing month. +- Sharp drop at the start of each month as the counter resets — the forecast handles the reset cleanly because the lookback (90 d) includes multiple resets. + +## Cardinality + +For 5 AWS service categories × 1 model × 1 level: + +- forecast lines: 15 +- threshold ETA lines: 5 +- will-breach lines: 5 +- action cost lines: 1 + +## Alerts + +```promql +# Projected to exceed monthly budget within 14 d +aws_estimated_charges_usd_forecast_will_breach{severity="warning"} == 1 +``` + +```promql +# Month-end charge moved by > $5k week-over-week +abs( + aws_estimated_charges_usd_forecast{horizon="14d"} + - aws_estimated_charges_usd_forecast{horizon="14d"} offset 7d +) > 5000 +``` + +## Pair with service-level health composite + +This recipe contributes one row per service category to the `forecast_service_health_score` aggregation, giving finance a single number per AWS service tier alongside the per-tier forecasts. diff --git a/docs/cookbook/http-requests-total.md b/docs/cookbook/http-requests-total.md new file mode 100644 index 0000000..54602a8 --- /dev/null +++ b/docs/cookbook/http-requests-total.md @@ -0,0 +1,70 @@ +# Recipe: `http_requests_total` + +> Per-service traffic forecasts for capacity planning and SLO error-budget projections. + +## The metric + +`http_requests_total` is the canonical request-rate counter. The forecastable shape is its rate (`rate(...[5m])`). + +## The config + +```yaml +groups: + - name: http_traffic + queries: + - id: http_requests_rate + promql: | + sum by (service) (rate(http_requests_total[5m])) + step: 1m + thresholds: + - name: capacity_soft + value: 5000 + comparator: gt + alert_within: 30m + severity: warning + emission: + growth_rate: + enabled: true + windows: [1h, 1d] + units: percent +``` + +## The model + +* **`AutoARIMA`** usually wins auto-select: strong daily cycle plus modest week-over-week growth. +* `SeasonalNaive` rides along as the cheap baseline. + +`season_length: 1440` matches the daily cycle at the 1-minute step. + +## Expected forecast shape + +- Smooth daily curve with a clear business-hours peak. +- Visible weekly modulation (weekdays vs. weekends). +- A deploy or marketing push appears as a short-horizon level shift; the 95 % band absorbs it within an hour or two. + +## Cardinality + +For 100 services × 1 model × 2 levels: + +- forecast lines: 500 +- threshold ETA: 300 +- will-breach lines: 300 +- growth rate lines: 200 +- deviation lines: 400 +- quality lines: 200 + +## Alerts + +```promql +# Predicted to breach 5k req/s within 30 minutes — scale ahead. +http_requests_rate_forecast_will_breach{severity="warning", level="95"} == 1 +``` + +```promql +# Traffic accelerating 100 % week-over-week — investigate. +http_requests_rate_forecast_growth_rate{window="1h", units="percent"} > 1.0 +``` + +## Pair with SLO error-budget forecasting + +Add an `slo:` block to the same query and the forecaster projects when the monthly budget exhausts at the predicted error rate. See [slo.md](../outputs/slo.md). diff --git a/docs/cookbook/jvm-gc-pause-seconds.md b/docs/cookbook/jvm-gc-pause-seconds.md new file mode 100644 index 0000000..19f9695 --- /dev/null +++ b/docs/cookbook/jvm-gc-pause-seconds.md @@ -0,0 +1,66 @@ +# Recipe: `jvm_gc_pause_seconds` + +> Long GC pauses correlate with user-facing latency. Forecast the rate so you can recycle pods before SLOs are impacted. + +## The metric + +`jvm_gc_pause_seconds` is a histogram of GC pause durations. The forecastable signal is the **rate of long pauses**: + +```promql +rate(jvm_gc_pause_seconds_count{action="end of major GC"}[5m]) +``` + +A baseline rate is normal; a step change to a sustained higher rate is the precursor to pods dropping requests. + +## The config + +```yaml +groups: + - name: jvm + queries: + - id: jvm_gc_long_pause_rate + promql: | + sum by (instance, pool) ( + rate(jvm_gc_pause_seconds_count{action="end of major GC"}[5m]) + ) + emission: + drift: + enabled: true + alert_threshold: 0.3 +``` + +## The model + +* **`AutoARIMA`** captures the baseline + noise pattern well. JVMs rarely show strong seasonality on this metric. +* `SeasonalNaive` rides along but rarely wins auto-select. + +## Expected forecast shape + +- Mostly flat with occasional spikes (usually deploy-driven cache invalidation or stop-the-world pauses). +- Relatively wide confidence band because the underlying signal is spiky. + +## Cardinality + +For 50 JVMs × 2 pools × 1 model × 2 levels: + +- forecast lines: 500 +- deviation lines: 200 +- drift lines: 100 +- quality lines: 100 + +## Alerts + +```promql +# Long-GC rate outside the predicted band — investigate. +forecast_deviation_outside_band{ + id="jvm_gc_long_pause_rate", level="95" +} == 1 +``` + +```promql +# Forecast itself is unstable — the model can't fit recent data. +forecast_drift_score{id="jvm_gc_long_pause_rate"} > 0.3 +for 4 fits +``` + +The drift alert is especially useful here — the [drift change-correlation explainer](../operations/drift-explained.md) will surface which regressor (deploy timing, hour-of-day, …) most correlates with the change. diff --git a/docs/cookbook/kafka-consumer-lag.md b/docs/cookbook/kafka-consumer-lag.md new file mode 100644 index 0000000..ebbfb9c --- /dev/null +++ b/docs/cookbook/kafka-consumer-lag.md @@ -0,0 +1,76 @@ +# Recipe: `kafka_consumer_lag` + +> Capacity *and* drain forecasts on the same backlog. + +## The metric + +`kafka_consumergroup_lag` is the number of messages a consumer group is behind the partition HEAD. Two questions matter: + +1. **Capacity** — at the current rate, when does the backlog exhaust broker retention? +2. **Drain** — when does the backlog reach zero, or is it not actually draining? + +`promforecast` answers both with one query block using the standard `thresholds` block plus the `le_drain` comparator. + +## The config + +```yaml +groups: + - name: kafka + queries: + - id: kafka_consumer_lag + promql: | + sum by (topic, consumergroup) ( + kafka_consumergroup_lag + ) + thresholds: + - name: backlog_warn + value: 100000 + comparator: gt + alert_within: 1h + severity: warning + - name: drained + value: 0 + comparator: le_drain +``` + +The `le_drain` comparator emits to `_forecast_time_to_drain_seconds` (separate metric name from the threshold ETA family) and reports `+Inf` when the forecast never crosses zero from above — alert on that to catch "queue is not actually draining". + +## The model + +* **`AutoARIMA`** captures the consumer-lag dynamics (autocorrelated, often piecewise linear). +* `SeasonalNaive` rides along; surprisingly often wins auto-select when consumption follows a daily / weekly pattern. + +## Expected forecast shape + +* Healthy queue: smooth oscillation around zero with brief positive excursions during traffic bursts. +* Stuck consumer: monotonic growth — capacity alert fires. +* Catching up: monotonic decrease — drain ETA reports a finite value; alerting on `> 4h` reads as "drain projected to take >4h". + +## Cardinality + +For 50 (topic, consumergroup) pairs × 1 model × 1 level: + +- forecast lines: 150 +- threshold ETA: 100 +- will-breach lines: 100 +- drain ETA: 100 (le_drain emits to a parallel family) + +## Alerts + +```promql +# Backlog projected to breach 100k within 1h — investigate the consumer. +kafka_consumer_lag_forecast_will_breach{severity="warning"} == 1 +``` + +```promql +# Queue is not actually draining. +kafka_consumer_lag_forecast_time_to_drain_seconds{name="drained"} == +Inf +for 30m +``` + +```promql +# Drain projected to take >4h. +kafka_consumer_lag_forecast_time_to_drain_seconds{name="drained"} > 4 * 60 * 60 +and +kafka_consumer_lag_forecast_time_to_drain_seconds{name="drained"} != +Inf +``` diff --git a/docs/cookbook/kube-deployment-status-replicas.md b/docs/cookbook/kube-deployment-status-replicas.md new file mode 100644 index 0000000..a6678c9 --- /dev/null +++ b/docs/cookbook/kube-deployment-status-replicas.md @@ -0,0 +1,54 @@ +# Recipe: `kube_deployment_status_replicas` + +> Predict autoscaler ramp before it surprises capacity. + +## The metric + +`kube_deployment_status_replicas` is the current replica count per deployment. The forecastable shape is the daily cycle of HPA-driven scaling — deployments breathe in and out across the day, and predicting the next peak lets capacity planning keep up. + +## The config + +```yaml +groups: + - name: kube_workloads + queries: + - id: kube_deployment_status_replicas + promql: | + sum by (namespace, deployment) ( + kube_deployment_status_replicas + ) + step: 1m + data_profile: count +``` + +`data_profile: count` enables count-aware emission safeguards (`clip_non_negative` and `round_to_integer` default to on) so predicted replica counts are whole integers and never negative. + +## The model + +* **`SeasonalNaive`** is surprisingly good — the pattern is largely “same shape as last week”. +* **`AutoARIMA`** rides along as the auto-select alternative for deployments with non-stationary ramp (slow week-over-week growth). + +## Expected forecast shape + +- Step-function pattern within the day — replicas land on whole integers. +- Confidence band typically narrow at level transitions, wider during transition windows. + +## Cardinality + +For 200 deployments × 1 model × 1 level: + +- forecast lines: 600 +- accuracy lines: 400 +- deviation lines: 200 + +## Alerts + +```promql +# Predicted to need >50 replicas in 24h — pre-warm capacity now. +kube_deployment_status_replicas_forecast{horizon="24h"} > 50 +``` + +```promql +# Predicted replica count drifting unexpectedly from the historical pattern. +forecast_drift_score{id="kube_deployment_status_replicas"} > 0.4 +``` diff --git a/docs/cookbook/node-filesystem-avail-bytes.md b/docs/cookbook/node-filesystem-avail-bytes.md new file mode 100644 index 0000000..3b3c6de --- /dev/null +++ b/docs/cookbook/node-filesystem-avail-bytes.md @@ -0,0 +1,73 @@ +# Recipe: `node_filesystem_avail_bytes` + +> Predict when each mount runs out of space. Alert before the pager. + +## The metric + +`node_filesystem_avail_bytes` (from `node_exporter`) reports free bytes per `(instance, device, mountpoint)`. The line that matters is the one that crosses a low watermark and stays there — disk-fill is rarely sudden. + +## The config + +```yaml +groups: + - name: disk_capacity + queries: + - id: node_filesystem_avail_bytes + promql: | + node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} + thresholds: + - name: critical + value: 5368709120 # 5 GiB + comparator: lt + alert_within: 24h + severity: critical + action_cost: + type: usd + value: 500 + emission: + growth_rate: + enabled: true + windows: [1d, 7d] + units: units_per_second +``` + +The full file ships a damped-trend model + auto-select to keep long-horizon extrapolation honest. + +## The model + +* **`AutoETSDamped`** with `damping_parameter: 0.98` — disk usage grows, but rarely at the rate the last 24 h would extrapolate. Damped-trend bends the long-horizon curve toward a level. +* `SeasonalNaive` rides along as the auto-select fallback when the ETS fit blows up. + +## Expected forecast shape + +- Steady downward slope, narrow band. +- Bursty backups + nightly cleanups appear as stair-steps within a day, smoothed to a weekly trend. +- Post-deploy log spike shows as a sharp local dip with visibly wider bands for the lookahead. + +If you see `yhat` going *up*, the model is fitting the wrong direction. Switch the threshold comparator or check the PromQL filter. + +## Cardinality + +For ~100 hosts × 1 model × 1 level: + +- forecast lines: 300 +- threshold ETA: 300 +- will-breach lines: 300 +- growth rate lines: 200 +- action cost lines: 1 + +## Alerts + +```promql +# Will breach 5 GiB within 24 h at the 95th percentile band. +node_filesystem_avail_bytes_forecast_will_breach{severity="critical", level="95"} == 1 +``` + +```promql +# Free-space loss rate has tripled compared to last week's baseline — something unusual is consuming the disk. +abs(node_filesystem_avail_bytes_forecast_growth_rate{window="1d"}) + / clamp_min( + abs(node_filesystem_avail_bytes_forecast_growth_rate{window="7d"} offset 1d), + 1 + ) > 3 +``` diff --git a/docs/cookbook/redis-connected-clients.md b/docs/cookbook/redis-connected-clients.md new file mode 100644 index 0000000..8037641 --- /dev/null +++ b/docs/cookbook/redis-connected-clients.md @@ -0,0 +1,58 @@ +# Recipe: `redis_connected_clients` + +> Forecast Redis pool exhaustion hours before `maxclients` trips. + +## The metric + +`redis_connected_clients` is the current connection count per Redis instance. The classic leak pattern: gradual accumulation over days, followed by a sudden outage when `maxclients` (typically 10k) is hit. + +## The config + +```yaml +groups: + - name: redis + queries: + - id: redis_connected_clients + promql: | + max by (instance) (redis_connected_clients) + thresholds: + - name: capacity + value: 9000 # tune against your maxclients + comparator: gt + alert_within: 4h + severity: critical + action_cost: + type: priority + value: 100 +``` + +## The model + +* **`AutoETSDamped`** (with `damping_parameter: 0.98`) — connection growth is steady but bounded. Damping prevents linear over-extrapolation on long horizons. +* `SeasonalNaive` as fallback when connections show strong daily cycling (aggressive pool recycling). + +## Expected forecast shape + +- Slow upward drift with a slight daily cycle. +- When a leak is present, `yhat_upper` crosses the threshold hours-to-days before the central forecast. + +## Cardinality + +For 20 Redis instances × 1 model × 1 level: + +- forecast lines: 60 +- threshold ETA: 60 +- will-breach lines: 60 +- action cost lines: 1 + +## Alerts + +```promql +# Predicted to hit 90 % of maxclients within 4 h — investigate or pre-emptively recycle. +redis_connected_clients_forecast_will_breach{severity="critical", level="95"} == 1 +``` + +```promql +# Leak rate looks unusually fast — surface connection-accumulation rate and gate on a sharp uptick. +redis_connected_clients_forecast - redis_connected_clients_forecast offset 24h > 500 +``` diff --git a/docs/governance/forecast-health.md b/docs/governance/forecast-health.md new file mode 100644 index 0000000..b7f81cb --- /dev/null +++ b/docs/governance/forecast-health.md @@ -0,0 +1,67 @@ +# Service-level forecast health composite + +`forecast_quality_score` answers “how reliable is this *individual* forecast?” +`forecast_service_health_score` collapses every quality score into one composite per service/environment bucket, giving executives and platform teams a single “how healthy are our forecasts today?” number. + +## Enable it + +```yaml +governance: + health_score: + aggregate_by: [service, environment] + aggregator: weighted_mean + weight_by: forecast_series_count + quality_floor: 0.6 # optional +``` + +After the next group run, one gauge appears per distinct bucket: + +``` +forecast_service_health_score{ + service="checkout", environment="prod", + aggregator="weighted_mean", weight_by="forecast_series_count", + series_count="42" +} 0.83 +``` + +## Configuration + +| Field | Required | Meaning | +|------------------|----------|---------| +| `aggregate_by` | Yes | Label keys that define each bucket | +| `aggregator` | No | Only `weighted_mean` supported today | +| `weight_by` | No | `uniform` or `forecast_series_count` (documentation label only) | +| `quality_floor` | No | Drop any individual quality score below this value before aggregating | + +- A quality point missing any key in `aggregate_by` is dropped from all buckets. +- `quality_floor` (default `0.6`) excludes low-quality scores so they don’t drag down the composite. + +## Worked example: executive dashboard + +Top-row panel for “our forecasts are X % reliable today”: + +```promql +# Per-service health, production only +forecast_service_health_score{environment="prod"} > 0.6 +``` + +Worst-first table: + +```promql +sort(forecast_service_health_score{environment="prod"}) +``` + +## Cardinality + +Bounded by the unique combinations of `aggregate_by` labels. +Typical case (`[service, environment]` on 20 services × 3 environments) produces ≤ 60 rows — easily dashboard-friendly. + +`promforecast validate` reports the upper bound under “service health”. + +## When to use vs. when not + +**Use it** for executive reporting, platform health overviews, or a single “is the prediction infrastructure healthy?” number. + +**Do not use it** as a replacement for per-series `forecast_quality_score` when debugging a specific bad forecast — drill into the individual series for that. + +It is also **not** an SRE-style SLI/SLO for the underlying services themselves — use `forecast_error_budget_*` for error-budget tracking (see [slo.md](slo.md)). diff --git a/docs/inputs/preprocessing.md b/docs/inputs/preprocessing.md index aa218cf..a2ba66c 100644 --- a/docs/inputs/preprocessing.md +++ b/docs/inputs/preprocessing.md @@ -1,11 +1,12 @@ -# Input preprocessing: gaps and outliers +# Input preprocessing: gaps, outliers, blacklist windows Production metrics are noisy — scrapes drop, deploys spike, autoscalers shift baselines. -Before fitting, the forecaster optionally runs three cleaning steps **in this order**: +Before fitting, the forecaster optionally runs four cleaning steps **in this order**: -1. Gap imputation -2. Outlier scrubbing -3. Change-point detection (see [`change-points.md`](change-points.md)) +1. Blacklist windows (operator-known anomalous spans) +2. Gap imputation +3. Outlier scrubbing +4. Change-point detection (see [`change-points.md`](change-points.md)) ## Gap imputation @@ -80,6 +81,59 @@ A five-day series with one 10× incident spike at hour 36: `AutoARIMA` widens confidence bands dramatically. With `outliers.enabled: true, method: hampel, sensitivity: medium` the spike is replaced by the local median and bands return to normal width (see `tests/test_preprocess.py::test_outlier_scrubbing_replaces_with_median`). +## Blacklist windows + +Sometimes the operator already knows that a chunk of history is anomalous and should not influence the forecast — a known incident week, a Black Friday traffic spike, a planned migration freeze. Sliding the `lookback` boundary discards unrelated normal data; outlier scrubbing is statistical and may miss multi-day periods. `preprocess.blacklist_windows` makes the operator-known case explicit. + +```yaml +groups: + - name: http_capacity + preprocess: + blacklist_windows: + # Absolute window — drop the November 2024 incident week. + - start: 2024-11-29T00:00:00Z + end: 2024-12-06T00:00:00Z + reason: nov2024_incident + # Annual recurring — drop Black Friday peak every year. + - start: 11-29T00:00 + end: 12-02T00:00 + reason: black_friday + recurring: annual +``` + +### How it works + +Samples whose timestamp falls inside any window are dropped *before* gap imputation. The configured `gaps.strategy` then treats the blacklisted span as a gap: a short window is forward-filled (or linearly interpolated) per the strategy; a window longer than `gaps.max_gap` triggers `drop_series` and skips the series for that fit. + +The half-open interval convention `[start, end)` lets two adjacent windows touch at the boundary without double-counting it. + +### Shapes + +| Field | Required | Description | +|---|---|---| +| `start` | Yes | ISO-8601 timestamp (absolute) or `MM-DDTHH:MM` (annual) | +| `end` | Yes | Same shape as `start`; must be strictly after `start` | +| `reason` | No | Free-text label surfaced on the per-window counter | +| `recurring` | No | `null` (default, absolute) or `annual` | +| `timezone` | No | IANA zone name for `recurring: annual` windows; ignored for absolute. Defaults to `UTC` | + +Annual windows materialise one absolute span per year present in the lookback so a 30-year retention does not pay the cost of materialising 30 spans on every fit. A window that wraps a year boundary (`12-29T00:00` → `01-02T00:00`) rolls the end forward to the next year. + +**Timezones.** Recurring annual windows are anchored in the configured `timezone` so a US Black Friday window flips at local midnight rather than UTC-midnight. Absolute windows are unaffected — the ISO-8601 string carries its own offset. Same convention as the [`holidays` regressor](regressors.md). An unrecognised IANA zone name causes the window to be silently skipped (the rest of the series is still fit); pair this with the per-(group, query) `forecast_input_blacklisted_samples_total` counter so an empty-counter row is the operator-visible signal that the window didn't expand. + +### Observability + +- `forecast_input_blacklisted_samples_total{group, query, reason}` — counter, monotonic across runs. Useful for confirming that an annual-recurring window actually fired for the most recent year of the lookback. +- `forecast_input_effective_lookback_seconds{group, query}` — gauge. Discounts the blacklisted span (`count × step`) so a 14-day lookback with two 6-hour blacklisted windows reports `14d − 12h`. + +### When to reach for it + +- A known incident window where the metric was driven by the failure rather than by the underlying workload. +- Recurring high-traffic events the forecaster shouldn't learn as the baseline (Black Friday, fiscal-quarter-end batches). +- A planned migration / cutover that produced a transient regime that has since returned to the baseline. + +Use a regressor instead when the anomalous behaviour is recurring *and* expected — the model should learn its effect rather than pretend the period never happened. + ## Interaction with conformal calibration Outlier scrubbing runs **before** conformal calibration (see [`../outputs/conformal.md`](../outputs/conformal.md)). diff --git a/docs/operations/auto-tuning.md b/docs/operations/auto-tuning.md new file mode 100644 index 0000000..f041ede --- /dev/null +++ b/docs/operations/auto-tuning.md @@ -0,0 +1,69 @@ +# Hyperparameter auto-tuning + +Most production configs pin `season_length`, `defaults.models`, and per-query overrides at install time. As the underlying signal drifts, those values can quietly become suboptimal — the only visible symptom is slowly degrading `forecast_quality_score` or rising `forecast_accuracy_mape`. + +`promforecast tune` evaluates a small grid of alternatives against the live backtest harness and emits one recommendation per (group, query, model) pair that beats the current baseline by a configurable margin. The operator reviews the recommendations and applies them manually — there is no auto-apply path. + +The recommended deployment is a weekly `CronJob` via the [`promforecast-tuner`](../../charts/promforecast-tuner/) sub-chart. It re-uses the same container image and config as the live forecaster. + +## Emitted metrics + +| Metric | Type | Description | +|--------|------|-------------| +| `forecast_recommended_param{group, query, model, name, current_value, recommended_value}` | gauge | One row per candidate that beats the baseline. Value = `expected_mape_improvement_pct`. | +| `forecast_tuner_last_run_timestamp_seconds{group, outcome}` | gauge | Timestamp of the most recent run per group (`outcome`: `ok` / `partial` / `error`). | +| `forecast_tuner_last_run_duration_seconds{group}` | gauge | Wall-clock duration of the most recent run per group. | + +## The grid (deliberately small) + +- **season_length**: half and double the configured value (e.g. `288` → `144`, `576`). +- **model choice**: every other model in `defaults.models`. + +The grid is intentionally limited so the weekly run stays cheap. (Larger grids and FFT-driven auto-seasonality are planned for v0.5.) + +## Should I apply this recommendation? + +Use the `expected_mape_improvement_pct` value: + +| Improvement | Recommended action | +|-------------|--------------------| +| < 5 % | Ignore (likely noise). | +| 5–15 % | Worth applying at the next config-change window. | +| 15–30 % | Apply soon — the current value has drifted. | +| > 30 % | Investigate first — usually indicates the original config was suboptimal. | + +## Operator workflow + +1. Weekly CronJob runs. +2. Tuner evaluates every group × query × candidate and pushes recommendations. +3. Operator opens the *Tuner recommendations* Grafana panel (ranked by improvement). +4. Operator edits the config, commits, and rolls out. +5. Next tuner run automatically drops the now-applied recommendation. + +## When the tuner fails + +Alert on `forecast_tuner_last_run_timestamp_seconds{outcome="error"} > 0` (or on staleness across all outcomes). + +Common causes: +- Datasource unreachable → fix upstream. +- Sink unreachable → next run regenerates recommendations (tuner is stateless). + +`outcome="partial"` is normal (some queries had too little data). + +Missing runs are caught by staleness: + +```promql +time() - max by (group) (forecast_tuner_last_run_timestamp_seconds) > 8 * 24 * 60 * 60 +``` + +(Adjust multiplier for your schedule.) + +## Scope + +- No auto-apply. +- No per-series tuning. +- No large hyperparameter optimisation (only the shipped grid). + +A future expansion (larger grid, per-model params) will keep the same metric shape and operator workflow. + +The tuner is a separate sub-chart so clusters that don’t want weekly jobs can install the forecaster alone and add the tuner later. It is a pure read-side operation against the long-term TSDB. diff --git a/docs/operations/backfill.md b/docs/operations/backfill.md new file mode 100644 index 0000000..e4e7054 --- /dev/null +++ b/docs/operations/backfill.md @@ -0,0 +1,87 @@ +# Historical backfill + +`promforecast backfill` replays the fit pipeline at simulated past timestamps and pushes the resulting forecast curves + accuracy summaries to the configured `sink.remote_write` URL with explicit historical timestamps. + +Use it right after a fresh install (or after a TSDB volume loss) so dashboards built on `forecast_quality_score`, `forecast_accuracy_mape`, and other rolling families become useful immediately instead of waiting weeks for natural accumulation. + +## When to use it + +- Fresh install on a TSDB that already holds weeks of source data. +- After a TSDB restore or volume loss. +- When extending a quality dashboard’s lookback window. + +Backfill is **not** for replaying missed alerts or correcting historical predictions — the sink dedupes by `(labels, timestamp)` so a second run simply overwrites the same points. + +## CLI usage + +```bash +promforecast backfill \ + --config /etc/promforecast/config.yaml \ + --from 30d \ + --to now \ + --groups node_capacity http_slo +``` + +### Flags + +| Flag | Purpose | +|---------------|---------| +| `--config` | Required. Same YAML the live runner uses. | +| `--from` | Required. How far back to walk (e.g. `30d`, `14d`). | +| `--to` | End timestamp (`now` or ISO-8601). | +| `--groups` | Optional space-separated list. Defaults to all groups. | +| `--sink-url` | Override `sink.remote_write.url`. | + +Exit codes: `0` success, `1` invalid input, `2` TSDB/network failure. + +## Cost + +The backfill cadence matches each group’s effective refresh interval. Dominant cost is model fits. + +| Window | Refresh | Fits per group | Single-core wall time (SeasonalNaive) | Wall time (AutoARIMA) | +|--------|---------|----------------|---------------------------------------|------------------------| +| 7d | 1h | 168 | ≈ 1 min | ≈ 5–10 min | +| 30d | 1h | 720 | ≈ 5 min | ≈ 30–60 min | +| 90d | 1h | 2160 | ≈ 15–30 min | ≈ 2–3 hours | +| 30d | 5m | 8640 | ≈ 1 hour | ≈ 6–12 hours | + +See the example Job in `examples/k8s/backfill-job.yaml` for a ready-to-run manifest. + +## Safety cap + +`safety.backfill_max_window` (default `90d`) is enforced at the CLI boundary so an accidental `--from 1y` fails fast. Raise it explicitly when needed: + +```yaml +safety: + backfill_max_window: 365d +``` + +## Audit trail + +After each group the backfill emits one gauge: + +``` +forecast_backfill_fits_last{group} +``` + +It records “this is how many fits the most recent backfill issued for this group”. + +## VM futureRetention + +Backfill writes historical timestamps (accepted by default in VictoriaMetrics). +Forecast curves still extend into the future, so the usual `futureRetention=2d` (or higher than `defaults.horizon`) still applies. + +## What backfill does **not** emit + +- Operational counters (`forecast_failures_total`, `forecast_run_duration_seconds`, leader-election, deploy-risk, etc.). +- Conditional forecasts, decomposition, drift gauges. +- Deviation ratios / outside-band flags (these are tied to live actuals). +- Band-coverage rolling fractions. + +These families describe live execution state and would be misleading when replayed at past timestamps. + +## Operational notes + +- Backfill is single-process and runs once (no scheduler). +- A config reload or change to `deploy_signals` resets active state on the next run. +- The example Kubernetes Job uses the same image as production and has `backoffLimit: 0` so a crashed walk can be retried with a narrower window. diff --git a/docs/operations/degraded-modes.md b/docs/operations/degraded-modes.md new file mode 100644 index 0000000..0b8808f --- /dev/null +++ b/docs/operations/degraded-modes.md @@ -0,0 +1,97 @@ +# Degraded-mode operations playbook + +The forecaster isolates failures per query, per series, and per group, and retries many transient errors automatically. As a result, several degraded states are invisible if you only watch the per-fit failure counter. + +This page lists the five operational failure modes worth monitoring, the metric that detects each, and the recovery path. + +The shipped example alerts in [`examples/alerts/promforecast-rules.yaml`](../../examples/alerts/promforecast-rules.yaml) include ready-to-tune rules for each row under the `promforecast.degraded.examples` group. + +## Failure-mode matrix + +| Failure mode | Detection signal | Recovery | +|-------------------------------|-------------------------------------------------------|----------| +| Datasource (PromQL read) unreachable | `forecast_source_retries_total` rate + `forecast_failures_total{reason=~"query_error\|query_timeout"}` | Check VM/Mimir/Thanos reachability, network policies, auth | +| Sink (`remote_write`) writing fails | `forecast_sink_write_failures_total` rate | Check TSDB reachability, `futureRetention`, auth headers | +| Partial group failure (one query degrades) | `forecast_failures_total{group, reason}` + `forecast_regressor_failures_total` | Per-query investigation | +| Sustained fit timeout | `forecast_slow_fits_total` + `forecast_failures_total{reason="fit_timeout"}` | Tighten `safety.fit_timeout`, lighter model, or shorter lookback | +| Stale snapshot (group hasn't refreshed) | `forecast_snapshot_age_seconds` + `forecast_last_run_timestamp_seconds` | Check pod health and the rows above | + +## What the forecaster does on each failure + +Failure isolation is layered so a single fault never escalates: + +| Layer | Caught by | Continues running? | +|----------------|------------------------------------|--------------------| +| Per-regressor | `required: false` (default) | Yes — forecast fits without the failing regressor | +| Per-query | `try/except` inside `run_group` | Yes — sibling queries continue | +| Per-series | `try/except` inside `_run_query` | Yes — sibling series continue | +| Per-group | `try/except` at top of `run_group` | Yes — sibling groups continue | +| Sink | `try/except` around `sink.write` | Yes — snapshot still served on `/metrics` | + +## Datasource read failure + +**Detection** +```promql +rate(forecast_source_retries_total[5m]) > 0.1 +or +rate(forecast_failures_total{reason=~"query_error|query_timeout"}[5m]) > 0.1 +``` + +**Recovery** +Check VictoriaMetrics / Mimir / Thanos health, network policies, and auth. Update `datasource.url` and reload if the endpoint moved. + +Shipped alert: `ForecastDatasourceUnreachable`. + +## Sink write failure + +**Detection** +```promql +rate(forecast_sink_write_failures_total[5m]) > 0 +``` + +**Recovery** +Check TSDB reachability, `futureRetention`, and auth headers in `sink.remote_write.headers`. + +Shipped alert: `ForecastSinkWriteFailing`. + +## Partial group failure + +**Detection** +`forecast_failures_total{group, reason}` and `forecast_regressor_failures_total`. + +**Recovery** +Filter failures by group + query in logs; fix the source (regressor rename, cardinality spike, model timeout) and reload. + +## Sustained fit timeout + +**Detection** +`forecast_slow_fits_total` or `forecast_failures_total{reason="fit_timeout"}`. + +**Recovery** +Raise `safety.fit_timeout`, switch to a lighter model, or shorten `lookback`. Isolate problematic series into their own group if needed. + +## Stale snapshot + +**Detection** +```promql +forecast_snapshot_age_seconds > 7200 +or +time() - forecast_last_run_timestamp_seconds > 7200 +``` + +**Recovery** +Check pod health (OOMKilled, `Recreate` strategy) and the rows above for the root cause. + +Shipped alert: `ForecastSnapshotStale`. + +## Putting it together + +Typical production alerting pages on `ForecastDatasourceUnreachable`, `ForecastSinkWriteFailing`, and `ForecastSnapshotStale`. Lower-severity rules (`ForecastModelFitFailing`, `ForecastSeriesStale`, quality/deviation) route to tickets. + +Each paged alert maps to a concrete recovery action in the matrix above. + +## Cross-references + +- [Reload verification](reload.md) — catches partial-config issues that lead to stale snapshots. +- [Telemetry](telemetry.md) — full catalogue of the operational metrics referenced here. +- [Cardinality budgeting](cardinality.md) — emission families assumed counted against `safety.max_total_series`. diff --git a/docs/operations/drift-explained.md b/docs/operations/drift-explained.md new file mode 100644 index 0000000..f591f7e --- /dev/null +++ b/docs/operations/drift-explained.md @@ -0,0 +1,90 @@ +# Drift change-correlation explainer + +When `forecast_drift_score` crosses `emission.drift.alert_threshold`, the forecaster automatically explains *why* the prediction shifted. + +It emits a structured `forecast_drift_explained` log event and bumps the counter `forecast_drift_explained_total{group, query, primary_suspect}` so dashboards can surface “which regressor most often explains drift on this query?” without log scraping. + +No extra config is required — the explainer activates whenever `emission.drift.enabled: true` (default). + +## What lands in the log event + +```json +{ + "event": "forecast_drift_explained", + "group": "node_capacity", + "query": "node_cpu_busy_pct", + "model": "AutoARIMA", + "series_key": "host-a", + "drift_score": 0.342, + "drift_threshold": 0.25, + "primary_suspect": "deploys", + "top_value_suspect": "deploys", + "top_correlation_suspect": "deploys", + "top_regressor_correlation": 0.82, + "top_regressor_correlation_confidence": "high", + "correlation_window_size": 30, + "regressor_values_at_drift": { + "deploys": 1.0, + "hour_of_day": 14 + } +} +``` + +| Field | Meaning | +|-------------------------------------|---------| +| `drift_score` / `drift_threshold` | The crossing that triggered the explanation | +| `top_value_suspect` | Regressor with largest absolute value at drift moment | +| `top_correlation_suspect` | Regressor whose history best predicts drift over the rolling window | +| `top_regressor_correlation` | Pearson correlation `[-1, 1]` (`null` if insufficient data) | +| `top_regressor_correlation_confidence` | `high` / `medium` / `low` / `insufficient_data` | +| `primary_suspect` | Label used on the counter (prefers correlation when confidence is `high` or `medium`) | +| `regressor_values_at_drift` | Snapshot of every regressor’s latest value at the drift moment | + +## Confidence bands + +| Band | Meaning | +|---------------------|---------| +| `high` | ≥0.7 correlation **and** ≥20 samples | +| `medium` | ≥0.4 correlation **and** ≥8 samples | +| `low` | Correlation exists but is weak | +| `insufficient_data` | Fewer than 4 valid pairs | + +## Tuning the rolling window + +```yaml +emission: + drift: + alert_threshold: 0.25 + correlation_window: 60 # one week at hourly refresh +``` + +Smaller windows react faster but are noisier. Larger windows give smoother correlations but lag when the driver changes. + +## Building alerts on the counter + +```promql +# A query keeps drifting and the same regressor explains it +sum by (group, query, primary_suspect) ( + rate(forecast_drift_explained_total[1h]) +) > 0.5 +``` + +Or surface the top-N causes fleet-wide: + +```promql +topk(10, sum by (primary_suspect) (increase(forecast_drift_explained_total[24h]))) +``` + +## When the counter stays at zero + +The counter only bumps when drift actually crosses the threshold. +A deployment whose forecasts never cross the threshold (or whose threshold is too high) leaves the counter at zero — this is expected, not a bug. + +## When `primary_suspect="unknown"` + +This appears when: +- No regressors are configured. +- Every regressor value is non-finite at the drift moment. +- No regressor produced a finite correlation or absolute value. + +The structured log still fires so you can see drift happened. diff --git a/docs/operations/forecast-stability.md b/docs/operations/forecast-stability.md new file mode 100644 index 0000000..8a87f2e --- /dev/null +++ b/docs/operations/forecast-stability.md @@ -0,0 +1,61 @@ +# Forecast stability and model agreement + +Two gauges catch a thrashing model or a set of models that disagree about the future **before** downstream alerts react. + +| Metric | Question it answers | +|--------|---------------------| +| `forecast_revision_magnitude{id, group, model, horizon}` | How much did this model’s `` forecast move since the last fit? | +| `forecast_model_disagreement{id, group, horizon}` | How much do the configured models disagree about this `` forecast? (explicit mode only) | + +## Revision magnitude + +The forecaster caches the previous central `yhat` value at each accuracy horizon. On the next fit it emits: + +``` +forecast_revision_magnitude = |new − old| / max(|old|, 1e-9) +``` + +A 30 % shift becomes `0.3`. The metric is **unsigned** — operators care about “the forecast moved a lot”, not the direction. + +Shipped example alert: + +```yaml +- alert: ForecastUnstable + expr: forecast_revision_magnitude > 0.3 + for: 20m + labels: + severity: info +``` + +**Note:** this is distinct from `forecast_drift_score`, which measures **input** drift (how much the underlying signal moved). Revision measures **output** drift (how much the model’s prediction moved). The two often correlate but can diverge — a stable input that suddenly produces wildly different forecasts usually signals a model over-reacting to noise or a real regime change. + +On the very first fit per (series, model, horizon) the value is `NaN` and the row is dropped. Steady-state behaviour starts after two refresh cycles. + +## Model disagreement + +In explicit mode (`auto_select: false` with 2+ models) each model produces its own forecast. The disagreement gauge is the **coefficient of variation** across the per-model central forecasts at the same horizon: + +``` +forecast_model_disagreement{horizon="24h"} = CoV(yhat_24h across models) +``` + +`0.0` = perfect agreement. Higher values indicate no single model is confident — a useful “should I switch to `auto_select`?” signal. + +Shipped example alert: + +```yaml +- alert: ForecastModelsDisagree + expr: forecast_model_disagreement > 0.4 + for: 30m + labels: + severity: info +``` + +Suppressed in `auto_select` and ensemble modes because only one model’s `yhat` is emitted per series. + +## Cardinality + +- `forecast_revision_magnitude`: `N_series × N_emitted_models × N_horizons` per query with `accuracy.evaluate: true`. +- `forecast_model_disagreement`: `N_series × N_horizons` per query in explicit mode with 2+ models. + +`promforecast validate` counts both families against the global budget (see [cardinality.md](cardinality.md)). diff --git a/docs/operations/inspect.md b/docs/operations/inspect.md new file mode 100644 index 0000000..f34c460 --- /dev/null +++ b/docs/operations/inspect.md @@ -0,0 +1,101 @@ +# Forecast inspector + +`promforecast inspect` is the debugging tool for “why does this one series have a weird forecast?”. + +It pulls the exact series from the TSDB, runs the configured fit pipeline exactly as the live forecaster does, and prints the inputs, the fit result, the forecast curve, and (optionally) the per-fold backtest. + +## When to use it + +- A specific series’s forecast in Grafana looks wrong (too conservative, weirdly seasonal, flat, predicting negatives). +- `forecast_quality_score` for one series is low while neighbours are fine. +- You need to compare model behaviour without a config change (run once with `--model AutoARIMA`, again with `--model SeasonalNaive`). +- You want to see the exact per-fold backtest predictions that produced a high MAPE. + +## Basic usage + +```bash +promforecast inspect \ + --config /etc/promforecast/config.yaml \ + --query node_cpu_busy_pct \ + --series-filter '{instance="host-a.example.com"}' +``` + +**Default text output** (first 10 forecast rows): + +``` +Query: node_cpu_busy_pct (group node_capacity) +Model: SeasonalNaive +Series: device=cpu0, instance=host-a.example.com +Step / season / horizon: 300s / 288 / 288 steps + +Input stats: + point_count : 4032 + gaps_filled : 12 + largest_gap_seconds: 600.0 + lookback_start : 2026-05-10T12:00:00+00:00 + lookback_end : 2026-05-24T12:00:00+00:00 + +Preprocessing applied: + gaps_filled : {'forward_fill': 12} + outliers_replaced : 0 + change_points : 0 + imputed_fraction : 0.0029 + effective_lookback_seconds: 1209600.0 + +Forecast (first 10 of 288 rows): + 2026-05-24T12:05:00+00:00 yhat=42.1234 [95: 30.0000..54.0000] + ... +``` + +Use `--json` for machine-readable output (ideal for `jq` or CI artifacts). + +## Backtest mode + +`--backtest` runs the same backtest harness the live runner uses and prints per-step predictions vs actuals: + +```bash +promforecast inspect --config cfg.yaml --query Q --series-filter '{...}' --backtest +``` + +Adds MAPE/MASE and the hold-out rows. +Useful pattern for model comparison: + +```bash +promforecast inspect ... --model AutoARIMA --backtest --json > a.json +promforecast inspect ... --model SeasonalNaive --backtest --json > b.json +``` + +## Flags + +| Flag | Purpose | +|---------------------|---------| +| `--config` | Path to the live forecaster YAML (required) | +| `--query` | Query `id` to inspect (required) | +| `--series-filter` | PromQL selector when the query returns multiple series | +| `--horizon` | Override `defaults.horizon` | +| `--lookback` | Override `defaults.lookback` | +| `--model` | Use this specific model (must be in `defaults.models`) | +| `--backtest` | Also run and print the backtest | +| `--json` | Output full report as JSON | + +## Debugging walkthroughs + +**Sparse signal misfit** +Symptom: forecast collapses to a constant, low quality score. +Look at “Input stats” for low `point_count` or high `imputed_fraction` → fix on the input side (extend lookback, fix scrape gaps, or add `cold_start`). + +**Regressor missing** +Symptom: forecast ignores a known driver (e.g. deploy spike). +Check `forecast_regressor_failures_total` for the query/regressor pair. The inspector does **not** fetch regressors (keeps cost bounded). + +**Change-point not detected** +Symptom: forecast anchored to pre-incident level. +Look at “Preprocessing applied / change_points”. If `0`, the detector is disabled or not sensitive enough. + +## Limitations + +- Fits **one model** per invocation (run multiple times to compare). +- Conditional forecasts, decomposition, and conformal calibration are **not** shown (inspector mirrors the standard fit only). +- Bands are raw model output (narrower than live conformal bands). +- No regressor values are fetched or applied. +- Count-aware safeguards (`clip_non_negative`, `round_to_integer`) **are** applied so output matches what the exporter would emit. diff --git a/docs/operations/quality-score.md b/docs/operations/quality-score.md new file mode 100644 index 0000000..46fb685 --- /dev/null +++ b/docs/operations/quality-score.md @@ -0,0 +1,58 @@ +# Forecast quality score + +`forecast_quality_score{id, group, model, horizon}` is a composite 0–1 gauge that multiplies three sub-scores: + +1. **Accuracy** — backtest MAPE (continuous) or MASE (intermittent/count). +2. **Band score** — penalises wide confidence bands relative to typical actual value. +3. **Coverage** — fraction of expected points present, scaled by `(1 - imputed_fraction)`. + +A score near `1` means the model fits cleanly with tight, well-calibrated bands. A score near `0` means “don’t trust this forecast”. + +See [`forecaster/src/promforecast/quality.py`](../../forecaster/src/promforecast/quality.py) for the exact formula. + +## Per-horizon segmentation + +The 1h forecast is often far more reliable than the 24h forecast. The quality score is emitted **per horizon** so you can gate short-horizon deviation alerts and long-horizon capacity alerts independently: + +``` +forecast_quality_score{id="...", model="...", horizon=""} # aggregated +forecast_quality_score{id="...", model="...", horizon="1h"} # 1h only +forecast_quality_score{id="...", model="...", horizon="24h"} # 24h only +``` + +The aggregated row (`horizon=""`) is always emitted, so existing alerts that don’t filter on horizon continue to work unchanged. Per-horizon rows appear whenever `accuracy.evaluate: true`. + +### Recommended alert gating + +```yaml +# Short-horizon deviation alert +- alert: ForecastDeviation + expr: | + forecast_deviation_outside_band{level="95"} == 1 + and on (id, group, model) forecast_quality_score{horizon="1h"} > 0.6 + +# Long-horizon capacity alert +- alert: ForecastCapacityWarning + expr: | + node_filesystem_avail_bytes_forecast < 0 + and on (id, group, instance) forecast_quality_score{horizon="24h"} > 0.5 +``` + +The shipped example alerts in [`examples/alerts/promforecast-rules.yaml`](../../examples/alerts/promforecast-rules.yaml) use the aggregated score for general rules (`ForecastQualityLow`, `ForecastQualityZero`) and horizon-specific rules where appropriate. + +## Cardinality + +Each (series, model) emits **1 aggregated row + 1 row per configured accuracy horizon**. +With the default 2 horizons (`1h`, `24h`) this triples the per-(series, model) quality footprint (1 + 2 = 3 rows). +`promforecast validate` counts the multiplier under the “quality lines” row. + +## Why multiplicative? + +Any single weak dimension drags the overall score down. Common patterns: + +| Pattern | Likely cause | +|--------------------------------|--------------| +| Score stuck at 0 | Degenerate MAPE (all-zero actuals), 0 % band, or 0 % coverage | +| 1h score >> 24h score | Expected long-horizon decay — use horizon-specific gating | +| Score dropped after deploy | Check `forecast_revision_magnitude` and `forecast_drift_score` | +| Cold-start row near cap | Expected — cold-start score is deliberately capped | diff --git a/docs/operations/right-sizing.md b/docs/operations/right-sizing.md new file mode 100644 index 0000000..4f90fcb --- /dev/null +++ b/docs/operations/right-sizing.md @@ -0,0 +1,91 @@ +# Right-sizing recommendations + +The forecaster already knows the future trajectory of every workload. +`recommendations.resource_sizing` turns that into a per-workload “you can downsize from X to Y” gauge that operators can park on a Grafana panel and act on systematically. + +The recommendation is read-only — no auto-apply, no controller integration. + +## Enable it + +```yaml +recommendations: + resource_sizing: + enabled: true + headroom_pct: 30 + quality_floor: 0.6 + workloads: + - workload: api-server + kind: cpu + forecast_query: api_server_cpu_usage_pct + current_promql: > + sum(kube_pod_container_resource_requests{ + workload="api-server", resource="cpu" + }) + - workload: api-server + kind: memory + forecast_query: api_server_memory_usage_bytes + current_promql: > + sum(kube_pod_container_resource_requests{ + workload="api-server", resource="memory" + }) + # current_promql is optional — recommendation still emits, + # just without savings_pct + - workload: search + kind: cpu + forecast_query: search_cpu_usage_pct +``` + +After the next refresh, three sibling gauges appear: + +``` +forecast_recommended_resource_size{workload="api-server", kind="cpu", confidence="high"} 2.6 +forecast_recommended_resource_current{workload="api-server", kind="cpu"} 4.0 +forecast_recommended_resource_savings_pct{workload="api-server", kind="cpu", confidence="high"} 35.0 +``` + +| Field | Default | Meaning | +|------------------|---------|---------| +| `enabled` | `false` | Opt-in | +| `headroom_pct` | `30` | Added to predicted peak | +| `quality_floor` | `0.6` | Minimum mean quality required to emit a recommendation | +| `workloads` | `[]` | List of (workload, kind, forecast_query, optional current_promql) | + +### How the recommendation is computed + +- Reads the forecast snapshot of `forecast_query`. +- Takes the peak (`max yhat_upper` at the highest confidence level, falling back to `yhat`). +- Applies `headroom_pct`. +- Compares to `current_promql` (if provided) to compute `savings_pct`. + +## Reading the recommendation + +| Value | Meaning | +|------------------------|---------| +| Positive `savings_pct` | Predicted peak + headroom is below current limit → safe to downsize | +| Negative `savings_pct` | Predicted peak exceeds current limit → needs upsizing | +| `confidence="low"` | Forecast quality too low to trust — improve the model first | +| Row missing | Quality below `quality_floor` for all models on this query | + +## Worked examples + +**Top-10 cost-saving opportunities** (high-confidence only) + +```promql +topk(10, forecast_recommended_resource_savings_pct{confidence="high"} > 0) +``` + +**Pair with absolute recommended size** + +```promql +forecast_recommended_resource_size{confidence="high"} + * on(workload, kind) group_left() + forecast_recommended_resource_savings_pct{confidence="high"} > 0 +``` + +**Bursty workloads** +A daily peak well above average pins the recommendation to the upper band — intentional. If occasional pressure is acceptable, raise `headroom_pct`. + +## Cardinality + +Up to three rows per configured workload. +`promforecast validate` accounts for the upper bound under “resource sizing”. diff --git a/docs/operations/telemetry.md b/docs/operations/telemetry.md index 8df8744..7071233 100644 --- a/docs/operations/telemetry.md +++ b/docs/operations/telemetry.md @@ -1,80 +1,146 @@ -# OpenTelemetry tracing +# Forecaster telemetry -Opt-in distributed tracing for the full query → fit → export pipeline. -Useful when metrics alone don’t explain a slow group, stuck regressor, or runaway fit. +Two layers of observability: -## Configuring +1. **Metrics** — always on. Per-series freshness, per-group run state, per-fit duration, failure counters, and the new per-series staleness gauge (documented below). Most families are described next to the feature that emits them. +2. **Tracing** — opt-in OpenTelemetry spans for the full query → fit → export pipeline. + +## Snapshot age (per group) + +`forecast_snapshot_age_seconds{group}` is a server-side computed gauge of seconds since the most recent successful run for the group — the same signal as `time() - forecast_last_run_timestamp_seconds` but pre-subtracted so dashboards and the `ForecastSnapshotStale` alert can read a single gauge instead of a PromQL subtraction. When the group has never produced a snapshot the value is 0; alert authors gating on the value should pair it with `forecast_last_run_timestamp_seconds > 0` to avoid treating "never ran" as healthy. + +The [degraded-mode playbook](degraded-modes.md) walks through the five operational failure modes this gauge surfaces, including the bounded sink-failure and source-read counters that distinguish the underlying causes. + +## Per-series staleness + +`forecast_data_staleness_seconds{id, group, model, ...}` is a per-(series, model) gauge of seconds since the last successful forecast emission. + +It complements the group-level `forecast_last_run_timestamp_seconds`: + +| Signal | Question it answers | +|--------|---------------------| +| `forecast_last_run_timestamp_seconds{group}` | Has the *group* refreshed recently? | +| `forecast_data_staleness_seconds{id, group, model}` | Is *this specific series* still being fit? | + +The staleness clock starts at the first successful fit for that (series, model) tuple and resets on every subsequent success. Failed fits leave the clock running, so the gauge grows until the series recovers. + +**Shipped alert** (`examples/alerts/promforecast-rules.yaml`): ```yaml -telemetry: - enabled: true - service_name: promforecast # OTel resource service.name - sample_ratio: 0.1 # head-based sampling ratio (0..1) - otlp: - endpoint: otel-collector:4317 # OTLP/gRPC endpoint - insecure: true # cleartext (common inside cluster) - headers: {} # optional auth/tenant headers - timeout: 10s +- alert: ForecastSeriesStale + expr: forecast_data_staleness_seconds > 14400 # ~4 × default refresh interval + for: 15m + labels: + severity: warning ``` -Requires the optional `[otel]` extra: +Tune the threshold to roughly four times your `server.refresh_interval`. -```bash -pip install 'promforecast[otel]' -``` +### Never-succeeded series -The official container image ships with it; enabling the feature needs no custom build. +The gauge is **absent** until the first successful fit. This prevents false positives on fresh installs. Use these complementary signals for first-attempt failures: -## When disabled +- `forecast_failures_total{reason=~"fit_error|fit_timeout|...", group="..."} > 0` +- `forecast_series_dropped_total{reason="min_points", group="..."} > 0` -`telemetry.enabled: false` (default) is **completely free**: -- The `opentelemetry` package is never imported. -- Every `Tracer.span(...)` becomes a no-op (`contextlib.nullcontext`). -- No extra dependencies are required. +For a fresh-install overview, run `promforecast warmup`. -This zero-cost path is explicitly tested in `forecaster/tests/test_telemetry.py`. +## Self-observability metrics -## Spans emitted +Three families track the forecaster’s own internal health: -| Span | Attributes | Context | -|--------------------------|-------------------------------------------------|---------| -| `promforecast.group_run` | `group`, `query_count` | Each group refresh tick | -| `promforecast.query` | `group`, `query` | Each PromQL fetch (one per rendered template for discovery) | -| `promforecast.fit` | `model`, `horizon`, `freq`, `season_length` | Every `fit_predict` (forecast + each backtest) | +| Metric | Question it answers | +|--------|---------------------| +| `forecast_model_fit_success_ratio{group, model, window}` | What fraction of recent fits for this model in this group succeeded? (1h / 24h windows) | +| `forecast_regressor_stability_score{group, query, regressor_id}` | Has this regressor’s recent value distribution drifted from its lookback baseline? | +| `forecast_group_memory_bytes{group}`
`forecast_group_cpu_seconds_total{group}` | Which group dominates memory or CPU usage? | + +- **Fit success ratio** rolls successes and failures into rolling windows. A value below ~0.9 for more than one refresh cycle usually indicates a model silently failing on a subset of series. +- **Regressor stability** is a 0–1 score (1.0 = no drift). Low scores point to noisy regressors that degrade conditional forecasts without raising fit errors. +- **Group resource attribution** gives comparative (not absolute) memory and CPU per group. + +## Service-level health composite + +`forecast_service_health_score{, aggregator, weight_by, series_count}` is the per-bucket aggregate of `forecast_quality_score` defined under the operator-configured [`governance.health_score`](../governance/forecast-health.md) block. + +| Label | Meaning | +|--------------------|------------------------------------------------------------------------------------------------------| +| `` | One label per dim listed in `governance.health_score.aggregate_by` (e.g. `service`, `environment`). | +| `aggregator` | Currently always `weighted_mean` — documents which aggregation produced the value. | +| `weight_by` | `uniform` or `forecast_series_count` — documents the weight interpretation. | +| `series_count` | Number of `forecast_quality_score` rows that contributed to the bucket. | + +Off until the block is configured. Use to surface "our forecasts are X% reliable today" per service/environment on an executive dashboard. See [governance/forecast-health.md](../governance/forecast-health.md) for the worked recipe. -Exceptions are automatically recorded; errored spans are marked `ERROR` with the exception message. +## Right-sizing recommendations -## Sampling +Three sibling gauges describe the per-workload sizing recommendation derived from the forecast peak: -`sample_ratio` is a head-based sampler. -- `1.0` traces everything (great for dev). -- `0.01` traces 1 % (typical production). +| Metric | Meaning | +|----------------------------------------------|------------------------------------------------------------------------------------------------------------------| +| `forecast_recommended_resource_size{workload, kind, confidence}` | Recommended limit/request. Sample value is in the metric's native units (CPU cores, memory bytes). | +| `forecast_recommended_resource_current{workload, kind}` | Operator-reported current limit/request (only when `current_promql` is wired). | +| `forecast_recommended_resource_savings_pct{workload, kind, confidence}` | `(current − recommended) / current * 100`. Positive = safe to downsize; negative = upsize required. | -Parent-based context propagation ensures all child spans of a sampled group run stay together. +The `confidence` label categorises the underlying `forecast_quality_score` (`high|medium|low`). Off until [`recommendations.resource_sizing.enabled: true`](right-sizing.md) is set; the quality gate uses the **mean** of finite scores for the workload's `forecast_query`. -## SDK missing +## Cost of action on capacity thresholds -If `telemetry.enabled: true` but the `[otel]` extra is not installed, the forecaster logs a clear warning and continues without tracing: +`forecast_threshold_action_cost{id, group, name, type}` is a config-derived gauge whose sample value is the operator-configured cost. The `type` label is free text (`usd`, `eur`, `engineer_hours`, ...) so non-monetary cost framings coexist with currency ones in a single family. Tied to threshold lifecycle: a threshold removed from YAML drops its row on the next refresh. See [outputs/capacity.md](../outputs/capacity.md#ranking-by-cost-of-action) for the join with `forecast_time_to_threshold_seconds`. -```json -{ - "event": "telemetry_disabled_sdk_missing", - "error": "No module named 'opentelemetry'", - "hint": "install the optional dependency: pip install 'promforecast[otel]'" -} +## Drift change-correlation explainer + +`forecast_drift_explained_total{group, query, primary_suspect}` counts threshold-crossing events on `forecast_drift_score`. The `primary_suspect` label names the regressor most likely to explain the drift — see [operations/drift-explained.md](drift-explained.md). Always paired with a `forecast_drift_explained` structured log event carrying every regressor value at the drift moment, the rolling Pearson correlation, the `last_config_reload_timestamp`, and the cumulative `change_point_count` so the investigation hypothesis isn't limited to regressor changes. + +## OpenTelemetry tracing + +Opt-in distributed tracing for the query → fit → export pipeline. Useful when metrics alone don’t explain a slow group, stuck regressor, or runaway fit. + +### Configuring + +```yaml +telemetry: + enabled: true + service_name: promforecast + sample_ratio: 0.1 + otlp: + endpoint: otel-collector:4317 + insecure: true + headers: {} + timeout: 10s ``` -Tracing is observability, never a correctness requirement. +Requires the optional `[otel]` extra (`pip install 'promforecast[otel]'`). The official container image ships with it. + +### When disabled + +`telemetry.enabled: false` (default) is completely free — no package is imported, every span is a no-op. + +### Spans emitted + +| Span | Attributes | Context | +|--------------------------|-------------------------------------------------|---------| +| `promforecast.group_run` | `group`, `query_count` | Each group refresh | +| `promforecast.query` | `group`, `query` | Each PromQL fetch | +| `promforecast.fit` | `model`, `horizon`, `freq`, `season_length` | Every `fit_predict` | + +Exceptions are automatically recorded and marked `ERROR`. + +### Sampling + +Head-based sampler controlled by `sample_ratio`. Parent-based context propagation keeps child spans together. + +### SDK missing + +If the `[otel]` extra is absent, the forecaster logs a clear warning and continues without tracing. -## Lifecycle +### Lifecycle -Tracing initialises once at boot in `main.py`. -The FastAPI lifespan teardown calls `tracer.shutdown()` to flush the `BatchSpanProcessor`. -A normal Kubernetes 30 s SIGTERM grace period is sufficient. +Initialised once at boot. FastAPI lifespan teardown flushes the `BatchSpanProcessor`. A normal Kubernetes 30 s SIGTERM is sufficient. -**Not hot-reloadable.** Changing `telemetry.*` via `/-/reload` has no effect on the running tracer. Restart the pod to pick up new settings (deliberate trade-off to avoid losing in-flight spans or double-initialisation issues). +**Not hot-reloadable** — changing `telemetry.*` requires a pod restart. -## Non-goals +### Non-goals -- Metrics over OTLP — Prometheus exposition on `/metrics` remains the metric API. -- Auto-instrumentation — only the three manual spans above are emitted. Full `opentelemetry-instrumentation-*` packages are deliberately omitted to keep span volume low and focused. +- Metrics over OTLP (Prometheus exposition on `/metrics` remains the metric API). +- Auto-instrumentation (only the three manual spans above are emitted). diff --git a/docs/operations/validate.md b/docs/operations/validate.md new file mode 100644 index 0000000..a530b4f --- /dev/null +++ b/docs/operations/validate.md @@ -0,0 +1,69 @@ +# `promforecast validate` + +The validator is the first gate CI runs against every config PR. It fails fast on bad YAML, prints a full cardinality estimate, and can optionally run a cost projection or a live sample fit. + +```bash +promforecast validate --config /path/to/config.yaml \ + [--strict-cardinality] \ + [--estimate-cost] \ + [--dry-run [--datasource-url URL]] +``` + +## Modes + +| Flag | What it does | +|--------------------------|--------------| +| (default) | Parse + schema validation + cardinality estimate | +| `--strict-cardinality` | Same as default, but exits `1` if projected total exceeds `safety.max_total_series` | +| `--estimate-cost` | Runs synthetic-data fits per model and prints projected CPU/memory budget per refresh | +| `--dry-run` | Runs a real sample fit against the datasource and prints first 10 forecast points + MAPE preview | + +Exit codes: +- `0` — success +- `1` — validation failure (schema, cardinality, etc.) +- `2` — dry-run or cost-estimate failure (fit raised) + +The command never modifies the config and only touches the datasource when `--dry-run` is used. + +## Cardinality estimate + +Printed as a per-family breakdown plus grand total. The estimate assumes every query returns `safety.max_series_per_query` series (worst case). Real deployments usually emit fewer lines, but the worst-case number is what can trip the global cap. + +See [cardinality.md](cardinality.md) for the full multiplier table. + +## Cost estimate (`--estimate-cost`) + +Runs one synthetic fit per configured model (720-point hourly sinusoid + trend) and projects the wall-clock time and peak memory across the real config. + +Output example: + +``` +Estimated runtime cost (synthetic-data fit per model): + group node_capacity: + refresh_interval 3600.0s + estimated series 1000 + per-fit AutoARIMA 0.142s + per-fit SeasonalNaive 0.003s + total fit time / refresh 145.00s + peak memory (tracemalloc) 8421376 bytes + + total fit time across groups (per refresh): 145.00s +``` + +A `WARN:` line appears when a group’s projected total exceeds its `refresh_interval`. This is advisory only — the command still exits `0` unless `--strict-cardinality` is also used. + +**Accuracy note** +The estimator is order-of-magnitude only. Real fits on noisy data are typically 1–4× slower. Regressor fan-out and parallelism are not modelled. Treat large discrepancies (e.g. 100×) as real problems. + +## Recommended CI shape + +```yaml +- name: Validate promforecast config + run: | + promforecast validate \ + --config charts/promforecast/values.yaml \ + --strict-cardinality \ + --estimate-cost +``` + +This combination catches schema errors, oversized cardinality, and unexpectedly expensive model sets before they reach production. diff --git a/docs/operations/warmup.md b/docs/operations/warmup.md new file mode 100644 index 0000000..935a91a --- /dev/null +++ b/docs/operations/warmup.md @@ -0,0 +1,100 @@ +# Warm-up status + +Right after `helm install` (or any restart that clears Prometheus data), the key question is: +**“What is still loading, and how long until forecasts appear?”** + +The `GET /forecast/warmup` endpoint answers exactly that — it shows per-query progress toward the first usable forecast. + +## Enabling + +```yaml +server: + expose_warmup_endpoint: true # default: false +``` + +Same network-level access control as `/-/reload` and `/forecast/preview`. + +## Endpoint + +```bash +curl -s http://forecaster:9091/forecast/warmup | jq . +``` + +```json +{ + "groups": [ + { + "name": "node_capacity", + "queries": [ + { + "id": "node_cpu_busy_pct", + "status": "ready", + "points_available": 200, + "points_needed": 100, + "blocked_by": "none", + "eta_seconds": 0 + }, + { + "id": "node_memory_used_bytes", + "status": "warming", + "points_available": 40, + "points_needed": 100, + "blocked_by": "min_points", + "eta_seconds": 3600 + } + ] + } + ] +} +``` + +**blocked_by** values: + +| Value | Meaning | +|-------------------|---------| +| `none` | Ready — next refresh will emit a forecast | +| `min_points` | Below `defaults.min_points` | +| `lookback` | No samples in the configured lookback window | +| `regressor:` | Required custom regressor is missing samples | +| `discovery` | Templated query — discovery probe failed | +| `change_point` | Reserved for future use | + +## Companion CLI + +```bash +promforecast warmup --url http://forecaster:9091 +``` + +Prints a clean table: + +``` +group query status points needed eta blocked_by +------------- ------------------------- ------- ------ ------ --- ---------- +node_capacity node_cpu_busy_pct ready 200 100 - none +node_capacity node_memory_used_bytes warming 40 100 1h min_points +``` + +Flags: +- `--json` → raw JSON output +- `--timeout ` → HTTP deadline (default 30s) + +The CLI exits `0` when every query is `ready`, `1` otherwise, `2` on network error — perfect for CI/pre-flight scripts. + +## Cost + +Each call issues one cheap `query_range` per non-event, non-templated query (plus one extra per required custom regressor). +All probes run concurrently, so wall time equals the slowest probe. + +The endpoint is opt-in because even these light probes are unnecessary once the system is warm. + +## Limitations + +- Event-profile queries (`data_profile: events`) and templated queries (`discover:`) always report `ready`. +- `eta_seconds` is a simple linear estimate (“one sample per step”). It can under- or over-estimate on sparse data or backfills. +- Only **required** custom regressors are probed; optional ones are ignored. + +## What to expect on a fresh install + +A typical 1 h refresh / 14 d lookback / `min_points: 200` setup needs ~17 hours of data before the first fit. +`warmup` will show `lookback` → `min_points` → `none`. +Once all queries are `ready`, the next refresh emits forecasts and the normal quality/accuracy/deviation metrics begin populating. diff --git a/docs/outputs/capacity.md b/docs/outputs/capacity.md index eb7ea31..3accf0c 100644 --- a/docs/outputs/capacity.md +++ b/docs/outputs/capacity.md @@ -170,6 +170,71 @@ same labelset is the operator-visible "when" for the breach. See `examples/alerts/promforecast-rules.yaml` for the `ForecastWillBreach` rule template. +### Ranking by cost of action + +`time_to_threshold_seconds` alone tells you *which* capacity event is +closest. It does not tell you which one is most expensive to ignore. +Add an `action_cost` annotation to surface a cost gauge that dashboards +and alerts can multiplex against the ETA: + +```yaml +queries: + - id: object_storage_used_bytes + promql: sum by (bucket) (object_storage_used_bytes) + thresholds: + - name: critical + value: 1099511627776 # 1 TiB + comparator: gt + alert_within: 7d + severity: critical + action_cost: + type: usd + value: 50000 # tier-upgrade cost if we breach + - name: backup_warn + value: 5368709120 # 5 GiB + comparator: gt + action_cost: + type: usd + value: 50 # nightly archive purge cost +``` + +| Field | Required | Meaning | +|---------|----------|--------------------------------------------------------------------------------------------------------| +| `type` | yes | Free-text label. Use a currency (`usd`, `eur`) or a non-monetary unit (`engineer_hours`, `priority`). | +| `value` | yes | Numeric cost. Must be `> 0` — zero or negative values would silently rank below every meaningful row. | + +Output: + +``` +forecast_threshold_action_cost{ + id="object_storage_used_bytes", group="capacity", + name="critical", type="usd" +} 50000 +``` + +The gauge is config-derived: one row per `(group, query, threshold)` +with `action_cost` configured, independent of model / series fan-out. +A threshold removed from YAML disappears from `/metrics` on the next +refresh (the family is rebuilt from current config each run). + +Rank by cost in Grafana: + +```promql +# Top 10 capacity events by cost-to-ignore — gated to "within 24h". +topk( + 10, + forecast_threshold_action_cost + * on(id, group, name) group_left() + (forecast_time_to_threshold_seconds < 86400) +) +``` + +`type` is free text on purpose: `usd` and `engineer_hours` coexist in +a single family so different teams can rank by their own currency +without forking the metric. Pin the label in your queries +(`forecast_threshold_action_cost{type="usd"}`) so a panel reports a +single currency and never sums apples with oranges. + ## Forecast growth rate Configure per-query growth-rate emission to publish how fast the @@ -291,6 +356,8 @@ All the capacity-style families count against the global - Will-breach: `N_series × N_models × N_thresholds_with_alert × (1 + N_levels)` per query — only thresholds with `alert_within` + `severity` count. - Growth rate: `N_series × N_models × N_windows` per query that opts in. +- Action cost: one line per threshold with `action_cost` configured. + No model / series fan-out — the metric is config-derived. Run `promforecast validate config.yaml` after adding thresholds or growth-rate emission to confirm the new totals fit within your safety diff --git a/docs/production-checklist.md b/docs/production-checklist.md new file mode 100644 index 0000000..45ff82a --- /dev/null +++ b/docs/production-checklist.md @@ -0,0 +1,70 @@ +# Production-readiness checklist and sizing guide + +A single reference for the questions an operator must answer before going live: how much CPU/memory to provision, which deployment shape to pick, and which knobs to tick off before flipping production traffic. + +Detailed how-tos live in the linked pages; this page integrates them. + +## Sizing (rule of thumb) + +CPU cost is dominated by per-series fits. AutoARIMA is the heaviest built-in model; SeasonalNaive is essentially free. + +| Series count | Refresh interval | Recommended CPU | Recommended memory | Replicas (HA mode) | +|--------------|------------------|-----------------|---------------------|--------------------| +| < 100 | 1h | 200m | 256Mi | 1 (no HA) | +| 100 – 500 | 1h | 500m | 512Mi | 1 (no HA) | +| 500 – 2,000 | 1h | 1 | 1Gi | 2 (HA) | +| 2,000 – 10,000 | 1h | 2 | 2Gi | 2 (HA) | +| 10,000 – 50,000 | 30m | 4 | 4Gi | 3 (HA) | +| > 50,000 | varies | — | — | Horizontal sharding (v1.10) | + +These are upper bounds. Use `promforecast validate --estimate-cost` for a config-specific projection. + +**Tightening refresh interval** +A 30-minute refresh roughly doubles CPU vs. 1h; a 5-minute refresh costs ~12×. Most operators stay at 30m or slower unless short-horizon SLOs require it. + +**Adding emission families** +Each v1.6+ family multiplies cardinality. Always re-run `promforecast validate --strict-cardinality` after enabling a new family. See [cardinality.md](cardinality.md) for the per-family table. + +## Reference architectures + +| Shape | When to use | Helm values | +|--------------------------------|-------------|-------------| +| **A. Single-replica small** | < 500 series, 30 s `/metrics` gap acceptable | `replicas: 1`
`highAvailability.enabled: false` | +| **B. Leader-elected medium** | > 500 series or `/metrics` availability required | `replicas: 2`
`highAvailability.enabled: true` | +| **C. Per-cluster + central** | Multi-cluster fleets | See [multi-cluster architectures](architecture/multi-cluster.md) | + +## Production-readiness checklist + +| Item | Expected end state | Reference | +|------|--------------------|-----------| +| `sink.remote_write.enabled: true` | Chart default; standalone installs set explicitly | [emission-paths.md](reference/emission-paths.md) | +| Scrape-drop applied | `serviceMonitor.dropForecastSnapshots: true` (or equivalent metric_relabel_configs) | [emission-paths.md](reference/emission-paths.md) | +| `ServiceMonitor` present | Forecaster operational metrics scraped (not dropped forecast snapshots) | [values.yaml](../charts/promforecast/values.yaml) | +| Alerts loaded | Shipped `promforecast-rules.yaml` rules deployed and tuned | [examples/alerts/promforecast-rules.yaml](../examples/alerts/promforecast-rules.yaml) | +| Dashboards provisioned | Five shipped Grafana dashboards installed | [dashboards/grafana/](../dashboards/grafana/) | +| Backfill considered | Optional 30 d backfill run so SLO dashboards are useful from day one | [backfill.md](operations/backfill.md) | +| Cost estimate run | `promforecast validate --estimate-cost` confirms projected fit time fits inside `refresh_interval` | [validate.md](operations/validate.md) | +| Quality gate on predictive autoscaling | Any HPA reading a forecast metric pairs it with `forecast_quality_score > 0.6` | [predictive-autoscaling.md](predictive-autoscaling.md) | +| Fleet what-if disabled in small installs | `server.expose_what_if_endpoint: false` unless resource headroom exists | [what-if-fleet.md](outputs/what-if-fleet.md) | +| `futureRetention` configured on VM | ≥ 2 d (chart default) | [promforecast-stack/values.yaml](../charts/promforecast-stack/values.yaml) | +| Degraded-mode alerts wired | `ForecastDatasourceUnreachable`, `ForecastSinkWriteFailing`, `ForecastSnapshotStale` mapped to on-call | [degraded-modes.md](operations/degraded-modes.md) | +| Auto-tuner installed (optional) | Weekly CronJob from `promforecast-tuner` sub-chart | [auto-tuning.md](operations/auto-tuning.md) | + +## What you can defer + +- HA mode (single-replica with `Recreate` is valid for small installs). +- Auto-tuner. +- Backfill (SLO dashboards become useful after a few refresh cycles). + +## What you cannot defer + +- Sink + scrape-drop combination (prevents dual-source state). +- `safety.max_total_series` cap (re-evaluate after enabling new families). +- Degraded-mode alerts (prevents silent stale snapshots). + +## Cross-references + +- [Multi-cluster architectures](architecture/multi-cluster.md) +- [Degraded-mode playbook](operations/degraded-modes.md) +- [Cardinality budgeting](operations/cardinality.md) +- [Migration to sink-first](how-to/migrate-to-sink-first.md) diff --git a/docs/reference/cli.md b/docs/reference/cli.md index ad09051..650af88 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -1,7 +1,8 @@ # CLI reference -The `promforecast` binary provides three subcommands: `run`, `validate`, -and `diagnose` (with `duplicates` and `sample-rate` sub-subcommands). +The `promforecast` binary provides seven subcommands: `run`, +`validate`, `adapter`, `diagnose` (with `duplicates` and `sample-rate` +sub-subcommands), `inspect`, `warmup`, and `backfill`. Running `promforecast --config foo.yaml` is identical to `promforecast run --config foo.yaml`, so existing container entrypoints continue to work unchanged. ## `promforecast run` @@ -125,3 +126,60 @@ alone would. See [`../operations/diagnose.md`](../operations/diagnose.md) for the full reference, authentication recipes, in-cluster invocation (`kubectl exec`), and CI-integration patterns. + +## `promforecast inspect` + +```bash +promforecast inspect \ + --config --query \ + [--series-filter '{label="value",...}'] \ + [--horizon ] [--lookback ] [--model ] \ + [--backtest] [--json] +``` + +Debugs a single weird forecast: fetches the configured series from +the TSDB, runs the configured fit pipeline exactly as the live +forecaster would, and prints the inputs, the fit, the forecast curve, +and (with `--backtest`) the per-fold backtest predictions vs. +actuals. See [`../operations/inspect.md`](../operations/inspect.md) +for debugging walkthroughs. + +Exit codes: `0` on success, `1` for inspector-level failures (unknown +query, ambiguous series, fit error), `2` for datasource / network +failures. + +## `promforecast warmup` + +```bash +promforecast warmup \ + [--url http://forecaster:9091] [--timeout 30] [--json] +``` + +Calls a running forecaster's `GET /forecast/warmup` endpoint and +prints per-(group, query) "what is still loading?" status. The +endpoint must be enabled via `server.expose_warmup_endpoint: true`; +see [`../operations/warmup.md`](../operations/warmup.md). + +Exit codes: `0` when every query reports `status=ready`, `1` when at +least one is still warming or errored, `2` for network failures +(suitable for a `until` loop in a pre-flight script). + +## `promforecast backfill` + +```bash +promforecast backfill \ + --config --from \ + [--to now|] [--groups GROUP1 GROUP2 ...] [--sink-url ] +``` + +Replays the fit pipeline at simulated past timestamps and pushes the +resulting forecast curves + accuracy summaries to the configured +`sink.remote_write` URL with explicit historical `timestamp_ms` +values. Use right after `helm install` so SLO / quality dashboards +populate from day one. The window is capped by +`safety.backfill_max_window` (default `90d`); see +[`../operations/backfill.md`](../operations/backfill.md) for the +resource-sizing trade-off. + +Exit codes: `0` on success, `1` for invalid input (over-budget +window, no configured sink), `2` for TSDB / network failures. diff --git a/examples/alerts/promforecast-rules.yaml b/examples/alerts/promforecast-rules.yaml index 8139eff..e8cf022 100644 --- a/examples/alerts/promforecast-rules.yaml +++ b/examples/alerts/promforecast-rules.yaml @@ -34,6 +34,33 @@ spec: {{ $labels.group }} in over 2 hours. Check pod logs and datasource availability. + # Per-series complement to ForecastStale above. The group-level + # gauge keeps ticking as long as *any* series of the group is + # fit successfully, so a single series whose model crashes + # silently is invisible there. ``forecast_data_staleness_seconds`` + # tracks the per-(series, model) clock so the silent failure + # surfaces — alert when staleness exceeds four refresh cycles + # (≈ four hours at the default 1h refresh). + - alert: ForecastSeriesStale + expr: forecast_data_staleness_seconds > 14400 + for: 15m + labels: + severity: warning + annotations: + summary: >- + Forecast {{ $labels.id }} model {{ $labels.model }} on + {{ $labels.instance }} hasn't been refit for >4h + description: | + The (series, model) tuple for {{ $labels.id }} on group + {{ $labels.group }} hasn't produced a successful forecast + for over four hours. The group-level ForecastStale alert + stays quiet because other series in the group are still + being fit successfully; this rule catches the silent + per-series failure (model crash, persistent fit timeout, + data gap that drops the series under min_points). Check + the forecaster logs filtered by group + query, then by + the source-side labels carried on the staleness gauge. + # Backtest accuracy has degraded — model needs retraining or # replacement. Threshold is intentionally loose; tune for the SLOs # of the metrics being forecasted. @@ -78,7 +105,7 @@ spec: - alert: ForecastFallbackPersistent expr: | count by (group) ( - forecast_quality_score{model_fallback="true", cold_start!="true"} + forecast_quality_score{model_fallback="true", cold_start!="true", horizon=""} ) > 0 for: 6h labels: @@ -98,7 +125,7 @@ spec: # noisily trip this alert until the underlying series matures. - alert: ForecastQualityLow expr: | - avg by (group) (forecast_quality_score{cold_start!="true"}) < 0.4 + avg by (group) (forecast_quality_score{cold_start!="true", horizon=""}) < 0.4 for: 6h labels: severity: info @@ -116,7 +143,7 @@ spec: # Excludes cold-start: those scores can land at a configured # ``max_quality_score`` and are not a model-degradation signal. - alert: ForecastQualityZero - expr: forecast_quality_score{cold_start!="true"} == 0 + expr: forecast_quality_score{cold_start!="true", horizon=""} == 0 for: 1h labels: severity: info @@ -168,7 +195,7 @@ spec: - alert: ForecastDeviation expr: | forecast_deviation_outside_band{level="95"} == 1 - and on (id, group, model) forecast_quality_score > 0.6 + and on (id, group, model) forecast_quality_score{horizon="1h"} > 0.6 for: 15m labels: severity: warning @@ -188,7 +215,7 @@ spec: - alert: ForecastDeviationSevere expr: | abs(forecast_deviation_ratio{level="95"}) > 2 - and on (id, group, model) forecast_quality_score > 0.6 + and on (id, group, model) forecast_quality_score{horizon="1h"} > 0.6 for: 5m labels: severity: critical @@ -223,6 +250,30 @@ spec: {{ $labels.instance }} ({{ $labels.mountpoint }}) is below zero. Provision capacity or clean up. + # Long-horizon capacity warning gated on the 24h quality slice. + # Per-horizon quality scores let capacity-style alerts gate on + # the long-horizon reliability rather than an aggregated score + # that's dominated by the (typically more reliable) short + # horizons. A 24h forecast scoring above 0.5 has enough + # backtested accuracy + tight bands at *that* horizon to be + # alertable; the equivalent ``horizon=""`` filter would let + # noisy long-horizon forecasts trigger off the back of a clean + # short-horizon average. + - alert: ForecastCapacityWarning + expr: | + node_filesystem_avail_bytes_forecast < 0 + and on (id, group, instance) forecast_quality_score{horizon="24h"} > 0.5 + for: 30m + labels: + severity: warning + annotations: + summary: "{{ $labels.instance }}:{{ $labels.mountpoint }} forecast to fill within 24h (reliable)" + description: | + The 24h forecast for {{ $labels.instance }} + ({{ $labels.mountpoint }}) projects exhausted free space, + and the 24h quality score is above 0.5 — the forecast is + reliable enough to act on. Provision capacity or clean up. + # Alert on the time-to-threshold ETA family directly. # Read against the conservative band (``level="95"``) so the # alert fires for "will hit threshold within N hours at 95% @@ -447,3 +498,238 @@ spec: outside its forecast. Roll back unless the deploy is expected to shift the metric (in which case retrain the model with the new regime). + + # ---------------------------------------------------------------- + # Forecast stability + cross-model agreement alerts. + # ---------------------------------------------------------------- + # ``forecast_revision_magnitude`` reports how much a model's + # prediction for a given horizon moved between successive fits. + # ``forecast_model_disagreement`` reports the coefficient of + # variation across configured models' central forecasts at the + # same horizon (explicit mode only — auto-select suppresses it). + - name: promforecast.stability.examples + interval: 1m + rules: + - alert: ForecastUnstable + expr: forecast_revision_magnitude > 0.3 + for: 20m + labels: + severity: info + annotations: + summary: "Forecast for {{ $labels.id }} ({{ $labels.model }}) is thrashing" + description: | + The {{ $labels.horizon }} forecast for {{ $labels.id }} + ({{ $labels.model }}) has been moving more than 30% relative + to the prior fit for 20m. Possible causes: a regime change + the model is over-reacting to, an unstable input regressor, + or a season-length misconfiguration. + + - alert: ForecastModelsDisagree + expr: forecast_model_disagreement > 0.4 + for: 30m + labels: + severity: info + annotations: + summary: "Configured models disagree on {{ $labels.id }} at {{ $labels.horizon }}" + description: | + The coefficient of variation across configured models' + central forecasts for {{ $labels.id }} at horizon + {{ $labels.horizon }} has been above 0.4 for 30m. No + single model is confident about this future — consider + switching the query to ``auto_select`` or changing the + configured model set. + + # ---------------------------------------------------------------- + # Forecaster self-observability alerts. + # ---------------------------------------------------------------- + # ``forecast_model_fit_success_ratio`` rolls per-(group, model) + # fit success/failure over 1h and 24h windows. + # ``forecast_regressor_stability_score`` flags regressors whose + # recent distribution drifted from their lookback distribution. + - name: promforecast.self_observability.examples + interval: 5m + rules: + - alert: ForecastModelFitFailing + expr: forecast_model_fit_success_ratio{window="1h"} < 0.9 + for: 15m + labels: + severity: warning + annotations: + summary: "Model {{ $labels.model }} fit failing in group {{ $labels.group }}" + description: | + Less than 90% of {{ $labels.model }} fit attempts in + {{ $labels.group }} succeeded over the last hour. Check + ``forecast_failures_total`` for the underlying reason + (fit_error / fit_timeout / query_error). + + - alert: ForecastRegressorUnstable + expr: forecast_regressor_stability_score < 0.3 + for: 1h + labels: + severity: info + annotations: + summary: "Regressor {{ $labels.regressor_id }} for {{ $labels.query }} is unstable" + description: | + The recent value distribution of regressor + {{ $labels.regressor_id }} (query {{ $labels.query }}, + group {{ $labels.group }}) has drifted significantly + from its lookback distribution. The forecast may + react to changing regressor signal in unexpected ways + until the regressor stabilises or the model re-fits. + + # ---------------------------------------------------------------- + # Degraded-mode alerts. + # ---------------------------------------------------------------- + # The forecaster isolates failures per (query, series, group) and + # papers over many of them with retries, so a failing datasource + # or a failing sink can stay invisible if you only watch the + # per-fit failure counter. These rules surface the three + # operational degraded states the playbook + # (``docs/operations/degraded-modes.md``) calls out as the ones + # worth paging on: PromQL reads against the long-term TSDB are + # failing, sink pushes are failing, or the per-group snapshot + # has aged past the freshness threshold operators care about. + - name: promforecast.degraded.examples + interval: 1m + rules: + # PromQL reads against the long-term TSDB are failing across + # multiple groups. ``forecast_source_retries_total`` ticks on + # transient transport errors and 5xx responses; a sustained + # rate above 0 is the early signal that the datasource is + # not reliably reachable. Pair with + # ``forecast_failures_total{reason="query_error"}`` if you + # want a stronger terminal-failure signal. + - alert: ForecastDatasourceUnreachable + expr: | + rate(forecast_source_retries_total[5m]) > 0.1 + or + rate(forecast_failures_total{reason=~"query_error|query_timeout"}[5m]) > 0.1 + for: 10m + labels: + severity: warning + annotations: + summary: "promforecast datasource is failing reads ({{ $labels.reason }})" + description: | + The forecaster is retrying or failing PromQL fetches + against the configured ``datasource.url`` at >0.1/s for + 10m. Existing snapshot data continues to serve via + ``/metrics`` but new fits cannot proceed — check + VictoriaMetrics (or the configured TSDB) reachability, + authentication, and network policies. See + ``docs/operations/degraded-modes.md`` for the recovery + path. + + # Sink (``remote_write``) writes are failing. Distinct from + # fit-side failures — a failing sink is invisible to + # ``forecast_failures_total`` because the runner records the + # failure on the dedicated ``forecast_sink_write_failures_total`` + # counter and continues. Forecasts still appear on ``/metrics`` + # via the snapshot path; only the long-term TSDB stops + # accruing the smooth curve. + - alert: ForecastSinkWriteFailing + expr: rate(forecast_sink_write_failures_total[5m]) > 0 + for: 15m + labels: + severity: warning + annotations: + summary: "promforecast sink writes are failing ({{ $labels.reason }})" + description: | + The forecaster has been failing to push forecast curves + to the configured ``sink.remote_write`` target for 15m + (reason {{ $labels.reason }}). Snapshots remain on + ``/metrics`` but the long-term TSDB stops accumulating + the future-dated forecast curve. Check the sink target's + reachability, the configured ``futureRetention`` + (VictoriaMetrics rejects past it), and the bearer/auth + headers. See ``docs/operations/degraded-modes.md``. + + # The per-group snapshot is stale. ``forecast_snapshot_age_seconds`` + # is the server-side complement to + # ``time() - forecast_last_run_timestamp_seconds`` — the same + # signal but pre-subtracted so the alert reads as a direct + # gauge comparison. Tune the threshold to roughly 4x the + # group's ``refresh_interval``; the default 2h matches a 30m + # refresh. + - alert: ForecastSnapshotStale + expr: forecast_snapshot_age_seconds > 7200 + for: 10m + labels: + severity: warning + annotations: + summary: "Snapshot for {{ $labels.group }} hasn't refreshed in over 2h" + description: | + The most recent successful snapshot for group + {{ $labels.group }} is older than 2h. Distinct from + ``ForecastStale``: this alert reads the pre-subtracted + ``forecast_snapshot_age_seconds`` gauge directly, which + makes the alert and dashboard expressions identical. + Check pod health, datasource reachability + (``ForecastDatasourceUnreachable``), and sink health + (``ForecastSinkWriteFailing``) for the underlying + cause. See ``docs/operations/degraded-modes.md``. + + # ---------------------------------------------------------------- + # Governance + drift-explainer alerts. + # ---------------------------------------------------------------- + # ``forecast_service_health_score`` and + # ``forecast_drift_explained_total`` are emission families the + # forecaster produces when the operator opts into the + # ``governance.health_score`` block and the drift-explainer + # (always on when ``emission.drift.enabled: true``) respectively. + # The rules below ship as templates — adjust the bucket labels + # (``service``, ``environment``) and thresholds to match your + # fleet. + - name: promforecast.governance.examples + interval: 1m + rules: + # Service-level forecast health is below the executive- + # dashboard floor. Use this as a quarterly review trigger, + # not a pager — a low aggregate score across a whole service + # bucket is rarely an immediate incident, but it points at + # systemic forecast-quality issues worth investigating before + # they degrade the alerts gated on + # ``forecast_quality_score``. + - alert: ForecastServiceHealthLow + expr: forecast_service_health_score < 0.6 + for: 24h + labels: + severity: info + annotations: + summary: "Forecast health for {{ $labels.service }} below 0.6" + description: | + Aggregate forecast quality across the + {{ $labels.service }} bucket has been below 0.6 for + 24h ({{ $labels.series_count }} series in the bucket). + Pages of derived alerts (deviation, capacity ETA) that + gate on ``forecast_quality_score`` for the same + services are silently rejecting their forecasts. Check + ``forecast_accuracy_mape`` and + ``forecast_band_empirical_coverage`` to identify which + dimension dropped. See + ``docs/governance/forecast-health.md``. + + # A single regressor keeps explaining drift events for the + # same query — strong hint that the regressor is the actual + # driver of the forecast moving. Useful for spotting + # regressors that need tuning (e.g. a noisy ``deploys`` + # counter), or signals worth promoting to a dedicated + # alert. + - alert: DriftExplainedRepeats + expr: | + sum by (group, query, primary_suspect) ( + increase(forecast_drift_explained_total[1h]) + ) >= 3 + for: 15m + labels: + severity: info + annotations: + summary: "{{ $labels.primary_suspect }} keeps explaining drift on {{ $labels.query }}" + description: | + ``forecast_drift_explained_total`` shows + {{ $labels.primary_suspect }} as the primary suspect + for >=3 drift events on (group={{ $labels.group }}, + query={{ $labels.query }}) in the last hour. Either + the regressor is genuinely driving the forecast — in + which case investigate the upstream signal — or the + correlation window is too short and the suspect is + noise. See ``docs/operations/drift-explained.md``. diff --git a/examples/configs/cookbook/aws-cloudwatch-estimated-charges-usd.yaml b/examples/configs/cookbook/aws-cloudwatch-estimated-charges-usd.yaml new file mode 100644 index 0000000..b5105b6 --- /dev/null +++ b/examples/configs/cookbook/aws-cloudwatch-estimated-charges-usd.yaml @@ -0,0 +1,49 @@ +# Cookbook: aws_cloudwatch_estimated_charges_usd +# +# Forecast the AWS monthly bill so finance + engineering get a +# month-end-projection trend before the invoice arrives. Use the +# damped-trend model — daily billing snapshots otherwise extrapolate +# linearly and overstate the projection. + +apiVersion: promforecast.io/v1 + +datasource: + url: http://victoriametrics:8428/ + +server: + refresh_interval: 1h + +defaults: + # 6h step is the CloudWatch billing publish cadence; longer step + # = fewer rows fetched and lighter on the TSDB. 90d gives the + # model enough seasonal context to spot month-over-month patterns. + lookback: 90d + step: 6h + horizon: 30d + models: + - name: AutoETSDamped + params: + damping_parameter: 0.95 + - SeasonalNaive + # 4 samples per day at 6h step → weekly seasonality is 28. + season_length: 28 + confidence_levels: [95] + accuracy: + evaluate: true + auto_select: true + +groups: + - name: cloud_cost + queries: + - id: aws_estimated_charges_usd + promql: | + aws_cloudwatch_estimated_charges_usd{service_name="Total"} + thresholds: + - name: monthly_budget + value: 50000 + comparator: gt + alert_within: 14d + severity: warning + action_cost: + type: usd + value: 50000 diff --git a/examples/configs/cookbook/http-requests-total.yaml b/examples/configs/cookbook/http-requests-total.yaml new file mode 100644 index 0000000..973ead4 --- /dev/null +++ b/examples/configs/cookbook/http-requests-total.yaml @@ -0,0 +1,50 @@ +# Cookbook: http_requests_total +# +# Forecast request rate per service/endpoint so capacity planning and +# SLO error-budget projections work from one source of truth. Couple +# the rate forecast with an SLO block to surface error-budget +# exhaustion forecasts. + +apiVersion: promforecast.io/v1 + +datasource: + url: http://victoriametrics:8428/ + +server: + refresh_interval: 5m + +defaults: + lookback: 14d + step: 1m + horizon: 6h + models: + - AutoARIMA + - SeasonalNaive + season_length: 1440 # daily seasonality at 1m step + confidence_levels: [80, 95] + accuracy: + evaluate: true + auto_select: true + horizons: [15m, 1h, 6h] + emission: + deviation: true + quality: true + +groups: + - name: http_traffic + queries: + - id: http_requests_rate + promql: | + sum by (service) (rate(http_requests_total[5m])) + # Predictive will-breach gauge: alert before capacity bites. + thresholds: + - name: capacity_soft + value: 5000 + comparator: gt + alert_within: 30m + severity: warning + emission: + growth_rate: + enabled: true + windows: [1h, 1d] + units: percent diff --git a/examples/configs/cookbook/jvm-gc-pause-seconds.yaml b/examples/configs/cookbook/jvm-gc-pause-seconds.yaml new file mode 100644 index 0000000..32c57bc --- /dev/null +++ b/examples/configs/cookbook/jvm-gc-pause-seconds.yaml @@ -0,0 +1,46 @@ +# Cookbook: jvm_gc_pause_seconds +# +# Long GC pauses correlate with user-facing latency; forecasting the +# rate of long pauses lets you proactively recycle pods before they +# breach an SLO. Use the deviation gauge to alert on "unusual GC +# activity right now" rather than absolute thresholds tuned per +# environment. + +apiVersion: promforecast.io/v1 + +datasource: + url: http://victoriametrics:8428/ + +server: + refresh_interval: 5m + +defaults: + lookback: 7d + step: 1m + horizon: 1h + models: + - AutoARIMA + - SeasonalNaive + season_length: 1440 + confidence_levels: [80, 95] + accuracy: + evaluate: true + auto_select: true + emission: + deviation: true + quality: true + +groups: + - name: jvm + queries: + - id: jvm_gc_long_pause_rate + promql: | + sum by (instance, pool) ( + rate(jvm_gc_pause_seconds_count{action="end of major GC"}[5m]) + ) + # No hard threshold — operators alert on the deviation gauge + # crossing the band rather than a magic GC pause threshold. + emission: + drift: + enabled: true + alert_threshold: 0.3 diff --git a/examples/configs/cookbook/kafka-consumer-lag.yaml b/examples/configs/cookbook/kafka-consumer-lag.yaml new file mode 100644 index 0000000..e0577d0 --- /dev/null +++ b/examples/configs/cookbook/kafka-consumer-lag.yaml @@ -0,0 +1,45 @@ +# Cookbook: kafka_consumer_lag +# +# Two-sided question: "when will lag exhaust the broker's retention?" +# (capacity) and "when will it drain?" (operations). The drain ETA +# fires +Inf when the lag isn't actually draining, which Alertmanager +# can pin as a distinct alert from "draining slowly". + +apiVersion: promforecast.io/v1 + +datasource: + url: http://victoriametrics:8428/ + +server: + refresh_interval: 1m + +defaults: + lookback: 7d + step: 1m + horizon: 4h + models: + - AutoARIMA + - SeasonalNaive + season_length: 1440 + confidence_levels: [95] + accuracy: + evaluate: true + auto_select: true + +groups: + - name: kafka + queries: + - id: kafka_consumer_lag + promql: | + sum by (topic, consumergroup) ( + kafka_consumergroup_lag + ) + thresholds: + - name: backlog_warn + value: 100000 + comparator: gt + alert_within: 1h + severity: warning + - name: drained + value: 0 + comparator: le_drain diff --git a/examples/configs/cookbook/kube-deployment-status-replicas.yaml b/examples/configs/cookbook/kube-deployment-status-replicas.yaml new file mode 100644 index 0000000..9e4e734 --- /dev/null +++ b/examples/configs/cookbook/kube-deployment-status-replicas.yaml @@ -0,0 +1,39 @@ +# Cookbook: kube_deployment_status_replicas +# +# Forecast deployment replica count so capacity planners see ahead +# of a sustained autoscale ramp instead of reacting to alerts. +# Predominantly piecewise-stable signals — SeasonalNaive performs +# well; ARIMA captures the autoscaler-driven ramp. + +apiVersion: promforecast.io/v1 + +datasource: + url: http://victoriametrics:8428/ + +server: + refresh_interval: 15m + +defaults: + lookback: 30d + step: 5m + horizon: 24h + # Counts-and-stairs signals — keep models cheap, prefer count-aware + # safeguards so the predicted replica count rounds to an integer. + models: + - SeasonalNaive + - AutoARIMA + season_length: 288 + confidence_levels: [95] + accuracy: + evaluate: true + auto_select: true + +groups: + - name: kube_workloads + queries: + - id: kube_deployment_status_replicas + promql: | + sum by (namespace, deployment) ( + kube_deployment_status_replicas + ) + data_profile: count diff --git a/examples/configs/cookbook/node-filesystem-avail-bytes.yaml b/examples/configs/cookbook/node-filesystem-avail-bytes.yaml new file mode 100644 index 0000000..6f016fc --- /dev/null +++ b/examples/configs/cookbook/node-filesystem-avail-bytes.yaml @@ -0,0 +1,51 @@ +# Cookbook: node_filesystem_avail_bytes +# +# Predict when each mount will run out of space and alert before the +# pager fires. Pair with a damped-trend model so a brief growth spike +# doesn't extrapolate into a doomsday forecast. + +apiVersion: promforecast.io/v1 + +datasource: + url: http://victoriametrics:8428/ + +server: + refresh_interval: 1h + +defaults: + lookback: 30d + step: 5m + horizon: 7d + models: + - name: AutoETSDamped + params: + damping_parameter: 0.98 + - SeasonalNaive + season_length: 288 + confidence_levels: [95] + accuracy: + evaluate: true + auto_select: true + fallback_model: SeasonalNaive + fallback_mape_threshold: 50 + +groups: + - name: disk_capacity + queries: + - id: node_filesystem_avail_bytes + promql: | + node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} + thresholds: + - name: critical + value: 5368709120 # 5 GiB + comparator: lt + alert_within: 24h + severity: critical + action_cost: + type: usd + value: 500 # cost of an emergency cleanup window + emission: + growth_rate: + enabled: true + windows: [1d, 7d] + units: units_per_second diff --git a/examples/configs/cookbook/redis-connected-clients.yaml b/examples/configs/cookbook/redis-connected-clients.yaml new file mode 100644 index 0000000..bf5a053 --- /dev/null +++ b/examples/configs/cookbook/redis-connected-clients.yaml @@ -0,0 +1,45 @@ +# Cookbook: redis_connected_clients +# +# Forecast the connection pool's exhaustion ahead of incidents. A +# Redis tier that's been quietly accumulating connections eventually +# hits maxclients and breaks every consumer at once; the time-to- +# threshold ETA catches that hours before the symptoms. + +apiVersion: promforecast.io/v1 + +datasource: + url: http://victoriametrics:8428/ + +server: + refresh_interval: 5m + +defaults: + lookback: 7d + step: 1m + horizon: 24h + models: + - name: AutoETSDamped + params: + damping_parameter: 0.98 + - SeasonalNaive + season_length: 1440 + confidence_levels: [95] + accuracy: + evaluate: true + auto_select: true + +groups: + - name: redis + queries: + - id: redis_connected_clients + promql: | + max by (instance) (redis_connected_clients) + thresholds: + - name: capacity + value: 9000 # tune against your maxclients + comparator: gt + alert_within: 4h + severity: critical + action_cost: + type: priority + value: 100 diff --git a/examples/k8s/backfill-job.yaml b/examples/k8s/backfill-job.yaml new file mode 100644 index 0000000..9918d82 --- /dev/null +++ b/examples/k8s/backfill-job.yaml @@ -0,0 +1,84 @@ +# promforecast historical backfill — one-shot Kubernetes Job +# +# Replays the configured fit pipeline at simulated past timestamps +# and pushes the resulting forecast curves + accuracy summaries to +# the configured remote_write sink with historical timestamps. Use +# right after `helm install` so SLO / quality dashboards built on +# `forecast_quality_score` / `forecast_accuracy_mape` produce +# meaningful aggregates from day one instead of starting from +# install time. +# +# The Job is intentionally a separate workload from the live +# Deployment — a long backfill walks many fits and would contend +# with the scheduled fit budget if it shared a process. The cap +# `safety.backfill_max_window` rejects accidental year-long +# requests at the CLI boundary. +# +# Tune `--from` to match the longest window your SLO / quality +# dashboards aggregate over (typically 30d for SLO error budgets, +# 7d for quality-score trends). See `docs/operations/backfill.md` +# for the per-fit-cost estimate. +apiVersion: batch/v1 +kind: Job +metadata: + name: promforecast-backfill + labels: + app.kubernetes.io/name: promforecast + app.kubernetes.io/component: backfill +spec: + # Single-shot — don't retry on container exit. A backfill that + # crashed halfway through is best re-run with a narrower window + # rather than re-doing the work end-to-end (the sink dedupes by + # (labels, timestamp) so partial re-runs are safe but wasteful). + backoffLimit: 0 + # Bound the worst case at one hour. A 30d backfill at the default + # 1h refresh interval is 720 fits per group; on a single core + # this should comfortably finish well inside the cap. Raise for + # high-cardinality configs after running `promforecast validate` + # to size the work. + activeDeadlineSeconds: 3600 + template: + spec: + restartPolicy: Never + containers: + - name: backfill + # Pin to the same image as the live Deployment so the + # backfill uses the exact model versions that production + # will see. + image: ghcr.io/esops-dev/promforecast:latest + imagePullPolicy: IfNotPresent + args: + - backfill + - --config=/etc/promforecast/config.yaml + - --from=30d + # --to defaults to "now"; pin to an explicit ISO-8601 + # timestamp if you want reproducibility across re-runs. + # - --to=2026-05-24T00:00:00Z + # --groups defaults to every configured group; narrow + # when iterating on a single dashboard. + # - --groups + # - node_capacity + volumeMounts: + - name: config + mountPath: /etc/promforecast + readOnly: true + resources: + # Backfill is CPU-bound (statsforecast fits); memory + # tracks the largest lookback frame. The defaults here + # match the live Deployment's; bump CPU when the walk + # is dominated by AutoARIMA fits on many series. + requests: + cpu: "500m" + memory: "512Mi" + limits: + cpu: "2000m" + memory: "2Gi" + volumes: + - name: config + configMap: + # Reuse the live forecaster's ConfigMap so backfill + # always uses the deployed config. If the ConfigMap + # ships under a different name in your install (the + # standalone chart names it after the release), update + # this `name`. + name: promforecast diff --git a/forecaster/src/promforecast/backfill.py b/forecaster/src/promforecast/backfill.py new file mode 100644 index 0000000..f20e07d --- /dev/null +++ b/forecaster/src/promforecast/backfill.py @@ -0,0 +1,671 @@ +"""Historical SLO-window backfill. + +The runner only emits metrics from "now" forward — a fresh install +gets an empty :metric:`forecast_quality_score`, empty +:metric:`forecast_accuracy_mape`, empty band-coverage gauges, and +so on, until enough refresh cycles have accumulated for an SLO +window to be meaningful. This module replays the fit pipeline at +simulated past timestamps and pushes the resulting forecast curves +(plus per-fit accuracy / quality summaries) to the configured +``sink.remote_write`` URL with explicit historical +``timestamp_ms`` values so VM / Mimir / Thanos store them as if +they had been emitted at that time. + +Runs as a one-shot pod separate from the live forecaster (see +``examples/k8s/backfill-job.yaml``) so a long backfill does not +contend with the scheduled fit budget. The live forecaster's +snapshot is never touched — backfill is a pure write-side +operation against the long-term TSDB. + +**Scope.** This module backfills the operator-visible families +that have a natural "as-of-time" interpretation: + +* the forecast curve (yhat + per-level bands), one per + (series, model) per emitted horizon step; +* the per-(series, model) backtest accuracy summary + (:metric:`forecast_accuracy_mape` / + :metric:`forecast_accuracy_mase`); and +* the composite :metric:`forecast_quality_score` for each + (series, model) emission. + +Snapshot-only operational metrics +(:metric:`forecast_failures_total`, +:metric:`forecast_run_duration_seconds`, the deploy-risk score) +are deliberately *not* backfilled: they describe live execution +state, not the underlying signal. An operator who wants them as +"historical zero" can rely on Prometheus's normal absent-series +treatment. The conditional / decomposition / drift / band-coverage +/ deviation families are also skipped — they need cross-fit state +(drift cache, conformal holdout, rolling band-coverage windows) +that the one-shot walk doesn't maintain; backfilling them would +either be wrong (rolling windows distorted by single synthetic +observations) or require running the live runner's full cycle +state machinery from a one-shot script. See +``docs/operations/backfill.md`` for the full omission list. + +Regressors are **not** fetched during backfill — the per-fit +pipeline always passes ``regressors=None`` so the historical +curves reflect the model's base seasonality without the deploy / +calendar exogenous shifts the live runner uses. The trade-off is +documented in ``docs/operations/backfill.md``. +""" + +from __future__ import annotations + +import asyncio +import math +import sys +import time +from collections.abc import Iterable +from dataclasses import dataclass +from datetime import UTC, datetime, timedelta +from pathlib import Path +from typing import IO, TYPE_CHECKING, Any + +import structlog + +from . import accuracy as accuracy_mod +from . import config as config_module +from . import models as model_registry +from . import preprocess as preprocess_mod +from . import quality as quality_mod +from .accuracy import overall_mape, overall_mase +from .config import ( + Config, + GroupConfig, + QueryConfig, + effective_emission_safeguards, + effective_refresh_interval, + effective_season_length, + effective_step, +) +from .point_factory import SeriesPointFactory +from .preview import _pandas_freq +from .runner import _apply_overflow, _apply_safeguards_to_curve_points +from .sink import ForecastCurvePoint, ForecastSink, RemoteWriteSink, SinkError +from .source import PromSource + +if TYPE_CHECKING: + import pandas as pd + +logger = structlog.get_logger(__name__) + + +class BackfillError(RuntimeError): + """Raised for any operator-visible backfill failure. + + Carries an exit code so ``main.cli`` can hand it straight to + ``sys.exit``. + """ + + def __init__(self, message: str, *, exit_code: int = 1) -> None: + super().__init__(message) + self.exit_code = exit_code + + +@dataclass(frozen=True) +class BackfillStats: + """Per-run summary returned for logs + the CLI footer.""" + + timestamps_walked: int + fits_issued: int + points_pushed: int + failed_fits: int + failed_pushes: int + elapsed_seconds: float + + +def run_backfill( # noqa: PLR0911 + *, + config_path: Path, + from_duration: str, + to_timestamp: str | None, + group_selector: list[str] | None, + sink_url_override: str | None, + out: IO[str] | None = None, +) -> int: + """Top-level entry point used by ``main.cli``. + + Returns ``0`` on success, ``1`` for invalid inputs (config issues, + over-budget window, no configured sink), ``2`` for TSDB / network + failures encountered during the walk. + """ + stream: IO[str] = out if out is not None else sys.stdout + + try: + cfg = config_module.load(config_path) + except FileNotFoundError: + sys.stderr.write(f"error: config not found: {config_path}\n") + return 2 + except Exception as exc: + sys.stderr.write(f"error: config validation failed: {exc}\n") + return 1 + + try: + from_seconds = _parse_duration_seconds(from_duration, flag="--from") + to_time = _parse_to_timestamp(to_timestamp) + _check_window(cfg, from_seconds) + except BackfillError as exc: + sys.stderr.write(f"error: {exc}\n") + return exc.exit_code + + sink_url = sink_url_override or cfg.sink.remote_write.url + if not sink_url: + sys.stderr.write( + "error: backfill requires a remote_write sink — either configure " + "sink.remote_write.{enabled: true, url: ...} or pass --sink-url.\n" + ) + return 1 + + selected_groups = _select_groups(cfg, group_selector) + if not selected_groups: + sys.stderr.write("error: --groups selector matched no configured groups\n") + return 1 + + sink: ForecastSink = RemoteWriteSink( + url=sink_url, + timeout_seconds=cfg.sink.remote_write.timeout.total_seconds(), + headers=dict(cfg.sink.remote_write.headers), + chunk_size=cfg.sink.remote_write.chunk_size, + max_connections=cfg.sink.remote_write.max_connections, + max_keepalive_connections=cfg.sink.remote_write.max_keepalive_connections, + ) + source = PromSource( + url=cfg.datasource.url, + timeout_seconds=cfg.datasource.timeout.total_seconds(), + max_concurrency=cfg.datasource.max_concurrent_requests, + max_keepalive_connections=cfg.datasource.max_keepalive_connections, + ) + + try: + stats = asyncio.run( + _walk_backfill( + config=cfg, + source=source, + sink=sink, + from_seconds=from_seconds, + to_time=to_time, + groups=selected_groups, + out=stream, + ) + ) + except BackfillError as exc: + sys.stderr.write(f"error: {exc}\n") + return exc.exit_code + except Exception as exc: + sys.stderr.write(f"error: unexpected backfill failure: {exc}\n") + return 2 + + stream.write( + f"backfill complete: timestamps={stats.timestamps_walked} " + f"fits={stats.fits_issued} points={stats.points_pushed} " + f"failed_fits={stats.failed_fits} failed_pushes={stats.failed_pushes} " + f"elapsed_s={stats.elapsed_seconds:.1f}\n" + ) + return 0 + + +async def _walk_backfill( + *, + config: Config, + source: PromSource, + sink: ForecastSink, + from_seconds: float, + to_time: datetime, + groups: list[GroupConfig], + out: IO[str], +) -> BackfillStats: + """Walk back-in-time, run fits, push to sink, return stats.""" + started = time.monotonic() + timestamps_walked = 0 + fits_issued = 0 + failed_fits = 0 + failed_pushes = 0 + points_pushed = 0 + + try: + for group in groups: + timestamps = list(_iter_backfill_timestamps(config, group, from_seconds, to_time)) + group_fits = 0 + for ts in timestamps: + timestamps_walked += 1 + buffer: list[ForecastCurvePoint] = [] + for query in group.queries: + try: + produced = await _run_query_at( + config=config, + group=group, + query=query, + source=source, + as_of=ts, + ) + except BackfillError: + # An I/O-shaped failure (e.g. query timeout) carries + # its own exit code; surface it to the top-level + # walker so the caller distinguishes "TSDB + # unreachable" (exit 2) from "fit walked, some + # series failed" (exit 1). Catching it under the + # generic ``Exception`` arm below would silently + # downgrade timeouts into per-query failures. + raise + except Exception as exc: + logger.warning( + "backfill_query_failed", + group=group.name, + query=query.id, + ts=ts.isoformat(), + error=str(exc), + ) + failed_fits += 1 + continue + fits_issued += len(produced.fit_count) + group_fits += len(produced.fit_count) + buffer.extend(produced.curve) + if buffer: + try: + await sink.write(buffer) + points_pushed += len(buffer) + except SinkError as exc: + logger.warning( + "backfill_push_failed", + group=group.name, + ts=ts.isoformat(), + reason=exc.reason, + error=str(exc), + ) + failed_pushes += 1 + # Per-group ``forecast_backfill_fits_last`` audit row pushed + # at the *final* as-of timestamp for the group. Named with + # the ``_last`` (gauge) suffix rather than ``_total`` so + # PromQL ``rate()`` / ``increase()`` don't false-fire on + # a second backfill run that issues fewer fits than the + # first: backfill runs are independent one-shot operations + # with no monotonic-counter semantics, and the ``_total`` + # suffix would lie about that contract. The row's timestamp + # still lets dashboards anchor "backfill happened then". + if timestamps: + try: + await sink.write( + [ + ForecastCurvePoint( + metric="forecast_backfill_fits_last", + labels={"group": group.name}, + value=float(group_fits), + timestamp_ms=int(timestamps[-1].timestamp() * 1000), + ) + ] + ) + points_pushed += 1 + except SinkError as exc: + logger.warning( + "backfill_audit_push_failed", + group=group.name, + reason=exc.reason, + error=str(exc), + ) + failed_pushes += 1 + out.write(f"group {group.name}: timestamps={len(timestamps)} fits={group_fits}\n") + finally: + await source.aclose() + await sink.aclose() + + elapsed = time.monotonic() - started + return BackfillStats( + timestamps_walked=timestamps_walked, + fits_issued=fits_issued, + points_pushed=points_pushed, + failed_fits=failed_fits, + failed_pushes=failed_pushes, + elapsed_seconds=elapsed, + ) + + +@dataclass +class _QueryProduced: + """Per-query backfill output for one historical timestamp.""" + + curve: list[ForecastCurvePoint] + fit_count: list[str] # one entry per emitted (series, model) tuple + + +async def _run_query_at( + *, + config: Config, + group: GroupConfig, + query: QueryConfig, + source: PromSource, + as_of: datetime, +) -> _QueryProduced: + """Fit ``query`` as-of ``as_of`` and translate to sink curve points. + + Built around the same primitives the runner uses — preprocess + + statsforecast wrappers — but with a stripped-down per-fit + pipeline that skips the live exporter state, the conditional + refit, the cold-start fallback, and any feature that needs + cross-run state. The point of backfill is to populate the + long-term TSDB with curves an operator can query *now*; nuanced + features can be replayed in a follow-up after the v1 shape is + accepted. + + Per-(series, model) failures are isolated: a bad fit for one + model doesn't abort the rest of the query. + """ + import pandas as pd # noqa: PLC0415 + + d = config.defaults + eff_step = effective_step(query, d) + eff_season_length = effective_season_length(query, d, step=eff_step) + step_seconds = max(1, int(eff_step.total_seconds())) + horizon_steps = max(1, int(d.horizon.total_seconds() // step_seconds)) + levels = list(d.confidence_levels) + freq = _pandas_freq(step_seconds) + + start = as_of - d.lookback + try: + frames = await asyncio.wait_for( + source.query_range(query.promql, start, as_of, step_seconds), + timeout=config.safety.query_timeout.total_seconds(), + ) + except TimeoutError as exc: + raise BackfillError(f"query timed out: {query.id}", exit_code=2) from exc + + # Apply the per-query series cap up front so a high-cardinality + # PromQL during backfill doesn't blow the one-shot Job's memory. + # The runner does the same cap inside ``_run_query`` for live + # fits; without it here a query that legitimately returns 5000 + # series at some point in the past would push 5000 * N_models + # fits into a single backfill timestamp. + safety = config.safety + allowed, _dropped = _apply_overflow(frames, safety.max_series_per_query, safety.series_overflow) + + clip, rounder = effective_emission_safeguards(query) + produced = _QueryProduced(curve=[], fit_count=[]) + loop = asyncio.get_running_loop() + for frame in allowed: + try: + cleaned = preprocess_mod.preprocess_series( + timestamps=list(frame.timestamps), + values=list(frame.values), + step=eff_step, + config=group.preprocess, + ) + except Exception as exc: + # Mirror the live runner's per-series failure isolation + # (see ``_scheduler.py::_run_one_series``): a single bad + # series — e.g. one whose timestamps trip a preprocess + # invariant — must not abort the rest of the query's + # backfill walk. + logger.warning( + "backfill_series_preprocess_failed", + group=group.name, + query=query.id, + series_labels=frame.labels, + error=str(exc), + ) + continue + if cleaned.dropped or len(cleaned.values) < d.min_points: + continue + df = pd.DataFrame( + { + "unique_id": ["s"] * len(cleaned.values), + "ds": pd.to_datetime(cleaned.timestamps), + "y": cleaned.values, + } + ) + factory = SeriesPointFactory( + metric_id=query.id, + base_labels=frame.labels, + group=group.name, + levels=levels, + ) + for spec in d.models: + try: + model = model_registry.build( + spec.name, season_length=eff_season_length, params=spec.params + ) + forecast = await asyncio.wait_for( + loop.run_in_executor( + None, model.fit_predict, df, horizon_steps, freq, levels, None + ), + timeout=config.safety.fit_timeout.total_seconds(), + ) + except Exception as exc: + logger.warning( + "backfill_fit_failed", + group=group.name, + query=query.id, + model=spec.name, + error=str(exc), + ) + continue + produced.fit_count.append(spec.name) + curve_points = list(factory.forecast_curve_points(forecast, spec.name, selection=None)) + # Mirror the live runner's count-aware emission policy so a + # count-shaped query (``data_profile: count`` or explicit + # ``clip_non_negative`` / ``round_to_integer``) doesn't + # silently push negative or fractional values into the + # historical TSDB. The curve helper is non-mutating so the + # raw output is preserved upstream. + if clip or rounder: + curve_points = _apply_safeguards_to_curve_points( + curve_points, clip=clip, rounder=rounder + ) + produced.curve.extend( + _rebase_curve_to_as_of(curve_points, as_of=as_of, step_seconds=step_seconds) + ) + produced.curve.extend( + _accuracy_curve_points( + cleaned_df=df, + model=model, + model_name=spec.name, + freq=freq, + levels=levels, + horizons=accuracy_mod.horizons_from_durations( + list(d.accuracy.horizons), eff_step + ), + season_length=eff_season_length, + factory=factory, + as_of=as_of, + ) + ) + return produced + + +def _rebase_curve_to_as_of( + points: list[ForecastCurvePoint], + *, + as_of: datetime, + step_seconds: int, +) -> Iterable[ForecastCurvePoint]: + """Rewrite curve point timestamps so they anchor at ``as_of``. + + :func:`SeriesPointFactory.forecast_curve_points` stamps each + point with ``now + i * step``; for backfill we want + ``as_of + i * step`` so the historical curves line up with what + a scrape at ``as_of`` would have observed. Counts grow with + ``step_seconds`` so the time delta is computed once per curve + rather than per point. + """ + if not points: + return () + first_ts_ms = min(p.timestamp_ms for p in points) + target_first_ms = int(as_of.timestamp() * 1000) + shift_ms = target_first_ms - first_ts_ms + return [ + ForecastCurvePoint( + metric=p.metric, + labels=p.labels, + value=p.value, + timestamp_ms=p.timestamp_ms + shift_ms, + ) + for p in points + ] + + +def _accuracy_curve_points( + *, + cleaned_df: pd.DataFrame, + model: Any, + model_name: str, + freq: str, + levels: list[int], + horizons: list[accuracy_mod.HorizonSpec], + season_length: int, + factory: SeriesPointFactory, + as_of: datetime, +) -> Iterable[ForecastCurvePoint]: + """Backtest the model at ``as_of`` and emit accuracy + quality samples. + + Pushes per-horizon ``forecast_accuracy_mape`` / + ``forecast_accuracy_mase`` samples and a composite + ``forecast_quality_score`` sample, all stamped with ``as_of``. + Failures are swallowed — the curve is the load-bearing emission; + the operational summaries are a bonus. + """ + if not horizons: + return () + try: + outcome = accuracy_mod.backtest( + model=model, + df=cleaned_df, + horizon_steps=horizons[0].steps, + freq=freq, + levels=levels, + ) + except Exception: + return () + if outcome is None: + return () + test, predictions = outcome + results = accuracy_mod.evaluate( + test=test, + predictions=predictions, + train=cleaned_df.iloc[: -horizons[0].steps].reset_index(drop=True), + season_length=season_length, + horizons=horizons, + model_name=model_name, + ) + if not results: + return () + as_of_ms = int(as_of.timestamp() * 1000) + out: list[ForecastCurvePoint] = [] + for r in results: + common = factory.drift_labels(model_name) + if math.isfinite(r.mape): + out.append( + ForecastCurvePoint( + metric="forecast_accuracy_mape", + labels={**common, "horizon": r.horizon}, + value=r.mape, + timestamp_ms=as_of_ms, + ) + ) + if math.isfinite(r.mase): + out.append( + ForecastCurvePoint( + metric="forecast_accuracy_mase", + labels={**common, "horizon": r.horizon}, + value=r.mase, + timestamp_ms=as_of_ms, + ) + ) + + # Quality score: same machinery the live runner uses + # (:func:`promforecast.quality.compute`). We don't have a + # post-fit band-width measurement in the backfill path so the + # helper falls back to the standard sub-score for missing input; + # ``coverage`` is assumed 1.0 because the cleaned dataframe is + # the lookback the runner would have used. + mape_value = overall_mape(results) + mase_value = overall_mase(results) + score_kind: quality_mod.ScoreKind = "mape" if mape_value is not None else "mase" + score_value = mape_value if score_kind == "mape" else mase_value + quality = quality_mod.compute( + backtest_mape=score_value, + band_width_relative=None, + coverage=1.0, + imputed_fraction=0.0, + score_kind=score_kind, + ) + if math.isfinite(quality): + out.append( + ForecastCurvePoint( + metric="forecast_quality_score", + labels=factory.drift_labels(model_name), + value=quality, + timestamp_ms=as_of_ms, + ) + ) + return out + + +def _iter_backfill_timestamps( + config: Config, + group: GroupConfig, + from_seconds: float, + to_time: datetime, +) -> Iterable[datetime]: + """Yield each "as-of" timestamp the backfill should issue fits at. + + The cadence is the group's effective refresh interval — the + shortest interval across the group's queries — so the backfill + samples match what the live runner would have produced for any + query in that group. + """ + cadence = min( + (effective_refresh_interval(q, config.server) for q in group.queries), + default=config.server.refresh_interval, + ) + cadence_s = max(60.0, cadence.total_seconds()) + earliest = to_time - timedelta(seconds=from_seconds) + current = earliest + while current <= to_time: + yield current + current = current + timedelta(seconds=cadence_s) + + +def _select_groups(config: Config, selector: list[str] | None) -> list[GroupConfig]: + """Filter ``config.groups`` against the operator's ``--groups`` flag.""" + if not selector: + return list(config.groups) + wanted = set(selector) + return [g for g in config.groups if g.name in wanted] + + +def _check_window(config: Config, from_seconds: float) -> None: + """Reject backfill windows larger than ``safety.backfill_max_window``.""" + cap = config.safety.backfill_max_window.total_seconds() + if cap > 0 and from_seconds > cap: + raise BackfillError( + f"--from {from_seconds:.0f}s exceeds safety.backfill_max_window " + f"({cap:.0f}s); raise the cap explicitly when the cluster is sized " + "for the work. See docs/operations/backfill.md." + ) + + +def _parse_duration_seconds(value: str, *, flag: str) -> float: + """Resolve a Prometheus-style duration to seconds. + + Reuses :func:`config.parse_duration` so the CLI accepts the same + ``5m`` / ``1h`` / ``30d`` syntax the YAML schema uses. + """ + parsed = config_module.parse_duration(value) + if not isinstance(parsed, timedelta): + raise BackfillError(f"{flag} must be a Prometheus duration; got {value!r}") + seconds = parsed.total_seconds() + if seconds <= 0: + raise BackfillError(f"{flag} must be a positive duration; got {value!r}") + return seconds + + +def _parse_to_timestamp(value: str | None) -> datetime: + """Resolve ``--to`` to an aware UTC datetime; ``None`` / ``now`` -> now.""" + if value is None or value.lower() == "now": + return datetime.now(tz=UTC) + try: + dt = datetime.fromisoformat(value.replace("Z", "+00:00")) + except ValueError as exc: + raise BackfillError(f"--to {value!r} must be 'now' or an ISO-8601 timestamp") from exc + if dt.tzinfo is None: + dt = dt.replace(tzinfo=UTC) + return dt diff --git a/forecaster/src/promforecast/capacity.py b/forecaster/src/promforecast/capacity.py index 31a40fa..ed85e3f 100644 --- a/forecaster/src/promforecast/capacity.py +++ b/forecaster/src/promforecast/capacity.py @@ -94,6 +94,46 @@ class WillBreach: value: float +@dataclass(frozen=True) +class ThresholdActionCost: + """Config-derived cost-of-action annotation for one threshold. + + Surfaces as the per-(group, query, threshold) sample of + ``forecast_threshold_action_cost`` so dashboards can rank capacity + ETAs by cost-to-ignore. ``cost_type`` corresponds to the + operator-configured ``action_cost.type`` and renders on the + ``type`` label (renamed locally to avoid shadowing the builtin). + """ + + name: str + cost_type: str + cost_value: float + + +def build_threshold_action_costs( + thresholds: list[ThresholdConfig], +) -> list[ThresholdActionCost]: + """Project a query's configured thresholds onto action-cost annotations. + + Skips thresholds without an ``action_cost`` block so the family + only carries the opted-in subset. The output ordering matches the + config ordering so successive runs publish a stable row sequence. + """ + out: list[ThresholdActionCost] = [] + for threshold in thresholds: + cost = threshold.action_cost + if cost is None: + continue + out.append( + ThresholdActionCost( + name=threshold.name, + cost_type=cost.type, + cost_value=float(cost.value), + ) + ) + return out + + def compute_will_breach( *, etas: list[ThresholdEta], diff --git a/forecaster/src/promforecast/config.py b/forecaster/src/promforecast/config.py index 9fea52e..05f7a9e 100644 --- a/forecaster/src/promforecast/config.py +++ b/forecaster/src/promforecast/config.py @@ -152,6 +152,16 @@ class ServerConfig(BaseModel): # ``safety.fleet_what_if_min_interval``. Same network-level # access-control model as ``/-/reload`` / ``/forecast/preview``. expose_what_if_endpoint: bool = False + # ``GET /forecast/warmup`` reports per-(group, query) "what is still + # loading?" status for a fresh install: how many points each query + # has versus how many it needs, the blocking reason (min_points, + # missing regressor, ...), and a rough ETA to first usable forecast. + # Off by default — the endpoint costs one cheap probe query per + # configured query when called, which is fine for ad-hoc debugging + # but would be wasteful for any monitoring system polling it on + # cadence. Same network-level access-control model as + # ``/-/reload``. See ``docs/operations/warmup.md``. + expose_warmup_endpoint: bool = False class SafetyConfig(BaseModel): @@ -191,6 +201,16 @@ class SafetyConfig(BaseModel): # interactive dashboards; setting it higher is appropriate for # forecasters that share an executor with high-cardinality groups. fleet_what_if_min_interval: Duration = timedelta(minutes=5) + # Hard upper bound on the backfill window passed to + # ``promforecast backfill --from ``. Backfill walks one + # fit per ``server.refresh_interval`` across the requested window — + # at the default 1h refresh a 90d window is 2160 fits per group, + # comfortably in the hours of CPU range. The cap rejects accidental + # year-long requests at the CLI boundary rather than letting them + # silently kick off an overnight job; raise it explicitly only when + # the operator has sized the cluster for the work. See + # ``docs/operations/backfill.md`` for the trade-off. + backfill_max_window: Duration = timedelta(days=90) class ConformalConfig(BaseModel): @@ -379,6 +399,29 @@ class DiscoveryVariable(BaseModel): promql: str +class ActionCostConfig(BaseModel): + """Optional cost-of-action annotation for a threshold. + + Surfaces as ``forecast_threshold_action_cost{id, group, name, type}`` + whose sample value is ``value``. Dashboards multiplex + ``forecast_time_to_threshold_seconds`` by this gauge to rank + "soon-and-expensive" first — an ETA of 7 days on a 50,000-currency + storage tier outranks an ETA of 1 day on a 50-currency backup + volume. + + ``type`` is a free-text discriminator label (``usd``, ``eur``, + ``priority``, ``engineer_hours``) so non-monetary cost framings + work alongside currency ones. ``value`` is the numeric amount; + ``> 0`` is enforced because a zero or negative cost makes no + sense and would silently rank below every meaningful threshold. + """ + + model_config = ConfigDict(extra="forbid") + + type: str + value: float = Field(gt=0.0) + + class ThresholdConfig(BaseModel): """One named threshold for the time-to-threshold ETA metric. @@ -402,6 +445,12 @@ class ThresholdConfig(BaseModel): rejected at load time) — the predictive metric is a derived shape and cannot be partially configured. + ``action_cost`` opts the threshold into the + ``forecast_threshold_action_cost`` gauge — see + :class:`ActionCostConfig`. The cost rides with the threshold's + lifecycle so a Grafana panel can rank capacity ETAs by + cost-to-ignore rather than just time-to-breach. + The ``le_drain`` comparator targets the queue-burn-down case. It differs from ``lt`` along four axes: @@ -431,6 +480,7 @@ class ThresholdConfig(BaseModel): comparator: Literal["gt", "lt", "le_drain"] = "gt" alert_within: Duration | None = None severity: Literal["warning", "critical"] | None = None + action_cost: ActionCostConfig | None = None class SLOConfig(BaseModel): @@ -552,6 +602,12 @@ class DriftEmissionConfig(BaseModel): enabled: bool = True method: Literal["mape", "wasserstein"] = "mape" alert_threshold: float | None = 0.25 + # Rolling window (in fits) used by the drift-explainer's Pearson + # correlation between drift_score and each configured regressor. + # Default 30 fits matches the v1.8 spec; lower values produce + # noisier correlations, higher values are slower to react to a + # genuine regime change in which regressor explains the drift. + correlation_window: int = Field(default=30, ge=2) class BandCoverageConfig(BaseModel): @@ -862,18 +918,93 @@ class ChangePointConfig(BaseModel): reweight_decay: float = Field(default=0.5, ge=0.0, le=1.0) +class BlacklistWindowConfig(BaseModel): + """One operator-known anomalous window to drop from the lookback. + + Samples whose timestamp falls inside ``[start, end)`` are dropped + *before* gap imputation, so the standard ``gaps`` strategy treats the + blacklisted span as a gap and applies the configured fill / drop + policy. The window is interpreted as a half-open interval so two + adjacent windows touching at the boundary cover the boundary point + exactly once. + + Two shapes are supported: + + * **Absolute** — ``start`` / ``end`` are ISO-8601 timestamps (e.g. + ``2024-11-29T00:00:00Z``). Useful for "the November 2024 incident + window" or "the Q1 migration freeze". The ``timezone`` field is + ignored for absolute windows because the ISO-8601 string already + carries an offset (or implies UTC when none is given). + * **Recurring** — ``recurring`` carries a calendar pattern (``annual`` + today; ``weekly`` / ``monthly`` are reserved for future expansion). + ``start`` / ``end`` are formatted as ``MM-DDTHH:MM`` (year omitted) + so the window applies to every year in the lookback. ``timezone`` + controls the local calendar the window is anchored in — defaults + to ``UTC``, but US Black-Friday-style operators typically want + ``America/New_York`` or their site's local zone so the window + flips at local midnight rather than UTC-midnight. Same convention + as the v1.4 ``holidays`` regressor's ``timezone`` field. + + The optional ``reason`` is a free-text tag surfaced on the per-window + counter (``forecast_input_blacklisted_samples_total{reason}``) so an + operator can audit which windows are actually firing without log + scraping. Reason cardinality is bounded by the operator's config — + avoid templating per-incident values into it. + """ + + model_config = ConfigDict(extra="forbid") + + start: str + end: str + reason: str = "" + # ``recurring: null`` (default) treats ``start``/``end`` as absolute + # ISO-8601 timestamps. ``recurring: annual`` treats them as + # ``MM-DDTHH:MM`` calendar patterns and applies the window across + # every year in the lookback. Other periodicities are reserved. + recurring: Literal["annual"] | None = None + # IANA timezone name used to anchor a recurring window's calendar + # boundary. Ignored when ``recurring`` is null (the absolute form + # carries its own offset in the ISO-8601 string). Defaults to + # ``UTC`` so a missing field never silently shifts the window. + timezone: str = "UTC" + + @model_validator(mode="after") + def _check_shape(self) -> BlacklistWindowConfig: + if not self.start or not self.end: + raise ValueError("blacklist window requires both start and end") + if self.recurring == "annual": + # ``MM-DDTHH:MM`` shape: 5 chars + ``T`` + 5 chars = 11. + # Accept the trailing seconds for symmetry with ISO-8601. + if "T" not in self.start or "T" not in self.end: + raise ValueError( + "annual blacklist windows require 'MM-DDTHH:MM' shape " + f"(got start={self.start!r}, end={self.end!r})" + ) + # Absolute window — the parser in preprocess.py reads via + # ``datetime.fromisoformat``; do a cheap structural check + # here so a typo fails at boot rather than mid-fit. + elif "T" not in self.start or "T" not in self.end: + raise ValueError( + "absolute blacklist windows require ISO-8601 timestamps " + f"(got start={self.start!r}, end={self.end!r})" + ) + return self + + class PreprocessConfig(BaseModel): """All pre-fit cleaning steps for a group. - Steps run in the order ``gaps -> outliers -> change_point``: gaps are - filled first so outlier detection sees a regularly-sampled signal, and - change-point detection runs last on the cleaned series so a single - spike doesn't masquerade as a regime shift. - - All three sub-sections default to inert behaviour (``forward_fill`` with - a 30-minute cap, outliers off, change-point off), so adding the block - to a config without further edits does nothing observable. Each - individual feature flips on independently. + Steps run in the order ``blacklist -> gaps -> outliers -> change_point``: + blacklisted samples are dropped first so gap imputation treats the window + as a gap (and may fall back to ``drop_series`` if the window is longer + than ``gaps.max_gap``); gaps are filled so outlier detection sees a + regularly-sampled signal; outliers are scrubbed before change-point + detection so a single spike doesn't masquerade as a regime shift. + + All four sub-sections default to inert behaviour (``blacklist_windows`` + empty, ``forward_fill`` with a 30-minute cap, outliers off, change-point + off), so adding the block to a config without further edits does nothing + observable. Each individual feature flips on independently. """ model_config = ConfigDict(extra="forbid") @@ -881,6 +1012,7 @@ class PreprocessConfig(BaseModel): gaps: GapImputationConfig = GapImputationConfig() outliers: OutlierConfig = OutlierConfig() change_point: ChangePointConfig = ChangePointConfig() + blacklist_windows: list[BlacklistWindowConfig] = Field(default_factory=list) class GroupConfig(BaseModel): @@ -1133,6 +1265,135 @@ class DeploySignalConfig(BaseModel): weight: float = Field(default=1.0, gt=0.0) +class ResourceSizingWorkloadConfig(BaseModel): + """One workload's right-sizing wiring. + + Off by default. Activated by configuring the parent + :class:`ResourceSizingConfig` block. Each entry binds a Kubernetes + workload to the forecast it should be sized against, plus + optionally a PromQL query that reports the *current* limit/request + so the runner can compute an expected savings percentage. + + ``forecast_query`` must match the ``id`` of a configured query; + the runner reads its predicted peak (max ``yhat_upper`` at the + highest configured confidence level, falling back to ``yhat``) + and applies the headroom multiplier. + + ``current_promql`` is optional. When set, the runner fetches it + via the existing :class:`PromSource` (a cheap instant-style + range query) and emits the matching ``current`` and + ``expected_savings_pct`` gauges alongside the recommendation. When + unset only the recommendation is emitted; operators can derive + the savings comparison via PromQL recording rules against their + own infrastructure metrics (``kube_pod_container_resource_requests``, + ...). + """ + + model_config = ConfigDict(extra="forbid") + + workload: str + kind: Literal["cpu", "memory"] + forecast_query: str + current_promql: str | None = None + + +class ResourceSizingConfig(BaseModel): + """Per-workload right-sizing recommendations from forecast peaks. + + Off by default. When configured, the runner walks each + :class:`ResourceSizingWorkloadConfig` after every group run, + reads the forecast snapshot for the named query, projects the + headroom-adjusted peak, and emits + ``forecast_recommended_resource_size{workload, kind, confidence}`` + plus optional ``current`` / ``savings_pct`` siblings. + + ``headroom_pct`` is added on top of the predicted peak — a 30% + headroom means the recommendation is ``peak * 1.30``. Defaults to + ``30`` because anything below 20% leaves no rope for unexpected + load and anything above 50% defeats the purpose of right-sizing. + + ``quality_floor`` gates the emission against + ``forecast_quality_score``. A forecast whose quality is below the + floor produces no recommendation for that workload — better to + omit the row than mislead operators with a recommendation built + on an untrustworthy forecast. + """ + + model_config = ConfigDict(extra="forbid") + + enabled: bool = False + headroom_pct: float = Field(default=30.0, ge=0.0, le=200.0) + quality_floor: float = Field(default=0.6, ge=0.0, le=1.0) + workloads: list[ResourceSizingWorkloadConfig] = Field(default_factory=list) + + +class RecommendationsConfig(BaseModel): + """Operator-facing recommendation outputs. + + Off by default. Today only :class:`ResourceSizingConfig` is + surfaced under this block; future recommendation families land + here so a config can enable them all under one umbrella. + """ + + model_config = ConfigDict(extra="forbid") + + resource_sizing: ResourceSizingConfig = Field(default_factory=ResourceSizingConfig) + + +class HealthScoreConfig(BaseModel): + """Per-service forecast health composite. + + Off by default — when configured, the runner aggregates + ``forecast_quality_score`` samples across all groups and emits + ``forecast_service_health_score`` per distinct combination of the + labels listed in ``aggregate_by``. The metric is intended for + "how reliable is our prediction infrastructure for this service / + environment?" reporting, distinct from "how reliable is the + service itself?". + + ``aggregate_by`` lists the source-side labels (e.g. ``service``, + ``environment``, ``cluster``) that become discriminator labels on + the emitted metric; quality_points whose label set is missing any + of these are dropped from the aggregation (so a single un-labelled + forecast can't silently inflate or deflate every bucket). + + ``aggregator`` picks the summary statistic — currently only + ``weighted_mean`` is supported (every quality_point is one series, + so the weight is implicitly 1 per row regardless of ``weight_by``). + ``weight_by`` is exposed for forward compatibility with future + pre-aggregated quality emission shapes; today it does not change + the math but the configured value rides through onto the emitted + ``weight_by`` documentation label so downstream BI dashboards + can record which interpretation was in effect. + + ``quality_floor`` (optional) drops quality scores below the floor + before aggregating. Useful when cold-start or low-quality forecasts + should not pull a service's health score down — set the floor at + the same value as the alert gate (typically ``0.6``) so the + aggregated number aligns with the alerting story. + """ + + model_config = ConfigDict(extra="forbid") + + aggregate_by: list[str] = Field(min_length=1) + aggregator: Literal["weighted_mean"] = "weighted_mean" + weight_by: Literal["uniform", "forecast_series_count"] = "forecast_series_count" + quality_floor: float | None = Field(default=None, ge=0.0, le=1.0) + + +class GovernanceConfig(BaseModel): + """Operator / platform-team aggregated views over forecast quality. + + Off by default. The only sub-block today is :class:`HealthScoreConfig`; + new governance-style aggregations will land here so a config can + enable them all under one block. + """ + + model_config = ConfigDict(extra="forbid") + + health_score: HealthScoreConfig | None = None + + class DeploySignalsConfig(BaseModel): """One service's deploy-risk composite definition. @@ -1198,6 +1459,12 @@ class Config(BaseModel): # queries. The runner emits ``forecast_deploy_risk_score`` per # active deploy. See :class:`DeploySignalsConfig`. deploy_signals: list[DeploySignalsConfig] = Field(default_factory=list) + # Governance aggregations over forecast quality. Off by default; + # see :class:`GovernanceConfig`. + governance: GovernanceConfig = Field(default_factory=GovernanceConfig) + # Operator-facing recommendations (right-sizing, ...). Off by + # default; see :class:`RecommendationsConfig`. + recommendations: RecommendationsConfig = Field(default_factory=RecommendationsConfig) def load(path: str | Path) -> Config: @@ -1238,9 +1505,54 @@ def load(path: str | Path) -> Config: for reg in query.regressors: _validate_regressor(group_name=group.name, query_id=query.id, reg=reg) _validate_deploy_signals(config) + _validate_resource_sizing(config) return config +def _validate_resource_sizing(config: Config) -> None: + """Cross-reference every right-sizing workload against the live queries. + + Each workload entry's ``forecast_query`` must resolve to a + configured query id; an unknown id is almost always a typo that + would otherwise silently emit nothing. Duplicate (workload, kind) + pairs are rejected so the emitted gauge stays single-valued — + otherwise two entries would write to the same label set and the + last one would silently win. + """ + cfg = config.recommendations.resource_sizing + if not cfg.enabled or not cfg.workloads: + return + queries_by_id: set[str] = {q.id for g in config.groups for q in g.queries} + seen: set[tuple[str, str]] = set() + for entry in cfg.workloads: + label = f"recommendations.resource_sizing.workloads[{entry.workload}/{entry.kind}]" + if entry.forecast_query not in queries_by_id: + raise ValueError( + f"{label}: forecast_query={entry.forecast_query!r} does not " + "match any configured query id" + ) + # ``current_promql`` is optional, but if the operator sets it + # to the empty string the runtime fetcher would treat the + # value as "configured" and issue a doomed PromQL request + # every refresh. Reject at load time so the misconfig is + # visible. + if entry.current_promql is not None and not entry.current_promql.strip(): + raise ValueError( + f"{label}: current_promql must be a non-empty PromQL query when " + "set (omit the field to skip the current / savings_pct emission)" + ) + if not entry.workload.strip(): + raise ValueError(f"{label}: workload must be non-empty") + key = (entry.workload, entry.kind) + if key in seen: + raise ValueError( + f"recommendations.resource_sizing.workloads: duplicate " + f"(workload={entry.workload!r}, kind={entry.kind!r}); " + "each pair must appear at most once" + ) + seen.add(key) + + def _validate_deploy_signals(config: Config) -> None: """Cross-reference every ``deploy_signals`` entry against the live queries. @@ -1398,6 +1710,14 @@ def _validate_query_overrides(*, group_name: str, query: QueryConfig) -> None: "(it would fire on a queue draining quickly). Use the +Inf-anchored " "QueueNotDraining-style alert on forecast_time_to_drain_seconds instead." ) + # ``type`` becomes a free-text label on /metrics — empty or + # whitespace-only values would silently render as an empty + # label which dashboards typically discard. Reject at boot. + if threshold.action_cost is not None and not threshold.action_cost.type.strip(): + raise ValueError( + f"{label}.thresholds[{threshold.name}].action_cost.type must be non-empty " + "(e.g. 'usd', 'eur', 'engineer_hours')" + ) _validate_cold_start(label=label, query=query) _validate_events_profile(label=label, query=query) growth = query.emission.growth_rate diff --git a/forecaster/src/promforecast/drift_explainer.py b/forecaster/src/promforecast/drift_explainer.py new file mode 100644 index 0000000..9c67594 --- /dev/null +++ b/forecaster/src/promforecast/drift_explainer.py @@ -0,0 +1,254 @@ +"""Change-correlation explanation for ``forecast_drift_score`` events. + +When ``forecast_drift_score`` crosses its configured threshold, the +runner emits a structured ``forecast_drift_explained`` log event whose +fields collect "what else moved at the same time?" context. That turns +a generic drift alert ("something drifted") into an actionable one +("start your investigation on the ``deploys`` regressor — it explains +0.82 of the drift variance over the last 30 fits"). + +Two suspect families coexist on every explanation: + +* **``top_value_suspect``** — the regressor with the largest absolute + value at the drift moment. Useful when the drift is driven by a + one-shot event (a deploy that just fired); the regressor is "the + thing that's loudest right now". +* **``top_correlation_suspect``** — the regressor whose value history + most predicts the drift_score history over the rolling + ``correlation_window``. Useful when the drift is driven by a slower + signal that's been trending alongside the forecast. + +The ``primary_suspect`` label on the +``forecast_drift_explained_total`` counter prefers the +correlation-based suspect when the rolling window has enough samples +for a meaningful Pearson coefficient; it falls back to the +value-based suspect when correlation is ``insufficient_data``. + +Pure helpers — the runner owns the rolling-window state and the log +emission; this module computes the suspects and the correlation +math. +""" + +from __future__ import annotations + +import math +from collections import deque +from dataclasses import dataclass, field +from typing import Literal + +CorrelationConfidence = Literal["high", "medium", "low", "insufficient_data"] + +# Minimum sample size to compute a Pearson correlation at all. Below +# this, the explanation falls back to the value-based suspect. +_MIN_PAIRS_FOR_CORRELATION = 4 +# Confidence thresholds tuned for "roughly aligned with operator +# intuition": at high confidence the operator should investigate the +# suspect first; at medium they should double-check; at low the +# correlation exists but the relationship is weak. +_CONFIDENCE_HIGH_ABS = 0.7 +_CONFIDENCE_HIGH_N = 20 +_CONFIDENCE_MEDIUM_ABS = 0.4 +_CONFIDENCE_MEDIUM_N = 8 + + +@dataclass(frozen=True) +class _Sample: + """One historical observation for the rolling window.""" + + drift_score: float + regressor_values: dict[str, float] + + +@dataclass +class DriftWindow: + """Per-(group, query) ring buffer of drift + regressor observations. + + The maxlen is bounded by the operator's ``correlation_window`` + config so a long-running deployment doesn't accumulate unbounded + history. ``record`` appends; ``rebuild`` is used when the operator + reloads with a different window size (so the buffer respects the + new bound on the next fit). + """ + + maxlen: int + _samples: deque[_Sample] = field(init=False) + + def __post_init__(self) -> None: + self._samples = deque(maxlen=max(1, self.maxlen)) + + def record(self, drift_score: float, regressor_values: dict[str, float]) -> None: + """Append one observation, evicting the oldest when full.""" + self._samples.append( + _Sample(drift_score=drift_score, regressor_values=dict(regressor_values)) + ) + + def samples(self) -> list[_Sample]: + """Return a snapshot of the current window (oldest-first).""" + return list(self._samples) + + def __len__(self) -> int: + return len(self._samples) + + +@dataclass(frozen=True) +class DriftExplanation: + """Composite explanation for one drift threshold-crossing event. + + ``primary_suspect`` is the operator-facing label on the + ``forecast_drift_explained_total`` counter — preferring the + correlation-based suspect when correlation confidence is high or + medium, falling back to the value-based suspect otherwise (and + to ``"unknown"`` when neither candidate produced a finite answer). + """ + + primary_suspect: str + top_value_suspect: str + top_regressor_correlation: float + top_regressor_correlation_confidence: CorrelationConfidence + top_correlation_suspect: str + correlation_window_size: int + + +def compute_pearson(xs: list[float], ys: list[float]) -> tuple[float, int]: + """Return ``(coefficient, n_finite_pairs)``. + + ``n_finite_pairs`` is the count of pairs where both ``x`` and + ``y`` are finite — non-finite values are skipped so a single NaN + regressor reading doesn't poison the coefficient. When fewer than + :data:`_MIN_PAIRS_FOR_CORRELATION` finite pairs exist, the + coefficient is reported as ``NaN`` (the caller treats this as + ``insufficient_data``). + + Division-by-zero (zero variance on either side) returns ``NaN`` + rather than raising — a flat regressor over the window can't + explain anything, and the caller should fall back to the + value-based suspect. + """ + if len(xs) != len(ys): + raise ValueError("compute_pearson: xs and ys must have equal length") + pairs = [(x, y) for x, y in zip(xs, ys, strict=True) if math.isfinite(x) and math.isfinite(y)] + n = len(pairs) + if n < _MIN_PAIRS_FOR_CORRELATION: + return float("nan"), n + sum_x = sum(p[0] for p in pairs) + sum_y = sum(p[1] for p in pairs) + mean_x = sum_x / n + mean_y = sum_y / n + cov = sum((p[0] - mean_x) * (p[1] - mean_y) for p in pairs) + var_x = sum((p[0] - mean_x) ** 2 for p in pairs) + var_y = sum((p[1] - mean_y) ** 2 for p in pairs) + if var_x == 0 or var_y == 0: + return float("nan"), n + return cov / math.sqrt(var_x * var_y), n + + +def classify_confidence(coefficient: float, n_pairs: int) -> CorrelationConfidence: + """Map ``(|coefficient|, n_pairs)`` to one of four named bands. + + The bands are deliberately coarse — operators read them as "should + I trust this suspect?", not as a p-value. The cutoffs are sized + around what an operator would call "really worth investigating + first": ``high`` requires both a strong correlation *and* enough + samples, ``medium`` accepts one of the two. + """ + if not math.isfinite(coefficient) or n_pairs < _MIN_PAIRS_FOR_CORRELATION: + return "insufficient_data" + abs_coef = abs(coefficient) + if abs_coef >= _CONFIDENCE_HIGH_ABS and n_pairs >= _CONFIDENCE_HIGH_N: + return "high" + if abs_coef >= _CONFIDENCE_MEDIUM_ABS and n_pairs >= _CONFIDENCE_MEDIUM_N: + return "medium" + return "low" + + +def explain( + window: DriftWindow, + current_regressor_values: dict[str, float], +) -> DriftExplanation: + """Produce a :class:`DriftExplanation` from a rolling window snapshot. + + The current observation is *also* fed in (separate from the + window) so the value-based suspect always reflects "what was loud + at the drift moment" — even when the window itself is empty + (e.g. a fresh install whose very first fit already trips the + drift threshold). + + Returns a sentinel explanation (``primary_suspect="unknown"``, + ``correlation_confidence="insufficient_data"``) when no regressors + are configured / available; the caller still emits the structured + log line so the operator can see "drift fired but no suspects" as + its own signal. + """ + samples = window.samples() + n_samples = len(samples) + top_value_suspect = _select_top_value_suspect(current_regressor_values) + top_correlation_suspect, coefficient = _select_top_correlation_suspect(samples) + confidence = classify_confidence(coefficient, n_samples) + # Primary suspect picks correlation when the band is high/medium — + # those are "trust this candidate enough to start the + # investigation". Low / insufficient_data falls back to the + # value-based suspect, which never has confidence semantics (it's + # a snapshot of "what's loud right now"). + primary = top_correlation_suspect if confidence in ("high", "medium") else top_value_suspect + if not primary: + primary = "unknown" + return DriftExplanation( + primary_suspect=primary, + top_value_suspect=top_value_suspect or "unknown", + top_regressor_correlation=coefficient, + top_regressor_correlation_confidence=confidence, + top_correlation_suspect=top_correlation_suspect or "unknown", + correlation_window_size=n_samples, + ) + + +def _select_top_value_suspect(regressor_values: dict[str, float]) -> str: + """Pick the regressor with the largest absolute value (or empty string).""" + best_id: str = "" + best_abs: float = float("-inf") + for regressor_id, value in regressor_values.items(): + if not math.isfinite(value): + continue + abs_val = abs(float(value)) + if abs_val > best_abs: + best_abs = abs_val + best_id = regressor_id + return best_id + + +def _select_top_correlation_suspect(samples: list[_Sample]) -> tuple[str, float]: + """Walk the window and return the (regressor_id, coefficient) with max |corr|. + + Returns ``("", NaN)`` when the window is too short for any + correlation. Ties on absolute correlation break by regressor id + alphabetically so successive runs report a stable suspect when + several regressors share the same coefficient (e.g. a uniform + flat signal across all of them). + """ + if len(samples) < _MIN_PAIRS_FOR_CORRELATION: + return "", float("nan") + # Pivot the per-sample dict-of-values into per-regressor lists, + # filling missing samples with NaN so the per-pair filter in + # ``compute_pearson`` drops them cleanly. + regressor_ids: set[str] = set() + for sample in samples: + regressor_ids.update(sample.regressor_values) + if not regressor_ids: + return "", float("nan") + drift_series = [sample.drift_score for sample in samples] + best_id: str = "" + best_abs: float = float("-inf") + best_signed: float = float("nan") + for regressor_id in sorted(regressor_ids): + reg_series = [sample.regressor_values.get(regressor_id, float("nan")) for sample in samples] + coef, _ = compute_pearson(drift_series, reg_series) + if not math.isfinite(coef): + continue + abs_coef = abs(coef) + if abs_coef > best_abs: + best_abs = abs_coef + best_id = regressor_id + best_signed = coef + if not best_id: + return "", float("nan") + return best_id, best_signed diff --git a/forecaster/src/promforecast/exporter.py b/forecaster/src/promforecast/exporter.py index a2807e0..4d0617c 100644 --- a/forecaster/src/promforecast/exporter.py +++ b/forecaster/src/promforecast/exporter.py @@ -18,6 +18,7 @@ import math import threading +import time from dataclasses import dataclass, field from typing import TYPE_CHECKING, Any @@ -25,6 +26,11 @@ from prometheus_client.core import CounterMetricFamily, GaugeMetricFamily from ._naming import InvalidMetricNameError, validate_metric_name +from .self_observability import ( + FitOutcomeTracker, + GroupResourceTracker, + RegressorStabilityTracker, +) if TYPE_CHECKING: from collections.abc import Callable, Iterator @@ -49,9 +55,11 @@ "GrowthRatePoint", "IncidentRecurrencePoint", "InvalidMetricNameError", + "ModelDisagreementPoint", "PreprocessStats", "QualityPoint", "RecommendedMaintenanceWindowPoint", + "RevisionPoint", "ThresholdEtaPoint", "TimeToDrainPoint", "WillBreachPoint", @@ -282,6 +290,59 @@ class GrowthRatePoint: value: float +@dataclass(frozen=True) +class ResourceSizingPoint: + """One ``forecast_recommended_resource_size{workload, kind, confidence}`` row. + + The sample value is the recommended size in the metric's native + units (CPU cores, memory bytes, ...). ``current`` / ``savings_pct`` + populate the sibling gauges below when the operator wired + ``current_promql`` on the workload; ``None`` skips that sibling + for the row. + """ + + labels: dict[str, str] + recommended: float + current: float | None + savings_pct: float | None + + +@dataclass(frozen=True) +class ServiceHealthScorePoint: + """One ``forecast_service_health_score`` sample. + + Global metric name (no per-id prefix) — the ``aggregate_by`` label + keys ride the label set so the row is uniquely identified by the + bucket (e.g. ``service="checkout", environment="prod"``). The + sample value is the aggregated quality score in ``[0, 1]``; + ``aggregator`` and ``weight_by`` ride as documentation labels so a + Grafana panel can record which interpretation was in effect. + ``series_count`` rides as a label (rather than a separate metric) + so the bucket size is visible alongside the aggregated number. + """ + + labels: dict[str, str] + score: float + + +@dataclass(frozen=True) +class ThresholdActionCostPoint: + """One ``forecast_threshold_action_cost`` sample. + + Global metric name (no per-id prefix) — ``id`` is a label so a + single Grafana panel can rank capacity ETAs by cost-to-ignore + across every threshold in the fleet. The sample value is the + operator-configured cost amount; the ``type`` label discriminates + the cost unit (``usd``, ``eur``, ``engineer_hours``, ...). + Emitted once per (group, query, threshold) — independent of model + / series fan-out — so the family stays small even on + high-cardinality groups. + """ + + labels: dict[str, str] + value: float + + @dataclass(frozen=True) class DriftPoint: """One ``forecast_drift_score`` sample. @@ -360,6 +421,38 @@ class DeployRiskPoint: score: float +@dataclass(frozen=True) +class RevisionPoint: + """One ``forecast_revision_magnitude`` sample. + + Renders as ``forecast_revision_magnitude{id, group, model, horizon, ...}`` + — the per-(series, model, horizon) magnitude of change between the + current fit's prediction and the previous fit's prediction for the + same horizon. Unsigned (operators care about "the forecast moved", + not "moved up vs. down"); normalised by the prior value's scale so + a 30% shift scores 0.3 regardless of magnitude. + """ + + labels: dict[str, str] + magnitude: float + + +@dataclass(frozen=True) +class ModelDisagreementPoint: + """One ``forecast_model_disagreement`` sample. + + Renders as ``forecast_model_disagreement{id, group, horizon}`` — the + coefficient of variation across the central forecasts of every + configured model at the given horizon. Emitted only in explicit + mode (when 2+ models contribute distinct forecasts); auto-select + suppresses the metric for that series because only the winning + model is emitted. + """ + + labels: dict[str, str] + score: float + + @dataclass(frozen=True) class IncidentRecurrencePoint: """One ``forecast_incident_recurrence_score`` sample. @@ -514,6 +607,12 @@ class GroupSnapshot: outliers_replaced_cumulative: dict[tuple[str, str], int] = field(default_factory=dict) # Cumulative per-(query) change-point detection counter. change_points_cumulative: dict[str, int] = field(default_factory=dict) + # Cumulative per-(query, reason) blacklist-window sample counter. + # ``reason`` defaults to the empty string for unreasoned windows so + # the counter is always single-valued even when the operator did not + # configure a reason. Merged with the previous snapshot in + # ``run_group`` so the exposed counter stays monotonic across runs. + blacklisted_samples_cumulative: dict[tuple[str, str], int] = field(default_factory=dict) # Cumulative per-(query, kind) emission clipping counter. # ``kind`` is bounded to ``negative`` (a value clipped from below zero # by ``emission.clip_non_negative``) and ``rounded`` (a value whose @@ -538,6 +637,24 @@ class GroupSnapshot: # separate ``BandCoverageTracker`` and snapshots the current # fractions here at render time. band_coverage_points: list[BandCoveragePoint] = field(default_factory=list) + # Per-(series, model, horizon) forecast-revision magnitude — how + # much the predicted value moved between the most recent fit and + # the one before it, normalised by the prior value's typical scale. + # Snapshot-style: only the latest run's samples land on /metrics. + revision_points: list[RevisionPoint] = field(default_factory=list) + # Per-(series, horizon) model-disagreement gauge — the coefficient + # of variation across the configured models' central forecasts at + # the same horizon. Emitted only in explicit mode with 2+ models + # contributing distinct forecasts; auto-select suppresses the + # metric because only the winner is emitted. + model_disagreement_points: list[ModelDisagreementPoint] = field(default_factory=list) + # Per-(group, query, threshold) cost-of-action gauge. Config-derived + # constant — one row per threshold with ``action_cost`` configured, + # independent of model / series fan-out. The runner rebuilds the + # list from the current config on every group run so a threshold + # that's been removed from YAML disappears from /metrics on the + # next refresh. + threshold_action_cost_points: list[ThresholdActionCostPoint] = field(default_factory=list) class Exporter: @@ -646,6 +763,54 @@ def __init__(self) -> None: # cycle's scores are emitted. Bounded cardinality: # 7 days * 24 hours = 168 rows per opted-in (group, query). self._incident_recurrence: dict[tuple[str, str, str, str], IncidentRecurrencePoint] = {} + # Service-level forecast health composite. Keyed by the + # operator's ``aggregate_by`` tuple of label values. Snapshot + # state lives on the exporter (rather than per-group snapshot) + # because the aggregation crosses groups by design — one + # bucket like ``service=checkout, env=prod`` typically pulls + # quality_points from multiple groups. + self._service_health_scores: dict[tuple[tuple[str, str], ...], ServiceHealthScorePoint] = {} + # Right-sizing recommendations. Keyed by (workload, kind) — + # operator-configured cardinality is small (typically a few + # workloads x 2 kinds). Lives on the exporter (rather than + # per-group snapshot) because a workload spec is a top-level + # config; the runner repopulates the whole map after every + # group run. + self._resource_sizing_recommendations: dict[tuple[str, str], ResourceSizingPoint] = {} + # Per-(group, query, primary_suspect) drift-explained counter. + # Bumped once per drift threshold-crossing event; the + # ``primary_suspect`` label is either the regressor id the + # change-correlation explainer identified or ``"unknown"`` + # when no candidate emerged. Cardinality is bounded by the + # configured regressor count plus one (``unknown``). + self._drift_explained: dict[tuple[str, str, str], int] = {} + # Per-(group, query, model, series_key) last-successful-fit + # timestamp. Renders as ``forecast_data_staleness_seconds`` — + # the per-series complement to the group-level + # ``forecast_last_run_timestamp_seconds``. Successful fits + # write a fresh timestamp; failed fits leave the previous + # value alone so the staleness clock keeps running. The labels + # captured at first record let the render path emit the + # complete labelset (source labels + id + group + model) + # without the runner having to thread them through every + # subsequent render. Bounded by ``N_series * N_models`` per + # opted-in query — already counted as part of the forecast + # emission so no new cardinality budget is required. + self._series_last_success: dict[ + tuple[str, str, str, str], tuple[float, dict[str, str]] + ] = {} + # Per-(group, model) rolling fit-outcome ledger backing the + # ``forecast_model_fit_success_ratio`` family — fed by every + # per-model fit attempt (success and failure) via + # :meth:`record_fit_outcome`. Per-(group, query, regressor_id) + # stability scores back ``forecast_regressor_stability_score`` + # and are refreshed from the runner once per fit. Per-group + # CPU + memory accounting backs the ``forecast_group_*`` + # gauges; both live on the exporter so /metrics can read + # them without reaching into the runner. + self._fit_outcome_tracker = FitOutcomeTracker() + self._regressor_stability_tracker = RegressorStabilityTracker() + self._group_resource_tracker = GroupResourceTracker() self._registry = CollectorRegistry() self._registry.register(_SnapshotCollector(self)) @@ -1004,6 +1169,247 @@ def retain_incident_recurrence_groups(self, names: set[str]) -> None: key: point for key, point in self._incident_recurrence.items() if key[0] in names } + def replace_service_health_scores(self, points: list[ServiceHealthScorePoint]) -> None: + """Atomically replace the per-bucket service-health composite. + + Whole-state replacement: buckets that disappear from + ``points`` (e.g. a service whose quality_points all dropped + below the configured ``quality_floor`` this refresh) leave + /metrics on the next scrape. The runner rebuilds the full + list from current snapshots after every group run; the + operator-configured ``aggregate_by`` decides the bucket key + so the dict shape is stable across calls. + + The bucket key sorts ``(label, value)`` pairs by label name + before tupling — operator-visible label values stay in their + configured order on the rendered metric, but the *dict key* + used to deduplicate is order-stable even when the producer + changes its label-insertion order. Without the sort, an + unrelated edit to the producer (extra labels injected between + aggregate_by dims, or a different Python build whose dict + order semantics shift) could silently fork buckets across + runs and operators would see the same bucket appear twice on + /metrics. + """ + new_state: dict[tuple[tuple[str, str], ...], ServiceHealthScorePoint] = {} + for point in points: + # Bucket key intentionally excludes the documentation + # labels (``aggregator`` / ``weight_by`` / ``series_count``) + # since they're consequences of the config and don't + # discriminate buckets. Sorting by label name keeps the + # tuple stable across producer-side insertion orders. + key = tuple( + sorted( + (k, v) + for k, v in point.labels.items() + if k not in ("aggregator", "weight_by", "series_count") + ) + ) + new_state[key] = point + if not new_state and not self._service_health_scores: + return + with self._lock: + self._service_health_scores = new_state + + def _service_health_state(self) -> list[ServiceHealthScorePoint]: + with self._lock: + return list(self._service_health_scores.values()) + + def replace_resource_sizing_recommendations(self, points: list[ResourceSizingPoint]) -> None: + """Atomically replace the right-sizing recommendation set. + + Whole-state replacement: a workload entry whose forecast + dropped below the quality floor this refresh (and therefore + produced no recommendation) leaves /metrics on the next + scrape. The runner rebuilds the full list from current + snapshots after every group run. + """ + new_state: dict[tuple[str, str], ResourceSizingPoint] = {} + for point in points: + workload = point.labels.get("workload", "") + kind = point.labels.get("kind", "") + new_state[(workload, kind)] = point + if not new_state and not self._resource_sizing_recommendations: + return + with self._lock: + self._resource_sizing_recommendations = new_state + + def _resource_sizing_state(self) -> list[ResourceSizingPoint]: + with self._lock: + return list(self._resource_sizing_recommendations.values()) + + def increment_drift_explained(self, *, group: str, query: str, primary_suspect: str) -> None: + """Bump the ``forecast_drift_explained_total`` counter for one event.""" + with self._lock: + key = (group, query, primary_suspect) + self._drift_explained[key] = self._drift_explained.get(key, 0) + 1 + + def _drift_explained_state(self) -> dict[tuple[str, str, str], int]: + with self._lock: + return dict(self._drift_explained) + + def retain_drift_explained_queries(self, live_pairs: set[tuple[str, str]]) -> None: + """Drop counter rows for (group, query) pairs removed by a reload.""" + with self._lock: + self._drift_explained = { + key: count + for key, count in self._drift_explained.items() + if (key[0], key[1]) in live_pairs + } + + def record_series_fit_success( + self, + *, + group: str, + query_id: str, + model_name: str, + series_key: str, + labels: dict[str, str], + ) -> None: + """Record a successful per-(series, model) fit for staleness tracking. + + The runner calls this once per emitted model per series after a + successful fit. ``series_key`` is the + :func:`_series_key_from_labels` digest used elsewhere in the + runner to address per-series state; ``labels`` is the full + labelset (source labels + ``id`` + ``group`` + ``model``) the + render path emits with the gauge — captured once at first record + so the runner doesn't have to re-thread it on every refresh. + + Failed fits never call this method, so the previous timestamp + lingers and the next ``render`` returns a growing staleness + value — the alert ``ForecastSeriesStale`` keys on that + signal. + """ + with self._lock: + self._series_last_success[(group, query_id, model_name, series_key)] = ( + time.time(), + dict(labels), + ) + + def retain_staleness_groups(self, names: set[str]) -> None: + """Drop staleness state for groups removed by a config reload. + + Without this an operator who renames a group would see the + old group's series stay on /metrics with a forever-growing + staleness value because no fit will ever reset their clock. + """ + with self._lock: + self._series_last_success = { + key: value for key, value in self._series_last_success.items() if key[0] in names + } + + def retain_staleness_queries(self, live_pairs: set[tuple[str, str]]) -> None: + """Drop staleness state for (group, query) pairs no longer in config. + + Mirrors :meth:`retain_incident_recurrence_queries`. Per-query + cleanup so a reload that drops a single query from a + still-present group also clears the staleness rows that query + contributed; the group-level :meth:`retain_staleness_groups` + only handles whole-group removal. + """ + with self._lock: + self._series_last_success = { + key: value + for key, value in self._series_last_success.items() + if (key[0], key[1]) in live_pairs + } + + def prune_stale_staleness(self, max_age_seconds: float) -> int: + """Drop staleness rows whose timestamp is older than ``max_age_seconds``. + + Catches the two failure modes the group/query retention helpers + don't cover: + + * **Model removal** — a model dropped from ``defaults.models`` + never refreshes its clock, so without a sweep the dropped + model's rows would fire ``ForecastSeriesStale`` forever. + * **Ephemeral series** — series whose label set rotates + (per-pod metrics, deployment-id labels) would accumulate + unbounded rows; the runner has no way to distinguish "gone" + from "temporarily missing" so we sweep by age rather than by + explicit liveness. + + The threshold is chosen by the caller so the runner can scale + it with ``refresh_interval`` rather than baking in a guess. A + sensible default is `max(1d, 24x refresh_interval)` — well + past the ``ForecastSeriesStale`` alert horizon (4x refresh) + so the operator gets multiple alert cycles before the row + disappears, then the row is pruned silently. + + Returns the number of pruned rows for log telemetry — callers + can flag a pruning storm as a config-change side effect rather + than a routine event. + """ + if max_age_seconds <= 0: + return 0 + cutoff = time.time() - max_age_seconds + with self._lock: + survivors = { + key: value for key, value in self._series_last_success.items() if value[0] >= cutoff + } + pruned = len(self._series_last_success) - len(survivors) + if pruned: + self._series_last_success = survivors + return pruned + + def _series_last_success_state( + self, + ) -> dict[tuple[str, str, str, str], tuple[float, dict[str, str]]]: + with self._lock: + return dict(self._series_last_success) + + def record_fit_outcome(self, *, group: str, model: str, success: bool) -> None: + """Record one per-(group, model) fit attempt for the rolling-success ratio. + + Called by the runner after every fit (successful or not). The tracker + owns its own lock so we don't take the exporter lock on the hot path — + per-series fits in the same group run concurrently against the shared + executor and would otherwise contend on every outcome. + """ + self._fit_outcome_tracker.record(group=group, model=model, success=success) + + def record_regressor_stability( + self, + *, + group: str, + query: str, + regressor_id: str, + values: list[float], + ) -> None: + """Record one regressor's recent value series for the stability score.""" + self._regressor_stability_tracker.record( + group=group, query=query, regressor_id=regressor_id, values=values + ) + + def add_group_cpu_seconds(self, group: str, seconds: float) -> None: + """Add to the per-group cumulative CPU counter.""" + self._group_resource_tracker.add_cpu(group, seconds) + + def set_group_memory_bytes(self, group: str, memory_bytes: float) -> None: + """Set the per-group memory gauge to ``memory_bytes``.""" + self._group_resource_tracker.set_memory(group, memory_bytes) + + def retain_self_observability_groups(self, names: set[str]) -> None: + """Drop per-group self-observability state for groups removed by a reload.""" + self._fit_outcome_tracker.retain_groups(names) + self._regressor_stability_tracker.retain_groups(names) + self._group_resource_tracker.retain_groups(names) + + def retain_fit_outcome_models(self, live_models: set[str]) -> None: + """Drop fit-success-ratio buckets for models no longer in the config. + + Forwarded to :class:`FitOutcomeTracker.retain_models` — needed so + a reload that drops a model from ``defaults.models`` clears its + success-ratio rows from /metrics instead of leaving the alert + bound to a frozen-in-time number. + """ + self._fit_outcome_tracker.retain_models(live_models) + + def retain_regressor_stability_queries(self, live_pairs: set[tuple[str, str]]) -> None: + """Drop regressor stability rows for (group, query) pairs removed by a reload.""" + self._regressor_stability_tracker.retain_queries(live_pairs) + def retain_incident_recurrence_queries(self, live_pairs: set[tuple[str, str]]) -> None: """Drop heatmap rows for (group, query) pairs no longer in config. @@ -1047,6 +1453,7 @@ def collect(self) -> Iterator[GaugeMetricFamily | CounterMetricFamily]: # noqa: ) deploy_risk_state = self._exporter._deploy_risk_state() incident_recurrence_state = self._exporter._incident_recurrence_state() + staleness_state = self._exporter._series_last_success_state() yield from _forecast_families(groups) yield from _conditional_forecast_families(groups) yield from _conditional_deviation_families(groups) @@ -1087,6 +1494,18 @@ def collect(self) -> Iterator[GaugeMetricFamily | CounterMetricFamily]: # noqa: yield from _conditional_aliased_families(conditional_aliased_state) yield from _deploy_risk_families(deploy_risk_state) yield from _incident_recurrence_families(incident_recurrence_state) + yield from _staleness_families(staleness_state) + yield from _fit_success_ratio_families(self._exporter._fit_outcome_tracker.render()) + yield from _regressor_stability_families( + self._exporter._regressor_stability_tracker.render() + ) + yield from _group_resource_families(self._exporter._group_resource_tracker.render()) + yield from _revision_families(groups) + yield from _model_disagreement_families(groups) + yield from _threshold_action_cost_families(groups) + yield from _service_health_score_families(self._exporter._service_health_state()) + yield from _resource_sizing_families(self._exporter._resource_sizing_state()) + yield from _drift_explained_families(self._exporter._drift_explained_state()) def _emit_gauge_family( @@ -1563,6 +1982,21 @@ def _operational_families( # noqa: PLR0912 "Unix timestamp of the most recent successful run for the group.", labels=["group"], ) + # ``forecast_snapshot_age_seconds`` is the same signal as + # ``time() - forecast_last_run_timestamp_seconds`` but pre-computed + # server-side so dashboards and degraded-mode alerts + # (``ForecastSnapshotStale``) can read a single gauge instead of + # subtracting time(). For groups that have never produced a snapshot + # the value is 0 — alert authors should gate on the timestamp + # being non-zero as well (or on `forecast_last_run_timestamp_seconds + # > 0`) before treating a zero age as healthy. + snapshot_age = GaugeMetricFamily( + "forecast_snapshot_age_seconds", + "Seconds since the most recent successful run for the group " + "(``time() - forecast_last_run_timestamp_seconds``); 0 when the " + "group has never produced a snapshot.", + labels=["group"], + ) # Group-aggregate gauges. Stable schema (single ``group`` label) — # dashboards relying on these built before per-query refresh intervals # existed keep working unchanged. Per-query breakdowns are emitted as @@ -1596,8 +2030,11 @@ def _operational_families( # noqa: PLR0912 ) have_per_query_duration = False have_per_query_series_count = False + now = time.time() for snap in groups: last_run.add_metric([snap.group], snap.last_run_timestamp) + age = (now - snap.last_run_timestamp) if snap.last_run_timestamp > 0 else 0.0 + snapshot_age.add_metric([snap.group], max(0.0, age)) duration.add_metric([snap.group], snap.run_duration_seconds) series_count.add_metric([snap.group], float(snap.series_count)) for query_id, dur in snap.run_duration_per_query.items(): @@ -1607,6 +2044,7 @@ def _operational_families( # noqa: PLR0912 per_query_series_count.add_metric([snap.group, query_id], float(count)) have_per_query_series_count = True yield last_run + yield snapshot_age yield duration yield series_count if have_per_query_duration: @@ -1737,10 +2175,23 @@ def _config_hash_families( def _remote_write_families( state: dict[tuple[str, str], int], ) -> Iterator[CounterMetricFamily]: - """Render the ``forecast_remote_write_failures_total`` counter. + """Render the per-(group, reason) and reason-aggregated sink-failure counters. Always emitted (even when empty) so dashboards can display "0 errors" rather than 404'ing the panel before the first failure occurs. + + Two families ship side by side: + + * ``forecast_remote_write_failures_total{group, reason}`` — the + per-group breakdown used by dashboards that want to see which + group's pushes are failing. + * ``forecast_sink_write_failures_total{reason}`` — the reason-only + roll-up the degraded-mode playbook + (``docs/operations/degraded-modes.md``) recommends alerting on + (``ForecastSinkWriteFailing``). The aggregate is a more honest + "is the sink up?" signal than the per-group counter, which fires + separately for every group that pushed while the sink was + unreachable. """ counter = CounterMetricFamily( "forecast_remote_write_failures", @@ -1750,6 +2201,20 @@ def _remote_write_families( for (group, reason), count in state.items(): counter.add_metric([group, reason], float(count)) yield counter + sink_counter = CounterMetricFamily( + "forecast_sink_write_failures", + "Total number of sink (remote_write) push attempts that failed, by reason " + "(timeout | http_error | unexpected). Reason-only roll-up of " + "``forecast_remote_write_failures_total`` — alert on this when you want " + "a single 'is the sink up?' signal rather than one alert per group.", + labels=["reason"], + ) + per_reason: dict[str, int] = {} + for (_group, reason), count in state.items(): + per_reason[reason] = per_reason.get(reason, 0) + count + for reason, count in per_reason.items(): + sink_counter.add_metric([reason], float(count)) + yield sink_counter def _remote_write_retries_families( @@ -2022,6 +2487,7 @@ def _preprocess_families( yield from _outliers_family(groups) yield from _change_points_family(groups) yield from _effective_lookback_family(groups) + yield from _blacklisted_samples_family(groups) def _gaps_filled_family(groups: list[GroupSnapshot]) -> Iterator[CounterMetricFamily]: @@ -2106,6 +2572,34 @@ def _effective_lookback_family(groups: list[GroupSnapshot]) -> Iterator[GaugeMet yield family +def _blacklisted_samples_family( + groups: list[GroupSnapshot], +) -> Iterator[CounterMetricFamily]: + """Render ``forecast_input_blacklisted_samples_total{group, query, reason}``. + + Counts samples removed by ``preprocess.blacklist_windows`` per + (group, query, reason). The reason value defaults to the empty string + when the operator did not configure one. Only emitted when at least + one group has produced a non-zero count so a config without any + blacklist windows does not carry an empty family on /metrics. + """ + family = CounterMetricFamily( + "forecast_input_blacklisted_samples", + "Lookback samples dropped by an operator-configured " + "``preprocess.blacklist_windows`` entry. Useful for confirming " + "that an annual-recurring window actually fired for the most " + "recent year of the lookback.", + labels=["group", "query", "reason"], + ) + have = False + for snap in groups: + for (query_id, reason), count in snap.blacklisted_samples_cumulative.items(): + family.add_metric([snap.group, query_id, reason], float(count)) + have = True + if have: + yield family + + def _emission_clipped_families( groups: list[GroupSnapshot], ) -> Iterator[CounterMetricFamily]: @@ -2279,3 +2773,346 @@ def _incident_recurrence_families( value_fn=lambda p: p.score, skip_fn=lambda p: not math.isfinite(p.score), ) + + +@dataclass(frozen=True) +class _StalenessPoint: + """Adapter so :func:`_emit_gauge_family` can render staleness state. + + Built from the captured ``(timestamp, labels)`` tuple at render + time, with ``value`` already converted to the elapsed-seconds + gauge value. Avoids hand-rolling the label-union recipe that + every other gauge family in this module already shares. + """ + + labels: dict[str, str] + value: float + + +def _staleness_families( + state: dict[tuple[str, str, str, str], tuple[float, dict[str, str]]], +) -> Iterator[GaugeMetricFamily]: + """Render ``forecast_data_staleness_seconds{id, group, model, ...}``. + + Per-(series, model) gauge of seconds since the last successful + forecast emission. The per-series complement to the group-level + ``forecast_last_run_timestamp_seconds``: a single series whose + model crashed silently goes undetected on the group gauge (the + other series keep the group's last-run timestamp ticking) but + its staleness clock visibly grows here, so an alert on + ``forecast_data_staleness_seconds > 4 * `` + catches the silent failure. + + Emitted only after at least one successful fit has been recorded + so /metrics stays quiet on a fresh deployment before the first + refresh cycle completes. Cardinality is bounded by + ``N_series * N_models`` per opted-in query — already part of the + forecast emission budget. + """ + if not state: + return + now = time.time() + points = [ + _StalenessPoint(labels=labels, value=max(0.0, now - timestamp)) + for timestamp, labels in state.values() + ] + yield from _emit_gauge_family( + points, + "forecast_data_staleness_seconds", + "Seconds elapsed since the last successful forecast emission for " + "the (series, model) tuple. The per-series complement to the " + "group-level ``forecast_last_run_timestamp_seconds`` — a single " + "series whose model crashed silently keeps the group's last-run " + "timestamp ticking via the other series, but its staleness clock " + "visibly grows here. Failed fits leave the clock running; " + "successful fits reset it to zero.", + value_fn=lambda p: p.value, + ) + + +def _fit_success_ratio_families( + points: list[Any], +) -> Iterator[GaugeMetricFamily]: + """Render ``forecast_model_fit_success_ratio{group, model, window}``. + + Emitted only when at least one observation has been recorded for the + (group, model) tuple inside the asked-for window — empty windows + short to absent rather than 0/0 noise so a fresh deployment doesn't + light up the model-health alert before any fit has run. + """ + if not points: + return + family = GaugeMetricFamily( + "forecast_model_fit_success_ratio", + "Fraction of attempted fits that completed within safety.fit_timeout " + "without raising, computed over a rolling window. 1.0 = every fit " + "succeeded in the window; 0.0 = every fit failed. Per (group, model, " + "window) tuple.", + labels=["group", "model", "window"], + ) + for point in points: + if not math.isfinite(point.ratio): + continue + family.add_metric([point.group, point.model, point.window_label], float(point.ratio)) + yield family + + +def _regressor_stability_families( + points: list[Any], +) -> Iterator[GaugeMetricFamily]: + """Render ``forecast_regressor_stability_score{group, query, regressor_id}``. + + Higher is more stable: 1.0 means the regressor's recent tail matches + its lookback head; 0.0 means it has drifted past the saturation + threshold. Non-finite scores (degenerate input series) are dropped + so dashboards don't see NaN markers. + """ + if not points: + return + family = GaugeMetricFamily( + "forecast_regressor_stability_score", + "How much a configured exogenous regressor's recent values differ " + "from its lookback distribution. 1.0 = the recent tail matches the " + "lookback head; 0.0 = it has drifted past the saturation threshold. " + "Per (group, query, regressor_id) tuple.", + labels=["group", "query", "regressor_id"], + ) + have = False + for point in points: + if not math.isfinite(point.score): + continue + family.add_metric([point.group, point.query, point.regressor_id], float(point.score)) + have = True + if have: + yield family + + +def _revision_families( + groups: list[GroupSnapshot], +) -> Iterator[GaugeMetricFamily]: + """Render ``forecast_revision_magnitude{id, group, model, horizon, ...}``. + + Unsigned per-(series, model, horizon) gauge: how much the predicted + value at this horizon moved between fits, normalised by the prior + forecast's typical scale. NaN samples (no prior fit cached) are + dropped — they're the bootstrap state, not a meaningful "magnitude". + """ + points = [p for snap in groups for p in snap.revision_points] + yield from _emit_gauge_family( + points, + "forecast_revision_magnitude", + "Magnitude of change in the predicted central value at the given " + "horizon between the current fit and the previous fit, normalised " + "by ``max(|previous|, scale_floor)``. Unsigned. Useful for " + "detecting an over-reactive model or a real regime change before " + "downstream alerts notice the shift.", + value_fn=lambda p: p.magnitude, + skip_fn=lambda p: not math.isfinite(p.magnitude), + ) + + +def _model_disagreement_families( + groups: list[GroupSnapshot], +) -> Iterator[GaugeMetricFamily]: + """Render ``forecast_model_disagreement{id, group, horizon, ...}``. + + Coefficient of variation across the configured models' central + forecasts at the same horizon — high values mean no model is + confident about this future. Only emitted in explicit mode with + 2+ models contributing; auto-select suppresses the metric because + only the winner is emitted. + """ + points = [p for snap in groups for p in snap.model_disagreement_points] + yield from _emit_gauge_family( + points, + "forecast_model_disagreement", + "Coefficient of variation across the central forecasts of every " + "configured model at the given horizon. 0 = perfect agreement; " + "higher values mean the configured models disagree more about the " + "future — a useful 'should I switch to auto-select?' signal.", + value_fn=lambda p: p.score, + skip_fn=lambda p: not math.isfinite(p.score), + ) + + +def _drift_explained_families( + state: dict[tuple[str, str, str], int], +) -> Iterator[CounterMetricFamily]: + """Render ``forecast_drift_explained_total{group, query, primary_suspect}``. + + Bumped once per drift threshold-crossing event. The + ``primary_suspect`` label is the regressor id the + change-correlation explainer identified (high/medium correlation + bands) or — falling back when correlation is + ``insufficient_data`` — the highest-value regressor at the drift + moment. ``"unknown"`` covers the case where no regressors are + configured or none produced a finite candidate. + + Emitted only after at least one drift event has fired so /metrics + stays quiet for deployments that never trip the drift threshold. + """ + if not state: + return + counter = CounterMetricFamily( + "forecast_drift_explained", + "Number of drift threshold-crossing events explained by the " + "change-correlation suspect for (group, query, primary_suspect). " + "primary_suspect prefers the rolling-window Pearson correlation " + "candidate when correlation confidence is high or medium; falls " + "back to the highest-value regressor at the drift moment " + "otherwise. ``unknown`` covers no-suspect cases.", + labels=["group", "query", "primary_suspect"], + ) + for (group, query, suspect), count in state.items(): + counter.add_metric([group, query, suspect], float(count)) + yield counter + + +def _resource_sizing_families( + points: list[ResourceSizingPoint], +) -> Iterator[GaugeMetricFamily]: + """Render the three right-sizing recommendation gauges. + + Three sibling metrics share the (workload, kind) label set so + operators can join them in PromQL: + + * ``forecast_recommended_resource_size{..., confidence}`` — + the recommended limit/request value in the metric's native + units (CPU cores, memory bytes, ...). + * ``forecast_recommended_resource_current{...}`` — the current + configured limit/request, emitted only when the workload's + ``current_promql`` resolved to a finite number. + * ``forecast_recommended_resource_savings_pct{..., confidence}`` — + ``(current - recommended) / current * 100``, emitted only + when both sides are available. Negative values mean upsize, + not downsize. + """ + # The recommended gauge always emits — that's the primary signal. + yield from _emit_gauge_family( + points, + "forecast_recommended_resource_size", + "Right-sized resource limit/request derived from the forecast peak. " + "Sample value is in the metric's native units (CPU cores, memory " + "bytes). The ``confidence`` label is derived from " + "``forecast_quality_score`` and gates how much to trust the row.", + value_fn=lambda p: p.recommended, + skip_fn=lambda p: not math.isfinite(p.recommended), + ) + # ``current`` only emits when ``current_promql`` resolved. + current_points = [p for p in points if p.current is not None] + yield from _emit_gauge_family( + current_points, + "forecast_recommended_resource_current", + "Current configured limit/request for the workload, fetched via " + "the operator-supplied ``current_promql``. Sample value is in " + "the metric's native units. Emitted only when ``current_promql`` " + "was configured.", + value_fn=lambda p: float(p.current) if p.current is not None else float("nan"), + skip_fn=lambda p: p.current is None or not math.isfinite(p.current), + ) + savings_points = [p for p in points if p.savings_pct is not None] + yield from _emit_gauge_family( + savings_points, + "forecast_recommended_resource_savings_pct", + "Expected savings percentage: ``(current - recommended) / current " + "* 100``. Positive values mean downsize is safe; negative values " + "mean the predicted peak exceeds the current limit and the " + "workload needs upsizing.", + value_fn=lambda p: float(p.savings_pct) if p.savings_pct is not None else float("nan"), + skip_fn=lambda p: p.savings_pct is None or not math.isfinite(p.savings_pct), + ) + + +def _service_health_score_families( + points: list[ServiceHealthScorePoint], +) -> Iterator[GaugeMetricFamily]: + """Render ``forecast_service_health_score{, ...}``. + + Single global family — the ``aggregate_by`` dimensions ride the + label set so one Grafana panel can show every configured bucket + (per-service, per-environment, ...) without a second query. The + sample value is the aggregated quality score in ``[0, 1]``. + ``aggregator``, ``weight_by``, and ``series_count`` ride as + documentation labels so the operator can read which + interpretation produced the number alongside its bucket size. + """ + yield from _emit_gauge_family( + points, + "forecast_service_health_score", + "Per-bucket aggregated forecast_quality_score in [0, 1]. The " + "aggregate_by labels (e.g. service, environment) ride the " + "label set so one Grafana panel can show every configured " + "bucket. ``aggregator``, ``weight_by``, and ``series_count`` " + "ride as documentation labels.", + value_fn=lambda p: p.score, + skip_fn=lambda p: not math.isfinite(p.score), + ) + + +def _threshold_action_cost_families( + groups: list[GroupSnapshot], +) -> Iterator[GaugeMetricFamily]: + """Render ``forecast_threshold_action_cost{id, group, name, type}``. + + Static per-(group, query, threshold) gauge carrying the + operator-configured cost-of-action. Joined against + ``forecast_time_to_threshold_seconds`` so dashboards can rank + "soon-and-expensive" capacity events ahead of "soon-and-cheap" + ones. The ``type`` label is free text (``usd``, ``eur``, + ``priority``, ``engineer_hours``, ...) so non-monetary cost + framings coexist with currency ones in a single family. + """ + points = [p for snap in groups for p in snap.threshold_action_cost_points] + yield from _emit_gauge_family( + points, + "forecast_threshold_action_cost", + "Operator-configured cost-of-action for a capacity threshold. " + "The ``type`` label is free text (``usd``, ``eur``, " + "``engineer_hours``, ...). Join against " + "``forecast_time_to_threshold_seconds`` on (id, group, name) " + "to rank capacity ETAs by cost-to-ignore.", + value_fn=lambda p: p.value, + skip_fn=lambda p: not math.isfinite(p.value), + ) + + +def _group_resource_families( + points: list[Any], +) -> Iterator[GaugeMetricFamily | CounterMetricFamily]: + """Render ``forecast_group_memory_bytes`` + ``forecast_group_cpu_seconds_total``. + + Memory is a best-effort snapshot of the per-group dataframes + the + exporter's per-group snapshot footprint, computed via + ``sys.getsizeof`` at the end of every group run. CPU is the + cumulative sum of per-fit wall-clock seconds reported by the + runner's ``fit_durations`` accumulator. + """ + if not points: + return + memory = GaugeMetricFamily( + "forecast_group_memory_bytes", + "Best-effort per-group memory footprint (bytes). Counts the size of " + "the per-group snapshot held by the exporter plus the in-flight " + "per-fit dataframes the runner retains across the per-series loop.", + labels=["group"], + ) + cpu = CounterMetricFamily( + "forecast_group_cpu_seconds", + "Cumulative wall-clock seconds spent fitting models attributable to " + "this group, summed across every per-model fit the runner has " + "executed since boot.", + labels=["group"], + ) + have_memory = False + have_cpu = False + for point in points: + if math.isfinite(point.memory_bytes) and point.memory_bytes > 0: + memory.add_metric([point.group], float(point.memory_bytes)) + have_memory = True + if math.isfinite(point.cpu_seconds_total) and point.cpu_seconds_total > 0: + cpu.add_metric([point.group], float(point.cpu_seconds_total)) + have_cpu = True + if have_memory: + yield memory + if have_cpu: + yield cpu diff --git a/forecaster/src/promforecast/health.py b/forecaster/src/promforecast/health.py new file mode 100644 index 0000000..cb55b35 --- /dev/null +++ b/forecaster/src/promforecast/health.py @@ -0,0 +1,125 @@ +"""Service-level forecast health composite. + +Aggregates ``forecast_quality_score`` rows across the whole fleet into a +single number per ``(service, environment, ...)`` bucket. The result lands +as ``forecast_service_health_score`` so executives and platform teams can +report on the reliability of the *prediction infrastructure* without +staring at 30 individual series. + +The aggregation is dimension-driven by the operator's +``governance.health_score.aggregate_by`` list — any subset of the +source-side labels carried on every quality_point qualifies. Series whose +label set is missing any of the requested keys are dropped, so a single +unlabelled forecast can't silently inflate or deflate every bucket. + +Pure functions: the runner feeds in a flat list of quality points (read +from the exporter's per-group snapshots) plus the operator's +``HealthScoreConfig`` and gets back a flat list of aggregated rows. The +exporter then renders them under the global metric name. +""" + +from __future__ import annotations + +import math +from dataclasses import dataclass +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from collections.abc import Iterable + + from .config import HealthScoreConfig + from .exporter import QualityPoint + + +@dataclass(frozen=True) +class ServiceHealthScore: + """One aggregated row destined for ``forecast_service_health_score``. + + ``labels`` carries the ``aggregate_by`` dimension values (e.g. + ``service="checkout"``, ``environment="prod"``) plus the + ``aggregator`` and ``weight_by`` documentation labels so a Grafana + panel can record which interpretation was in effect. ``score`` is + the aggregated value in ``[0, 1]``. + """ + + labels: dict[str, str] + score: float + series_count: int + + +def compute_service_health_scores( + quality_points: Iterable[QualityPoint], + config: HealthScoreConfig, +) -> list[ServiceHealthScore]: + """Aggregate per-series quality scores into per-dim composites. + + * ``config.aggregate_by`` enumerates the discriminator labels. A + quality_point missing *any* of these labels is silently dropped + — without the drop one un-labelled forecast would land in + every bucket and skew the report. + * ``config.quality_floor`` (when set) drops scores below the floor + before aggregation, so cold-start / low-quality emissions + don't pull a service's composite down below the alert gate the + operator already set elsewhere. + * ``config.aggregator`` picks the summary statistic; only + ``weighted_mean`` ships today. With per-series quality_points + each row contributes weight ``1``, so the result is the + arithmetic mean of the bucket. ``config.weight_by`` is + forwarded as a documentation label only — the math is the same + across both options until a future per-group aggregated emission + shape exists. + * Non-finite quality_points are dropped before aggregation so a + single NaN-shaped backtest can't poison the bucket. + + Output ordering is deterministic — buckets sort by their tuple of + label values so successive runs publish a stable row sequence and + Grafana series colours stay pinned across refreshes. + """ + buckets: dict[tuple[str, ...], list[float]] = {} + for point in quality_points: + if not math.isfinite(point.score): + continue + if config.quality_floor is not None and point.score < config.quality_floor: + continue + key = _bucket_key(point.labels, config.aggregate_by) + if key is None: + continue + buckets.setdefault(key, []).append(float(point.score)) + out: list[ServiceHealthScore] = [] + for key in sorted(buckets): + scores = buckets[key] + if not scores: + continue + labels = {dim: value for dim, value in zip(config.aggregate_by, key, strict=True)} + labels["aggregator"] = config.aggregator + labels["weight_by"] = config.weight_by + # ``series_count`` rides as a label so a dashboard can read the + # bucket size alongside the aggregated value. A separate count + # gauge would force callers to join on the same labels and + # double the storage footprint for no extra signal. + labels["series_count"] = str(len(scores)) + out.append( + ServiceHealthScore( + labels=labels, + score=sum(scores) / len(scores), + series_count=len(scores), + ) + ) + return out + + +def _bucket_key(labels: dict[str, str], aggregate_by: list[str]) -> tuple[str, ...] | None: + """Project ``labels`` onto the ``aggregate_by`` tuple, or None if any are missing. + + Empty-string label values (which happen when a label is technically + present but rendered as ``label=""``) are treated as missing — a + bucket like ``service=""`` is almost certainly a labelling bug, and + surfacing it as a real row would dilute every meaningful bucket. + """ + out: list[str] = [] + for dim in aggregate_by: + value = labels.get(dim, "") + if not value: + return None + out.append(value) + return tuple(out) diff --git a/forecaster/src/promforecast/inspect.py b/forecaster/src/promforecast/inspect.py new file mode 100644 index 0000000..4898b98 --- /dev/null +++ b/forecaster/src/promforecast/inspect.py @@ -0,0 +1,689 @@ +"""Forecast inspector — debug "why is this one series weird?". + +Pulls a single (series, model) tuple from the configured TSDB, runs +the configured fit pipeline exactly as the live forecaster would, and +prints the inputs, the fit, the forecast curve, the per-component +decomposition (when available), and the most recent backtest scores. +Optionally surfaces the per-fold backtest predictions vs. actuals so +"this fit backtests well but predicts badly" becomes a concrete two- +column comparison instead of a single summary MAPE. + +The CLI talks to VM directly via the same :class:`PromSource` the +runner uses — no new HTTP surface, no shared scheduler state, no +risk of starving scheduled work. Use it from a bastion / debugging +pod or locally with a port-forwarded TSDB. + +The output is a compact text table by default; ``--json`` switches +to a machine-friendly envelope for piping into ``jq`` or saving as +a CI artifact alongside a flaky forecast for later analysis. +""" + +from __future__ import annotations + +import asyncio +import math +import sys +from collections.abc import Iterable +from dataclasses import dataclass, field +from datetime import UTC, datetime, timedelta +from pathlib import Path +from typing import IO, TYPE_CHECKING, Any + +import structlog + +from . import accuracy as accuracy_mod +from . import config as config_module +from . import models as model_registry +from . import preprocess as preprocess_mod +from .config import Config, effective_emission_safeguards, effective_season_length, effective_step +from .preview import _pandas_freq, find_query +from .runner import _apply_safeguards_to_forecast_frame +from .source import PromSource, SeriesFrame + +if TYPE_CHECKING: + import pandas as pd + +logger = structlog.get_logger(__name__) + + +# Default number of forecast curve rows to render in the table when +# ``--json`` is off. The full curve is always available via ``--json``; +# the table cap keeps the human view scannable on an 80-column terminal. +_TABLE_HEAD_ROWS = 10 + + +class InspectError(RuntimeError): + """Raised for any operator-visible inspector failure. + + Carries an integer exit code so :func:`run_inspect` can hand it + straight to ``sys.exit`` without a second try/except. + """ + + def __init__(self, message: str, *, exit_code: int = 1) -> None: + super().__init__(message) + self.exit_code = exit_code + + +@dataclass(frozen=True) +class InputStats: + """Summary statistics for the inspected series's lookback window.""" + + point_count: int + gap_count: int + largest_gap_seconds: float + lookback_start: str + lookback_end: str + + +@dataclass(frozen=True) +class BacktestFoldRow: + """One per-step row from the backtest hold-out comparison. + + Carries the ``ds`` timestamp, the held-out ``actual``, and the + model's ``yhat`` / lower / upper. Bands are dict-shaped so a + multi-level config emits them all without changing the row's + schema; missing levels are simply absent rather than NaN. + """ + + ds: str + actual: float + yhat: float + bands: dict[str, dict[str, float]] + + +@dataclass(frozen=True) +class BacktestSummary: + """Compact per-fold backtest summary for the inspector output.""" + + model: str + n_holdout: int + mape: float + mase: float + rows: list[BacktestFoldRow] = field(default_factory=list) + + +@dataclass(frozen=True) +class InspectReport: + """Full inspector report — every section the operator may want.""" + + query_id: str + group: str + model: str + series_labels: dict[str, str] + step_seconds: int + season_length: int + horizon_steps: int + input_stats: InputStats + forecast_timestamps: list[str] + forecast_yhat: list[float] + forecast_bands: dict[str, dict[str, list[float]]] + preprocessing: dict[str, Any] + backtest: BacktestSummary | None = None + + def to_dict(self) -> dict[str, Any]: + out: dict[str, Any] = { + "query_id": self.query_id, + "group": self.group, + "model": self.model, + "series_labels": dict(self.series_labels), + "step_seconds": self.step_seconds, + "season_length": self.season_length, + "horizon_steps": self.horizon_steps, + "input_stats": { + "point_count": self.input_stats.point_count, + "gap_count": self.input_stats.gap_count, + "largest_gap_seconds": self.input_stats.largest_gap_seconds, + "lookback_start": self.input_stats.lookback_start, + "lookback_end": self.input_stats.lookback_end, + }, + "forecast": { + "timestamps": list(self.forecast_timestamps), + "yhat": list(self.forecast_yhat), + "bands": { + level: { + "lower": list(payload["lower"]), + "upper": list(payload["upper"]), + } + for level, payload in self.forecast_bands.items() + }, + }, + "preprocessing": dict(self.preprocessing), + } + if self.backtest is not None: + out["backtest"] = { + "model": self.backtest.model, + "n_holdout": self.backtest.n_holdout, + "mape": _json_value(self.backtest.mape), + "mase": _json_value(self.backtest.mase), + "rows": [ + { + "ds": row.ds, + "actual": _json_value(row.actual), + "yhat": _json_value(row.yhat), + "bands": { + level: { + "lower": _json_value(values["lower"]), + "upper": _json_value(values["upper"]), + } + for level, values in row.bands.items() + }, + } + for row in self.backtest.rows + ], + } + return out + + +def run_inspect( + *, + config_path: Path, + query_id: str, + series_filter: dict[str, str] | None, + horizon_seconds: float | None, + lookback_seconds: float | None, + model: str | None, + backtest: bool, + json_output: bool, + out: IO[str] | None = None, +) -> int: + """Top-level inspector entrypoint called by ``main.cli``. + + Returns a process exit code: + + * ``0`` on success; + * ``1`` on inspector-level failure (unknown query, ambiguous + series, fit error, etc.); + * ``2`` on datasource / I/O failure. + """ + stream: IO[str] = out if out is not None else sys.stdout + + try: + cfg = config_module.load(config_path) + except FileNotFoundError: + sys.stderr.write(f"error: config not found: {config_path}\n") + return 2 + except Exception as exc: + sys.stderr.write(f"error: config validation failed: {exc}\n") + return 1 + + try: + report = asyncio.run( + _build_report( + config=cfg, + query_id=query_id, + series_filter=series_filter, + horizon_seconds=horizon_seconds, + lookback_seconds=lookback_seconds, + model_override=model, + with_backtest=backtest, + ) + ) + except InspectError as exc: + sys.stderr.write(f"error: {exc}\n") + return exc.exit_code + except Exception as exc: + sys.stderr.write(f"error: unexpected inspector failure: {exc}\n") + return 2 + + if json_output: + import json # noqa: PLC0415 + + stream.write(json.dumps(report.to_dict(), indent=2) + "\n") + else: + stream.write(render_text(report)) + return 0 + + +async def _build_report( + *, + config: Config, + query_id: str, + series_filter: dict[str, str] | None, + horizon_seconds: float | None, + lookback_seconds: float | None, + model_override: str | None, + with_backtest: bool, +) -> InspectReport: + """Fetch + fit + (optionally) backtest, returning a populated report.""" + import pandas as pd # noqa: PLC0415 + + try: + query, group_name = find_query(config, query_id) + except Exception as exc: + raise InspectError(f"unknown query {query_id!r}: {exc}") from exc + + eff_step = effective_step(query, config.defaults) + eff_season_length = effective_season_length(query, config.defaults, step=eff_step) + lookback = ( + timedelta(seconds=lookback_seconds) + if lookback_seconds is not None + else config.defaults.lookback + ) + horizon_secs = ( + horizon_seconds if horizon_seconds is not None else config.defaults.horizon.total_seconds() + ) + horizon_steps = max(1, int(horizon_secs // max(1, eff_step.total_seconds()))) + levels = list(config.defaults.confidence_levels) + + source = PromSource( + url=config.datasource.url, + timeout_seconds=config.datasource.timeout.total_seconds(), + max_concurrency=config.datasource.max_concurrent_requests, + max_keepalive_connections=config.datasource.max_keepalive_connections, + ) + try: + end = datetime.now(tz=UTC) + start = end - lookback + step_seconds = max(1, int(eff_step.total_seconds())) + try: + frames = await asyncio.wait_for( + source.query_range(query.promql, start, end, step_seconds), + timeout=config.safety.query_timeout.total_seconds(), + ) + except TimeoutError as exc: + raise InspectError("query timed out", exit_code=2) from exc + except Exception as exc: + raise InspectError(f"query failed: {exc}", exit_code=2) from exc + + frame = _pick_frame(frames, series_filter) + + cleaned = preprocess_mod.preprocess_series( + timestamps=list(frame.timestamps), + values=list(frame.values), + step=eff_step, + config=config.groups[_group_index(config, group_name)].preprocess, + ) + if cleaned.dropped: + raise InspectError("series was dropped by the preprocessor (too many gaps)") + + df = pd.DataFrame( + { + "unique_id": ["s"] * len(cleaned.values), + "ds": pd.to_datetime(cleaned.timestamps), + "y": cleaned.values, + } + ) + + spec = _resolve_model_spec(config, model_override) + model = model_registry.build(spec.name, season_length=eff_season_length, params=spec.params) + + loop = asyncio.get_running_loop() + freq = _pandas_freq(step_seconds) + try: + forecast = await asyncio.wait_for( + loop.run_in_executor( + None, model.fit_predict, df, horizon_steps, freq, levels, None + ), + timeout=config.safety.fit_timeout.total_seconds(), + ) + except TimeoutError as exc: + raise InspectError("fit timed out", exit_code=2) from exc + except Exception as exc: + raise InspectError(f"fit failed: {exc}") from exc + + # Apply count-aware safeguards (``clip_non_negative`` / + # ``round_to_integer``) so the inspector mirrors what the live + # runner actually emits. Without this, a ``data_profile: count`` + # query inspected by an operator would show raw model output + # — possibly negative or fractional values — that the live + # exporter would have clipped or rounded. The inspector exists + # to debug "what the live forecaster would do right now"; the + # safeguards are part of that contract. + clip, rounder = effective_emission_safeguards(query) + if clip or rounder: + forecast = _apply_safeguards_to_forecast_frame( + forecast, levels=levels, clip=clip, rounder=rounder + ) + + backtest_summary = None + if with_backtest: + backtest_summary = await _run_backtest( + config=config, + model=model, + df=df, + freq=freq, + levels=levels, + model_name=spec.name, + season_length=eff_season_length, + step_seconds=step_seconds, + ) + + return InspectReport( + query_id=query.id, + group=group_name, + model=spec.name, + series_labels={k: v for k, v in frame.labels.items() if k != "__name__"}, + step_seconds=step_seconds, + season_length=eff_season_length, + horizon_steps=horizon_steps, + input_stats=InputStats( + point_count=len(cleaned.values), + gap_count=sum(cleaned.gaps_filled.values()), + largest_gap_seconds=cleaned.max_gap_seconds, + lookback_start=cleaned.timestamps[0].isoformat() if cleaned.timestamps else "", + lookback_end=cleaned.timestamps[-1].isoformat() if cleaned.timestamps else "", + ), + forecast_timestamps=[str(ts) for ts in forecast.get("ds", [])], + forecast_yhat=_column_as_floats(forecast, "yhat"), + forecast_bands=_extract_bands(forecast, levels), + preprocessing={ + "gaps_filled": dict(cleaned.gaps_filled), + "outliers_replaced": cleaned.outliers_replaced, + "change_points": cleaned.change_points, + "imputed_fraction": cleaned.imputed_fraction, + "effective_lookback_seconds": cleaned.effective_lookback_seconds, + }, + backtest=backtest_summary, + ) + finally: + await source.aclose() + + +async def _run_backtest( + *, + config: Config, + model: Any, + df: pd.DataFrame, + freq: str, + levels: list[int], + model_name: str, + season_length: int, + step_seconds: int, +) -> BacktestSummary | None: + """Per-fold backtest summary — actuals vs. predicted with bands. + + Mirrors the live runner's hold-out sizing exactly: + ``max(h.steps for h in horizons)`` — large enough to score every + configured horizon from a single fit. Using the smallest horizon + here would silently produce different MAPE/MASE numbers from the + ones the live runner publishes, exactly the kind of "looks good + in backtest, looks wrong in production" confusion the inspector + exists to debug. + """ + horizons = accuracy_mod.horizons_from_durations( + list(config.defaults.accuracy.horizons), timedelta(seconds=step_seconds) + ) + if not horizons: + return None + horizon_steps = max(h.steps for h in horizons) + loop = asyncio.get_running_loop() + try: + outcome = await asyncio.wait_for( + loop.run_in_executor( + None, + lambda: accuracy_mod.backtest( + model=model, + df=df, + horizon_steps=horizon_steps, + freq=freq, + levels=levels, + ), + ), + timeout=config.safety.fit_timeout.total_seconds(), + ) + except TimeoutError: + return BacktestSummary(model=model_name, n_holdout=0, mape=math.nan, mase=math.nan, rows=[]) + except Exception: + return BacktestSummary(model=model_name, n_holdout=0, mape=math.nan, mase=math.nan, rows=[]) + if outcome is None: + return None + test, predictions = outcome + + n = min(len(test), len(predictions)) + if n == 0: + return BacktestSummary(model=model_name, n_holdout=0, mape=math.nan, mase=math.nan, rows=[]) + + results = accuracy_mod.evaluate( + test=test, + predictions=predictions, + train=df.iloc[:-horizon_steps].reset_index(drop=True), + season_length=season_length, + horizons=horizons[:1], + model_name=model_name, + ) + mape = results[0].mape if results else math.nan + mase = results[0].mase if results else math.nan + rows = _materialise_backtest_rows(test, predictions, n=n, levels=levels) + return BacktestSummary(model=model_name, n_holdout=n, mape=mape, mase=mase, rows=rows) + + +def _materialise_backtest_rows( + test: pd.DataFrame, + predictions: pd.DataFrame, + *, + n: int, + levels: list[int], +) -> list[BacktestFoldRow]: + """Stitch the hold-out actuals and per-step predictions into rows.""" + rows: list[BacktestFoldRow] = [] + yhat_col = predictions["yhat"].tolist() if "yhat" in predictions.columns else [] + actuals = test["y"].tolist() if "y" in test.columns else [] + timestamps = test["ds"].tolist() if "ds" in test.columns else [] + band_cols: dict[str, dict[str, list[float]]] = {} + for level in levels: + lo = f"yhat_lower_{level}" + hi = f"yhat_upper_{level}" + if lo in predictions.columns and hi in predictions.columns: + band_cols[str(level)] = { + "lower": predictions[lo].tolist(), + "upper": predictions[hi].tolist(), + } + for i in range(n): + bands_at_step: dict[str, dict[str, float]] = {} + for level_label, values in band_cols.items(): + bands_at_step[level_label] = { + "lower": float(values["lower"][i]), + "upper": float(values["upper"][i]), + } + rows.append( + BacktestFoldRow( + ds=str(timestamps[i]) if i < len(timestamps) else "", + actual=float(actuals[i]) if i < len(actuals) else math.nan, + yhat=float(yhat_col[i]) if i < len(yhat_col) else math.nan, + bands=bands_at_step, + ) + ) + return rows + + +def _group_index(config: Config, group_name: str) -> int: + for i, g in enumerate(config.groups): + if g.name == group_name: + return i + raise InspectError(f"group {group_name!r} disappeared between find_query and preprocess") + + +def _resolve_model_spec( + config: Config, + override: str | None, +) -> Any: + """Pick the model spec for the inspector fit. + + Mirrors :func:`promforecast.preview._pick_model_spec` shape: an + explicit override must match one of the configured models so the + inspector always reflects an actually-deployable choice. Falls + back to the first configured model when no override is given. + """ + available = list(config.defaults.models) + if not available: + raise InspectError("config.defaults.models is empty; cannot inspect without a model") + if override is None: + return available[0] + for spec in available: + if spec.name == override: + return spec + names = ", ".join(spec.name for spec in available) + raise InspectError(f"model {override!r} not in configured models ({names})") + + +def _pick_frame( + frames: list[SeriesFrame], + selector: dict[str, str] | None, +) -> SeriesFrame: + """Pick one frame from the source response, matching :func:`series_filter`. + + Same multi-frame disambiguation contract as the preview endpoint: + a single frame returns trivially, multiple frames require a + selector that picks exactly one. + """ + if not frames: + raise InspectError("query returned no series", exit_code=1) + if selector is None: + if len(frames) == 1: + return frames[0] + raise InspectError(f"query returned {len(frames)} series; pass --series-filter to pick one") + matches = [f for f in frames if all(f.labels.get(k) == v for k, v in selector.items())] + if not matches: + raise InspectError(f"series-filter {selector!r} matched no series among {len(frames)}") + if len(matches) > 1: + raise InspectError( + f"series-filter {selector!r} matched {len(matches)} series; tighten the filter" + ) + return matches[0] + + +def parse_series_filter(value: str | None) -> dict[str, str] | None: + """Parse ``--series-filter '{instance="a",device="sda1"}'`` into a dict. + + The shape mirrors PromQL label selectors (``label="value"`` pairs + inside braces) so an operator can copy a selector straight out of + a Grafana panel or a recording-rule body. Only the equality + operator is supported — anything richer should be expressed + server-side via the query's PromQL. + """ + if value is None: + return None + stripped = value.strip() + if not stripped: + return None + body = stripped[1:-1] if stripped.startswith("{") and stripped.endswith("}") else stripped + out: dict[str, str] = {} + if not body.strip(): + return None + for part in _split_top_level_commas(body): + if "=" not in part: + raise InspectError( + f"--series-filter token {part!r} is missing an '=' — use label=\"value\"" + ) + label, _, raw = part.partition("=") + out[label.strip()] = raw.strip().strip('"').strip("'") + return out + + +def _split_top_level_commas(body: str) -> Iterable[str]: + depth = 0 + current: list[str] = [] + for ch in body: + if ch == "," and depth == 0: + piece = "".join(current).strip() + if piece: + yield piece + current = [] + continue + if ch == '"': + depth ^= 1 + current.append(ch) + tail = "".join(current).strip() + if tail: + yield tail + + +def _column_as_floats(forecast: pd.DataFrame, col: str) -> list[float]: + if col not in forecast.columns: + return [] + out: list[float] = [] + for v in forecast[col].tolist(): + try: + f = float(v) + except (TypeError, ValueError): + out.append(math.nan) + continue + out.append(f) + return out + + +def _extract_bands(forecast: pd.DataFrame, levels: list[int]) -> dict[str, dict[str, list[float]]]: + bands: dict[str, dict[str, list[float]]] = {} + for level in levels: + lo = f"yhat_lower_{level}" + hi = f"yhat_upper_{level}" + if lo not in forecast.columns or hi not in forecast.columns: + continue + bands[str(level)] = { + "lower": _column_as_floats(forecast, lo), + "upper": _column_as_floats(forecast, hi), + } + return bands + + +def render_text(report: InspectReport) -> str: + """Human-friendly inspector output for the CLI default mode.""" + lines: list[str] = [] + lines.append(f"Query: {report.query_id} (group {report.group})") + lines.append(f"Model: {report.model}") + label_str = ( + ", ".join(f"{k}={v}" for k, v in sorted(report.series_labels.items())) + if report.series_labels + else "(none)" + ) + lines.append(f"Series: {label_str}") + lines.append( + f"Step / season / horizon: {report.step_seconds}s / " + f"{report.season_length} / {report.horizon_steps} steps" + ) + lines.append("") + lines.append("Input stats:") + lines.append(f" point_count : {report.input_stats.point_count}") + lines.append(f" gaps_filled : {report.input_stats.gap_count}") + lines.append(f" largest_gap_seconds: {report.input_stats.largest_gap_seconds:.1f}") + lines.append(f" lookback_start : {report.input_stats.lookback_start}") + lines.append(f" lookback_end : {report.input_stats.lookback_end}") + lines.append("") + lines.append("Preprocessing applied:") + for key, value in report.preprocessing.items(): + lines.append(f" {key:<26} : {value}") + lines.append("") + head_n = min(_TABLE_HEAD_ROWS, len(report.forecast_yhat)) + lines.append(f"Forecast (first {head_n} of {len(report.forecast_yhat)} rows):") + if head_n == 0: + lines.append(" (no forecast rows)") + else: + for i in range(head_n): + ts = report.forecast_timestamps[i] if i < len(report.forecast_timestamps) else "" + yhat = report.forecast_yhat[i] + band_summary = "" + if report.forecast_bands: + first_level = sorted(report.forecast_bands.keys())[0] + lo = report.forecast_bands[first_level]["lower"][i] + hi = report.forecast_bands[first_level]["upper"][i] + band_summary = f" [{first_level}: {lo:.4f}..{hi:.4f}]" + lines.append(f" {ts} yhat={yhat:.4f}{band_summary}") + if report.backtest is not None: + lines.append("") + lines.append("Backtest (most-recent hold-out):") + lines.append(f" model : {report.backtest.model}") + lines.append(f" n_holdout: {report.backtest.n_holdout}") + lines.append(f" MAPE : {_format_metric(report.backtest.mape)}%") + lines.append(f" MASE : {_format_metric(report.backtest.mase)}") + lines.append(f" per-fold rows (first {min(_TABLE_HEAD_ROWS, len(report.backtest.rows))}):") + for row in report.backtest.rows[:_TABLE_HEAD_ROWS]: + band_summary = "" + if row.bands: + first_level = sorted(row.bands.keys())[0] + band_summary = ( + f" [{first_level}: {row.bands[first_level]['lower']:.4f}.." + f"{row.bands[first_level]['upper']:.4f}]" + ) + lines.append( + f" {row.ds} actual={_format_metric(row.actual)} " + f"yhat={_format_metric(row.yhat)}{band_summary}" + ) + return "\n".join(lines) + "\n" + + +def _format_metric(value: float) -> str: + return f"{value:.4f}" if math.isfinite(value) else "NaN" + + +def _json_value(value: float) -> float | None: + return value if math.isfinite(value) else None diff --git a/forecaster/src/promforecast/main.py b/forecaster/src/promforecast/main.py index 4572ebe..a4d3dd0 100644 --- a/forecaster/src/promforecast/main.py +++ b/forecaster/src/promforecast/main.py @@ -11,6 +11,7 @@ from collections.abc import AsyncIterator, Awaitable, Callable from contextlib import asynccontextmanager from pathlib import Path +from typing import Any import structlog import uvicorn @@ -18,12 +19,16 @@ from fastapi.responses import JSONResponse, PlainTextResponse from . import adapter as adapter_module +from . import backfill as backfill_module from . import config as config_module from . import diagnose as diagnose_module from . import ha_cache as ha_cache_module +from . import inspect as inspect_module from . import preview as preview_module from . import telemetry as telemetry_module +from . import tuner as tuner_module from . import validate as validate_module +from . import warmup as warmup_module from . import what_if as what_if_module from .cache import MemoryQueryCache, QueryCache, QueryCacheBackend, RedisQueryCache from .exporter import Exporter @@ -54,6 +59,7 @@ def cli() -> None: dry_run=args.dry_run, datasource_url_override=args.datasource_url, strict_cardinality=args.strict_cardinality, + estimate_cost=args.estimate_cost, ) sys.exit(rc) @@ -65,6 +71,58 @@ def cli() -> None: ) return + if args.command == "backfill": + rc = backfill_module.run_backfill( + config_path=args.config, + from_duration=args.from_, + to_timestamp=args.to, + group_selector=args.groups, + sink_url_override=args.sink_url, + ) + sys.exit(rc) + + if args.command == "tune": + rc = tuner_module.run_tuner( + config_path=args.config, + group_selector=args.groups, + sink_url_override=args.sink_url, + lookback_override=args.lookback, + min_improvement_pct=args.min_improvement_pct, + ) + sys.exit(rc) + + if args.command == "inspect": + try: + series_filter = inspect_module.parse_series_filter(args.series_filter) + except inspect_module.InspectError as exc: + sys.stderr.write(f"error: {exc}\n") + sys.exit(exc.exit_code) + horizon_seconds = ( + _parse_prom_duration(args.horizon, flag_name="--horizon") if args.horizon else None + ) + lookback_seconds = ( + _parse_prom_duration(args.lookback, flag_name="--lookback") if args.lookback else None + ) + rc = inspect_module.run_inspect( + config_path=args.config, + query_id=args.query, + series_filter=series_filter, + horizon_seconds=horizon_seconds, + lookback_seconds=lookback_seconds, + model=args.model, + backtest=args.backtest, + json_output=args.json, + ) + sys.exit(rc) + + if args.command == "warmup": + rc = _run_warmup_cli( + url=args.url, + timeout_seconds=args.timeout, + json_output=args.json, + ) + sys.exit(rc) + if args.command == "diagnose": if args.diagnose_command == "duplicates": datasource_url = _resolve_diagnose_datasource(args) @@ -103,7 +161,7 @@ def cli() -> None: run(config_path=args.config, log_level=args.log_level) -def _build_parser() -> argparse.ArgumentParser: +def _build_parser() -> argparse.ArgumentParser: # noqa: PLR0915 parser = argparse.ArgumentParser(prog="promforecast") sub = parser.add_subparsers(dest="command") @@ -166,6 +224,17 @@ def _build_parser() -> argparse.ArgumentParser: "the global series cap. See docs/operations/cardinality.md." ), ) + validate_p.add_argument( + "--estimate-cost", + action="store_true", + help=( + "Run a cheap synthetic-data fit per (group, configured model) " + "pair and print the expected per-refresh CPU + memory budget. " + "Estimator is order-of-magnitude only — synthetic input doesn't " + "capture every real fit-time distribution. See " + "docs/operations/validate.md for accuracy bounds." + ), + ) adapter_p = sub.add_parser( "adapter", @@ -202,6 +271,229 @@ def _build_parser() -> argparse.ArgumentParser: help="Path to the private key for ``--tls-cert``.", ) + backfill_p = sub.add_parser( + "backfill", + help=( + "Replay the fit pipeline at simulated past timestamps and push " + "the resulting forecast curves + accuracy summaries to the " + "configured remote_write sink with historical timestamps. Use " + "to populate the long-term TSDB so SLOs / quality dashboards " + "are meaningful from day one rather than only from install " + "time forward. See ``docs/operations/backfill.md`` for the " + "resource-sizing trade-off." + ), + ) + backfill_p.add_argument( + "--config", + type=Path, + required=True, + help="Path to the forecaster YAML config (same shape the live runner uses).", + ) + backfill_p.add_argument( + "--from", + dest="from_", + required=True, + help=( + "How far back to walk, as a Prometheus duration (e.g. ``30d``). " + "Capped by ``safety.backfill_max_window`` so an accidental " + "year-long request fails fast." + ), + ) + backfill_p.add_argument( + "--to", + default=None, + help=( + "End timestamp for the walk. Accepts ``now`` (default) or an " + "ISO-8601 timestamp (``2026-05-24T12:00:00Z``)." + ), + ) + backfill_p.add_argument( + "--groups", + nargs="+", + default=None, + help=( + "Optional space-separated list of group names to backfill. " + "Defaults to every configured group when omitted." + ), + ) + backfill_p.add_argument( + "--sink-url", + default=None, + help=( + "Override ``sink.remote_write.url`` for the backfill push. " + "Useful when running the one-shot Job against a different TSDB " + "than the live forecaster's sink (e.g. a dedicated backfill " + "endpoint with a relaxed ``futureRetention``)." + ), + ) + + inspect_p = sub.add_parser( + "inspect", + help=( + "Debug a single forecast: fetch the configured series from the " + "TSDB, run the configured fit pipeline exactly as the live " + "forecaster would, and print the inputs, the fit, the forecast " + "curve, and optionally per-fold backtest predictions. See " + "``docs/operations/inspect.md`` for a debugging walkthrough." + ), + ) + inspect_p.add_argument( + "--config", + type=Path, + required=True, + help="Path to the forecaster YAML config (the live runner's config).", + ) + inspect_p.add_argument( + "--query", + required=True, + help=( + "ID of the query to inspect (matches one of ``groups[].queries[].id``). " + "The first match across groups wins." + ), + ) + inspect_p.add_argument( + "--series-filter", + default=None, + help=( + "PromQL-style label selector to pick one series when the query " + 'returns multiple frames (e.g. \'{instance="host-a",device="sda1"}\'). ' + "Required when the query returns more than one series." + ), + ) + inspect_p.add_argument( + "--horizon", + default=None, + help=( + "Override the forecast horizon for this run (Prometheus duration, " + "e.g. ``1h``). Falls back to ``defaults.horizon`` from the config." + ), + ) + inspect_p.add_argument( + "--lookback", + default=None, + help=( + "Override the lookback window for this run (Prometheus duration, " + "e.g. ``14d``). Falls back to ``defaults.lookback`` from the config." + ), + ) + inspect_p.add_argument( + "--model", + default=None, + help=( + "Override the model used for the inspector fit. Must be one of " + "the configured ``defaults.models`` entries; defaults to the first." + ), + ) + inspect_p.add_argument( + "--backtest", + action="store_true", + help=( + "Also run the configured backtest on the most recent hold-out and " + "print per-step actuals vs. predictions with bands and MAPE/MASE. " + "Useful for diagnosing 'backtests well, predicts badly' scenarios." + ), + ) + inspect_p.add_argument( + "--json", + action="store_true", + help=( + "Emit the full inspector report as JSON instead of the aligned " + "text table. Suitable for piping into ``jq`` or saving as a CI " + "artifact alongside a flaky forecast." + ), + ) + + tune_p = sub.add_parser( + "tune", + help=( + "Recommend hyperparameter values for the configured groups. Runs a " + "small grid over season_length and the configured models against " + "the past ``--lookback`` of data and pushes one ``forecast_recommended_param`` " + "row per candidate whose backtest MAPE beats the configured " + "baseline by ``--min-improvement-pct``. Typical schedule: a weekly " + "``CronJob`` (shipped via the ``promforecast-tuner`` sub-chart). " + "See ``docs/operations/auto-tuning.md`` for a 'should I apply this " + "recommendation?' decision guide." + ), + ) + tune_p.add_argument( + "--config", + type=Path, + required=True, + help="Path to the forecaster YAML config (the live runner's config).", + ) + tune_p.add_argument( + "--groups", + nargs="+", + default=None, + help=( + "Optional space-separated list of group names to tune. " + "Defaults to every configured group when omitted." + ), + ) + tune_p.add_argument( + "--sink-url", + default=None, + help=( + "Override ``sink.remote_write.url`` for the recommendation push. " + "Useful when running the tuner as a one-shot Job against a " + "different TSDB than the live forecaster's sink." + ), + ) + tune_p.add_argument( + "--lookback", + default=None, + help=( + "Override the lookback window for the grid search (Prometheus " + "duration, e.g. ``14d``). Falls back to ``defaults.lookback`` " + "from the config." + ), + ) + tune_p.add_argument( + "--min-improvement-pct", + type=_positive_float, + default=5.0, + help=( + "Minimum percentage MAPE improvement a candidate must show over " + "the configured baseline to be emitted as a recommendation. " + "Default 5.0 — keeps small / noisy improvements out of the " + "Grafana panel." + ), + ) + + warmup_p = sub.add_parser( + "warmup", + help=( + "Print per-(group, query) warm-up status by calling a forecaster's " + "/forecast/warmup endpoint. The typical operator workflow on a " + "fresh install: ``promforecast warmup --url http://localhost:9091`` " + "until every query reports status=ready." + ), + ) + warmup_p.add_argument( + "--url", + default="http://localhost:9091", + help=( + "Base URL of the forecaster to query. Defaults to " + "``http://localhost:9091`` for the in-cluster port-forward " + "and docker-compose dev stack." + ), + ) + warmup_p.add_argument( + "--timeout", + type=_positive_float, + default=30.0, + help="HTTP timeout for the /forecast/warmup call, in seconds.", + ) + warmup_p.add_argument( + "--json", + action="store_true", + help=( + "Print the raw JSON response instead of the aligned table. Useful " + "for piping into ``jq`` or consuming from a CI pre-flight script." + ), + ) + diagnose_p = sub.add_parser( "diagnose", help="Operator diagnostics against the long-term TSDB.", @@ -388,6 +680,88 @@ def _positive_int(value: str) -> int: return parsed +def _positive_float(value: str) -> float: + """argparse type for the warmup / inspect CLI timeout knobs.""" + try: + parsed = float(value) + except ValueError as exc: + raise argparse.ArgumentTypeError(f"must be a positive number; got {value!r}") from exc + if parsed <= 0: + raise argparse.ArgumentTypeError(f"must be > 0; got {parsed}") + return parsed + + +def _run_warmup_cli(*, url: str, timeout_seconds: float, json_output: bool) -> int: + """Call ``GET /forecast/warmup`` and render the response. + + Returns ``0`` when every query in every group reports ``ready`` + *or* when the forecaster has no configured queries to warm up + (``until promforecast warmup`` loops would otherwise spin forever + against a freshly-installed empty config). Returns ``1`` when at + least one query is still warming or in error so a CI pre-flight + script can gate "is the install done?" on the exit code. Network + failures return ``2`` so they're distinguishable from "still + warming". + """ + import json # noqa: PLC0415 + + import httpx # noqa: PLC0415 + + endpoint = url.rstrip("/") + "/forecast/warmup" + try: + response = httpx.get(endpoint, timeout=timeout_seconds) + response.raise_for_status() + payload = response.json() + except httpx.HTTPError as exc: + sys.stderr.write(f"error: warmup request to {endpoint!r} failed: {exc}\n") + return 2 + + if json_output: + sys.stdout.write(json.dumps(payload, indent=2) + "\n") + else: + report = _report_from_payload(payload) + sys.stdout.write(warmup_module.render_table(report)) + + statuses = { + q.get("status", "") for group in payload.get("groups", []) for q in group.get("queries", []) + } + if not statuses: + # Empty config — nothing to warm up. Treat as "done" rather + # than "still warming" so a docs-recommended ``until + # promforecast warmup`` loop terminates instead of spinning + # forever against a fresh install whose ``groups`` is empty + # (the operator hasn't added any queries yet, but the + # forecaster itself is healthy). + return 0 + return 0 if statuses == {"ready"} else 1 + + +def _report_from_payload(payload: dict[str, Any]) -> warmup_module.WarmupReport: + """Reconstruct a :class:`WarmupReport` from the JSON envelope. + + The HTTP body is a dict already shaped by ``WarmupReport.to_dict``; + this round-trip lets the CLI reuse the renderer without duplicating + the table formatting. + """ + groups: list[warmup_module.GroupWarmupStatus] = [] + for group in payload.get("groups", []): + queries = [ + warmup_module.QueryWarmupStatus( + id=str(q.get("id", "")), + status=q.get("status", "ready"), + points_available=int(q.get("points_available", 0)), + points_needed=int(q.get("points_needed", 0)), + blocked_by=str(q.get("blocked_by", "none")), + eta_seconds=float(q.get("eta_seconds", 0.0)), + ) + for q in group.get("queries", []) + ] + groups.append( + warmup_module.GroupWarmupStatus(name=str(group.get("name", "")), queries=queries) + ) + return warmup_module.WarmupReport(groups=groups) + + def _resolve_diagnose_datasource(args: argparse.Namespace) -> str: """``--datasource`` wins; otherwise pull ``datasource.url`` from ``--config``. @@ -535,6 +909,7 @@ def run(config_path: Path, log_level: str) -> None: expose_config_endpoint=cfg.server.expose_config_endpoint, expose_preview_endpoint=cfg.server.expose_preview_endpoint, expose_what_if_endpoint=cfg.server.expose_what_if_endpoint, + expose_warmup_endpoint=cfg.server.expose_warmup_endpoint, source=source, snapshot_cache=snapshot_cache, elector=elector, @@ -610,6 +985,7 @@ def _build_app( # noqa: PLR0915 expose_config_endpoint: bool = False, expose_preview_endpoint: bool = False, expose_what_if_endpoint: bool = False, + expose_warmup_endpoint: bool = False, source: PromSource | None = None, snapshot_cache: ha_cache_module.SnapshotCache | None = None, elector: LeaderElector | None = None, @@ -827,6 +1203,36 @@ async def what_if_endpoint(req: Request) -> Response: exporter.increment_what_if_run("admitted") return JSONResponse(result) + if expose_warmup_endpoint and source is None: + logger.warning( + "warmup_endpoint_disabled_no_source", + reason="server.expose_warmup_endpoint=true but no datasource was wired", + ) + if expose_warmup_endpoint and source is not None: + + @app.get("/forecast/warmup") + async def warmup_endpoint() -> Response: + """Report per-(group, query) warm-up status. + + See ``docs/operations/warmup.md``. Each call issues one + cheap probe per configured query — non-event, non-template + queries get a short ``query_range`` against the configured + lookback so the report reflects what the runner would + actually see on its next refresh tick. + """ + try: + report = await warmup_module.compute_warmup( + config=runner._config, + source=source, + ) + except Exception as exc: + logger.exception("warmup_endpoint_failed") + return JSONResponse( + {"error": "internal", "message": str(exc)}, + status_code=500, + ) + return JSONResponse(report.to_dict()) + return app diff --git a/forecaster/src/promforecast/point_factory.py b/forecaster/src/promforecast/point_factory.py index 0303737..03f3d56 100644 --- a/forecaster/src/promforecast/point_factory.py +++ b/forecaster/src/promforecast/point_factory.py @@ -940,6 +940,16 @@ def quality_points( series would fire the ``ForecastQualityZero`` alert without signal; the ``ForecastFallbackPersistent`` alert covers that case better. + Emission shape: each (series, model) yields one aggregated row + with ``horizon=""`` (the historical default — existing alert + matchers without a ``horizon`` selector keep working unchanged) + *plus* one row per configured accuracy horizon labelled with + that horizon's label. Capacity-relevant alerts on long horizons + can gate against ``forecast_quality_score{horizon="24h"}`` while + short-horizon deviation alerts gate on + ``forecast_quality_score{horizon="1h"}`` without a single + average masking either. + See :mod:`promforecast.quality` for the exact formula. """ abs_train = [abs(v) for v in train_y] @@ -964,9 +974,27 @@ def quality_points( score_kind=score_kind, ) yield QualityPoint( - labels={**common, "model": model_name}, + labels={**common, "model": model_name, "horizon": ""}, score=score, ) + # Per-horizon segmentation: same band-width + coverage inputs + # (those are series-level, not horizon-level) but the accuracy + # term is replaced with the per-horizon MAPE / MASE. The 1h + # forecast may score 0.9 while the 24h scores 0.5 — operator + # alerts can gate on the right horizon's reliability. + for res in outcome.results: + per_horizon_accuracy = res.mase if score_kind == "mase" else res.mape + per_horizon_score = quality_mod.compute( + backtest_mape=per_horizon_accuracy, + band_width_relative=band_width_relative, + coverage=coverage, + imputed_fraction=imputed_fraction, + score_kind=score_kind, + ) + yield QualityPoint( + labels={**common, "model": model_name, "horizon": res.horizon}, + score=per_horizon_score, + ) def _common_with_id(self) -> dict[str, str]: """Common labels for the operational families (carry ``id`` + ``group``). diff --git a/forecaster/src/promforecast/preprocess.py b/forecaster/src/promforecast/preprocess.py index 5fda987..d697471 100644 --- a/forecaster/src/promforecast/preprocess.py +++ b/forecaster/src/promforecast/preprocess.py @@ -26,8 +26,9 @@ import math from dataclasses import dataclass, field -from datetime import datetime, timedelta +from datetime import UTC, datetime, timedelta from typing import TYPE_CHECKING +from zoneinfo import ZoneInfo, ZoneInfoNotFoundError import numpy as np import structlog @@ -36,6 +37,7 @@ from numpy.typing import NDArray from .config import ( + BlacklistWindowConfig, ChangePointConfig, GapImputationConfig, OutlierConfig, @@ -76,6 +78,13 @@ class PreprocessResult: Counters are dict-of-counts so the runner can fold them into the exporter's per-(group, query, ...) counters without losing the sub-key breakdown (strategy, method). + + ``blacklisted_samples`` is a per-``reason`` counter (defaulting to + the empty string for unreasoned windows) for samples removed by the + ``preprocess.blacklist_windows`` list. The values flow into + ``forecast_input_blacklisted_samples_total{reason}`` so an operator + can confirm an annual-recurring blacklist actually fired for the + most recent year in the lookback. """ timestamps: list[datetime] @@ -86,6 +95,7 @@ class PreprocessResult: change_points: int = 0 effective_lookback_seconds: float = 0.0 imputed_fraction: float = 0.0 + blacklisted_samples: dict[str, int] = field(default_factory=dict) dropped: bool = False drop_reason: str | None = None @@ -113,7 +123,28 @@ def preprocess_series( if len(timestamps) != len(values): raise ValueError("preprocess: timestamps/values length mismatch") + blacklist_counts: dict[str, int] = {} + if config.blacklist_windows: + timestamps, values, blacklist_counts = _drop_blacklisted( + timestamps=timestamps, + values=values, + windows=config.blacklist_windows, + ) + + if not timestamps or not values: + # The blacklist consumed every sample. Surface as a drop so the + # caller bumps ``forecast_series_dropped_total{reason="preprocess_drop"}`` + # rather than continuing into the fit with an empty series. + return PreprocessResult( + timestamps=[], + values=[], + blacklisted_samples=blacklist_counts, + dropped=True, + drop_reason="blacklist_removed_all_samples", + ) + result = _fill_gaps(timestamps=timestamps, values=values, step=step, config=config.gaps) + result.blacklisted_samples = blacklist_counts if result.dropped: return result @@ -126,9 +157,153 @@ def preprocess_series( if not result.effective_lookback_seconds and result.timestamps: span = result.timestamps[-1] - result.timestamps[0] result.effective_lookback_seconds = span.total_seconds() + # Discount the blacklist-removed duration from the effective + # lookback so the gauge reflects the actual fit-input duration + # (a 14-day lookback with two 6-hour blacklisted windows reports + # ``14d - 12h`` rather than the raw 14d span). Calculated from + # ``count * step`` rather than the raw window length so a window + # that only partially overlaps the lookback is counted accurately. + total_blacklisted = sum(blacklist_counts.values()) + if total_blacklisted and step.total_seconds() > 0: + deduction = total_blacklisted * step.total_seconds() + result.effective_lookback_seconds = max(0.0, result.effective_lookback_seconds - deduction) return result +# ---------- blacklist windows ---------------------------------------------- + + +def _drop_blacklisted( # noqa: PLR0912 + *, + timestamps: list[datetime], + values: list[float], + windows: list[BlacklistWindowConfig], +) -> tuple[list[datetime], list[float], dict[str, int]]: + """Remove samples whose timestamp falls inside any blacklist window. + + Returns the surviving (timestamps, values) plus a per-``reason`` + counter of samples removed. Empty windows (zero-length absolute + spans or unparseable patterns) are skipped silently — pydantic + catches the gross-shape errors at boot, so anything reaching this + helper is structurally valid. + + The windows are interpreted as half-open intervals ``[start, end)``; + annual-recurring patterns expand to one absolute span per year + seen in the input timestamps (so a 30-year-lookback config does + not pay the cost of materialising 30 spans on every fit). + """ + if not timestamps: + return list(timestamps), list(values), {} + + # Expand recurring patterns once across the years actually present + # in the input window so the per-sample comparison stays cheap. + span_years = _years_present(timestamps) + expanded: list[tuple[datetime, datetime, str]] = [] + for window in windows: + if window.recurring == "annual": + for year in span_years: + pair = _parse_annual_window( + window.start, window.end, year, timezone_name=window.timezone + ) + if pair is not None: + expanded.append((pair[0], pair[1], window.reason)) + else: + absolute = _parse_absolute_window(window.start, window.end) + if absolute is not None: + expanded.append((absolute[0], absolute[1], window.reason)) + + if not expanded: + return list(timestamps), list(values), {} + + out_ts: list[datetime] = [] + out_vals: list[float] = [] + counts: dict[str, int] = {} + for ts, val in zip(timestamps, values, strict=True): + ts_aware = ts if ts.tzinfo is not None else ts.replace(tzinfo=UTC) + hit_reason: str | None = None + for start, end, reason in expanded: + if start <= ts_aware < end: + hit_reason = reason + break + if hit_reason is None: + out_ts.append(ts) + out_vals.append(val) + else: + counts[hit_reason] = counts.get(hit_reason, 0) + 1 + return out_ts, out_vals, counts + + +def _years_present(timestamps: list[datetime]) -> list[int]: + """Return the distinct years present in ``timestamps``.""" + if not timestamps: + return [] + first = timestamps[0] + last = timestamps[-1] + return list(range(first.year, last.year + 1)) + + +def _parse_absolute_window(start: str, end: str) -> tuple[datetime, datetime] | None: + """Parse an ISO-8601 ``start`` / ``end`` pair into UTC-aware datetimes. + + Returns ``None`` on a malformed input — the pydantic validator + catches structurally-wrong shapes at config load, so reaching this + branch implies a deeper format error worth swallowing rather than + crashing the per-fit pipeline. + """ + try: + # ``fromisoformat`` is strict about the trailing ``Z`` only on + # Python 3.11+, but ``Z`` is the standard UTC suffix the rest + # of the codebase emits; normalise to ``+00:00`` for portability. + s = datetime.fromisoformat(start.replace("Z", "+00:00")) + e = datetime.fromisoformat(end.replace("Z", "+00:00")) + except ValueError: + return None + if s.tzinfo is None: + s = s.replace(tzinfo=UTC) + if e.tzinfo is None: + e = e.replace(tzinfo=UTC) + if e <= s: + return None + return s, e + + +def _parse_annual_window( + start_pattern: str, end_pattern: str, year: int, *, timezone_name: str = "UTC" +) -> tuple[datetime, datetime] | None: + """Materialise an annual ``MM-DDTHH:MM`` window for ``year``. + + Returns ``None`` on malformed input or an unrecognised IANA + ``timezone_name``. The window is anchored in the operator's + configured local zone (defaulting to UTC) so a Black-Friday-style + rule flips at local midnight rather than UTC-midnight; the + returned datetimes are normalised to UTC so the per-sample + comparison in :func:`_drop_blacklisted` can stay zone-agnostic. + """ + try: + tz = ZoneInfo(timezone_name) + except ZoneInfoNotFoundError: + return None + try: + s_local = datetime.fromisoformat(f"{year:04d}-{start_pattern}").replace(tzinfo=tz) + e_year = year + # Allow the recurring window to wrap a year boundary + # ("12-29T00:00" -> "01-02T00:00") by rolling ``end`` forward. + e_local = datetime.fromisoformat(f"{year:04d}-{end_pattern}").replace(tzinfo=tz) + if e_local <= s_local: + e_year = year + 1 + e_local = datetime.fromisoformat(f"{e_year:04d}-{end_pattern}").replace(tzinfo=tz) + except ValueError: + return None + # Normalise to UTC so the half-open ``[start, end)`` comparison + # against UTC-aware sample timestamps is correct regardless of the + # zone chosen by the operator. + s = s_local.astimezone(UTC) + e_candidate = e_local.astimezone(UTC) + if e_candidate <= s: + return None + return s, e_candidate + + # ---------- gap imputation ------------------------------------------------- diff --git a/forecaster/src/promforecast/revision.py b/forecaster/src/promforecast/revision.py new file mode 100644 index 0000000..847fc20 --- /dev/null +++ b/forecaster/src/promforecast/revision.py @@ -0,0 +1,158 @@ +"""Forecast-to-forecast revision magnitude. + +Captures how much the *forecast itself* changes between fits. The +predicted value for "tomorrow 3pm" moving 30% since the last refit is +a signal in its own right, distinct from the input-data drift the +``forecast_drift_score`` family captures. + +The metric is normalised by the prior forecast's typical scale — +``|new - old| / max(|old|, scale_floor)`` — so a 5%-of-baseline shift +scores 0.05 regardless of whether the underlying values are tens or +gigabytes. ``scale_floor`` is a small absolute floor (default 1e-9) +to prevent division-by-zero on values near zero; it's not configurable +on a per-query basis because the floor is a numerical guard rather +than a policy knob. + +Per-(series, model, horizon) cache lives on the runner (one entry per +emitted snapshot row, keyed by the same labelset that lands on +/metrics). A reload that drops a (group, query) prunes the matching +cache entries via :meth:`RevisionCache.prune_to`; the runner already +calls this on reload so the cache stays bounded by the live config. + +To survive series-key churn on workloads with high label cardinality +(rolling pod IPs, ephemeral container IDs) without leaking forever +between reloads, the cache enforces an LRU cap. New writes evict the +least-recently-touched entry when the cap is hit. +""" + +from __future__ import annotations + +import math +from collections import OrderedDict +from dataclasses import dataclass + +# Numerical floor for the relative-shift denominator. Values smaller +# than this are clamped up so a near-zero baseline doesn't blow the +# ratio to infinity. Picked well below the smallest realistic +# operational value but above ordinary float64 noise. +_SCALE_FLOOR = 1e-9 + +# LRU cap on the per-(series, model, horizon) cache. Sized so a typical +# deployment (low thousands of series x a handful of models x a handful +# of horizons) sits well below the cap, while an ephemeral-labelset +# deployment that would otherwise leak indefinitely caps at a bounded +# memory footprint (~few MB worst case at one float plus a 5-tuple key +# per entry). Chosen as a round number rather than tuned — the cache is +# only deciding whether the next fit reports a magnitude or a NaN. +_DEFAULT_MAX_ENTRIES = 100_000 + + +@dataclass(frozen=True) +class RevisionPoint: + """One ``forecast_revision_magnitude{...}`` sample. + + ``magnitude`` is the relative shift between the new and previous + forecast for the same horizon. ``NaN`` when no prior fit was + cached (first fit after boot or after a reload that pruned the + entry) so the exporter can drop it without firing an alert on + "no prior data". + """ + + labels: dict[str, str] + magnitude: float + + +def compute_revision_magnitude(*, previous: float, current: float) -> float: + """Relative absolute shift between two scalar forecasts. + + Returns ``NaN`` when either input is non-finite. Otherwise the + return value is ``|current - previous| / max(|previous|, _SCALE_FLOOR)`` + so a 30% shift maps to 0.3 regardless of magnitude. Negative + returns are not possible — the metric is unsigned by design + (operators alert on "the forecast moved a lot", not "the forecast + moved up"). + """ + if not (math.isfinite(previous) and math.isfinite(current)): + return float("nan") + denom = max(abs(previous), _SCALE_FLOOR) + return abs(current - previous) / denom + + +class RevisionCache: + """LRU-bounded cache of the previous central-forecast value per emitted row. + + Keys are ``(group, query, series_key, model, horizon)`` so the + cache rotates correctly when the underlying labelset changes + (e.g. a series rotated out of the source's response). One float + per key — the cache holds only the *previous* central value, not + the full curve, so the per-(emitted-row) footprint stays + minimal. Reload-time pruning via :meth:`prune_to` clears + (group, query) pairs that have left the live config; in-fit churn + on series labels (high-cardinality ephemeral labels like rolling + pod IPs) is bounded by the LRU cap so the cache never grows + unboundedly between reloads. + """ + + def __init__(self, *, max_entries: int = _DEFAULT_MAX_ENTRIES) -> None: + # OrderedDict gives us O(1) move-to-end on read + O(1) popitem + # on the LRU eviction path, both of which the dict primitive + # doesn't expose. + self._previous: OrderedDict[tuple[str, str, str, str, str], float] = OrderedDict() + self._max_entries = max_entries + + def get( + self, + *, + group: str, + query: str, + series_key: str, + model: str, + horizon: str, + ) -> float | None: + key = (group, query, series_key, model, horizon) + value = self._previous.get(key) + if value is not None: + # Touch on read so a series that's still being scored each + # tick stays at the most-recent end of the LRU even if its + # value happens to match the prior fit exactly (a put + # touches separately). + self._previous.move_to_end(key) + return value + + def put( + self, + *, + group: str, + query: str, + series_key: str, + model: str, + horizon: str, + value: float, + ) -> None: + if not math.isfinite(value): + return + key = (group, query, series_key, model, horizon) + if key in self._previous: + self._previous.move_to_end(key) + self._previous[key] = value + # Evict from the oldest end until we're back under the cap. + # popitem(last=False) is O(1). + while len(self._previous) > self._max_entries: + self._previous.popitem(last=False) + + def prune_to(self, live_pairs: set[tuple[str, str]]) -> None: + """Drop entries whose (group, query) no longer exists in the live config.""" + survivors = [ + (key, value) for key, value in self._previous.items() if (key[0], key[1]) in live_pairs + ] + self._previous = OrderedDict(survivors) + + def __len__(self) -> int: + return len(self._previous) + + +__all__ = [ + "RevisionCache", + "RevisionPoint", + "compute_revision_magnitude", +] diff --git a/forecaster/src/promforecast/right_sizing.py b/forecaster/src/promforecast/right_sizing.py new file mode 100644 index 0000000..2d2cf3c --- /dev/null +++ b/forecaster/src/promforecast/right_sizing.py @@ -0,0 +1,205 @@ +"""Per-workload right-sizing recommendations from forecast peaks. + +The forecaster already has the model machinery, the per-series quality +signal, and the future trajectory. This module turns those into a +per-workload "you can downsize from X to Y" gauge — one row per +operator-configured (workload, kind) entry, gated on +``forecast_quality_score`` so untrustworthy forecasts never produce a +recommendation. + +The math is intentionally simple: + +* **Predicted peak** — ``max(yhat_upper_)`` across the + forecast horizon (falling back to ``max(yhat)`` when no bands were + emitted). The upper band is the conservative answer; right-sizing on + the central forecast would systematically under-provision for the + expected variability. +* **Recommended size** — ``peak * (1 + headroom_pct / 100)``. Headroom + defaults to 30% which is the SRE-standard "leave some rope for + unexpected load" cushion. +* **Confidence label** — derived from the forecast's quality score so + operators can see at a glance which recommendations are worth acting + on (``high`` ≥ 0.8, ``medium`` ≥ 0.6, ``low`` below). + +Pure functions — the runner threads the per-series snapshots through +and gets back a flat list of recommendation rows ready for the +exporter. +""" + +from __future__ import annotations + +import math +from dataclasses import dataclass +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from .config import ResourceSizingConfig, ResourceSizingWorkloadConfig + from .exporter import ForecastPoint, QualityPoint + +# Confidence buckets — match the gates operators typically wire on +# ``forecast_quality_score`` so the categorical signal aligns with the +# alerting story. +_CONFIDENCE_HIGH = 0.8 +_CONFIDENCE_MEDIUM = 0.6 + + +@dataclass(frozen=True) +class ResourceSizingRecommendation: + """One ``forecast_recommended_resource_size`` row destined for /metrics. + + ``recommended`` is the sample value of the primary gauge; the + optional ``current`` / ``savings_pct`` populate the sibling + gauges when the operator wired ``current_promql`` on the workload. + ``confidence`` rides as a categorical label + (``high|medium|low``) so dashboards can filter quickly. + """ + + workload: str + kind: str + confidence: str + recommended: float + current: float | None = None + savings_pct: float | None = None + + +def confidence_for(quality_score: float) -> str: + """Map a quality score in ``[0, 1]`` to a categorical confidence band.""" + if not math.isfinite(quality_score): + return "low" + if quality_score >= _CONFIDENCE_HIGH: + return "high" + if quality_score >= _CONFIDENCE_MEDIUM: + return "medium" + return "low" + + +def compute_predicted_peak( + forecast_points: list[ForecastPoint], + query_id: str, +) -> float | None: + """Return the conservative peak from the forecast's upper band, or None. + + Walks the snapshot's forecast points filtered to ``query_id`` and + picks the maximum of the highest-confidence ``yhat_upper`` row + (looking for ``_forecast_upper`` metric names). Falls back to + the central ``_forecast`` peak when no bands are present. NaN + samples are skipped so a single degenerate row doesn't poison the + peak; an entirely-non-finite forecast returns ``None`` so the + caller can skip the workload silently rather than emit a NaN + recommendation. + + Two-pass implementation: bucket the upper-band rows by level + first, then pick the max value at the highest level. Clearer + than interleaving "which level wins" with "which value wins" in + a single branch. + """ + upper_metric = f"{query_id}_forecast_upper" + central_metric = f"{query_id}_forecast" + upper_by_level: dict[int, float] = {} + central_max: float = float("-inf") + for point in forecast_points: + if not math.isfinite(point.value): + continue + if point.metric == upper_metric: + level_str = point.labels.get("level", "") + try: + level = int(level_str) + except ValueError: + continue + prev = upper_by_level.get(level, float("-inf")) + upper_by_level[level] = max(prev, point.value) + elif point.metric == central_metric: + central_max = max(central_max, point.value) + if upper_by_level: + # Highest configured level wins — the conservative-est band + # the operator opted into. + best_level = max(upper_by_level) + return float(upper_by_level[best_level]) + if math.isfinite(central_max): + return float(central_max) + return None + + +def compute_recommendation( + *, + workload: ResourceSizingWorkloadConfig, + config: ResourceSizingConfig, + forecast_points: list[ForecastPoint], + quality_points: list[QualityPoint], + current: float | None = None, +) -> ResourceSizingRecommendation | None: + """Build a single recommendation row, or ``None`` when ungated. + + Returns ``None`` (the workload is skipped) when: + + * the forecast snapshot has no finite peak for ``forecast_query``, + * every quality_point for the query is non-finite, or + * the (max) quality score is below ``config.quality_floor``. + + When ``current`` is provided we compute the savings percentage as + ``(current - recommended) / current * 100`` (negative means the + forecast predicts a need *above* the current limit — operators + read that as "upsize, don't downsize"). A zero or negative + ``current`` is treated as missing because the percentage is + undefined / misleading there. + """ + peak = compute_predicted_peak(forecast_points, workload.forecast_query) + if peak is None: + return None + # Quality gate: take the *mean* of finite quality scores for the + # query so the gate reflects "this forecast is broadly trustworthy", + # not "at least one emission was good". A naive max-quality gate + # would let a single high-quality model pull a workload's + # recommendation through even when every other emission for the + # same query is poor — and the peak we picked above might have + # come from one of those poor emissions (max-quality on one model + # doesn't constrain max-peak across models). The mean is the + # safer default; operators who want a per-model gate can configure + # `auto_select: true` so only the winner contributes a + # quality_point. + quality = _mean_quality_for_query(quality_points, workload.forecast_query) + if quality is None or quality < config.quality_floor: + return None + recommended = peak * (1.0 + config.headroom_pct / 100.0) + # Drop ``current`` (and therefore ``savings_pct``) when the value + # is missing, non-finite, or non-positive. A zero / negative + # current makes the percentage undefined and surfacing a + # ``current=0`` row would mislead operators into reading the + # workload as "unprovisioned" instead of "we couldn't measure it". + has_current = current is not None and math.isfinite(current) and current > 0 + savings: float | None = None + if has_current: + # mypy: ``has_current`` already narrowed current to a non-None + # finite positive float. + assert current is not None + savings = (current - recommended) / current * 100.0 + return ResourceSizingRecommendation( + workload=workload.workload, + kind=workload.kind, + confidence=confidence_for(quality), + recommended=recommended, + current=current if has_current else None, + savings_pct=savings, + ) + + +def _mean_quality_for_query( + quality_points: list[QualityPoint], + query_id: str, +) -> float | None: + """Return the mean of finite quality scores for ``query_id``, or None. + + Empty / all-non-finite input returns ``None`` so the caller can + skip the workload silently rather than treating a missing + measurement as a passing gate. + """ + finite_scores: list[float] = [] + for point in quality_points: + if point.labels.get("id") != query_id: + continue + if not math.isfinite(point.score): + continue + finite_scores.append(float(point.score)) + if not finite_scores: + return None + return sum(finite_scores) / len(finite_scores) diff --git a/forecaster/src/promforecast/runner/_helpers.py b/forecaster/src/promforecast/runner/_helpers.py index 1ad6fef..3039665 100644 --- a/forecaster/src/promforecast/runner/_helpers.py +++ b/forecaster/src/promforecast/runner/_helpers.py @@ -245,3 +245,101 @@ def _run_fit( return model.fit_predict( df=df, horizon=horizon, freq=freq, levels=levels, regressors=regressors ) + + +# Floor for relative-metric denominators across the per-horizon +# revision + disagreement helpers below; matches the +# :mod:`promforecast.revision` scale floor. +_RELATIVE_METRIC_FLOOR = 1e-9 +# Minimum number of distinct finite values required to compute the +# disagreement coefficient of variation. Two is the smallest mathematically +# meaningful count; below that there's no spread to measure. +_MIN_DISAGREEMENT_VALUES = 2 + + +def _extract_per_horizon_yhat(forecast: Any, horizon_specs: list[Any]) -> dict[str, float]: + """Return ``{horizon_label: yhat_value}`` from a forecast frame. + + ``horizon_specs`` is the list produced by + :func:`promforecast.accuracy.horizons_from_durations`. Each entry's + ``steps`` is the 1-indexed step count for that horizon — we read + row ``steps - 1`` of ``forecast`` for that horizon's yhat. Rows + past the end of the frame and non-finite values are skipped. + """ + import math # noqa: PLC0415 + + if forecast is None or len(forecast) == 0: + return {} + out: dict[str, float] = {} + try: + yhat_col = forecast["yhat"] + except (KeyError, TypeError): + return out + n = len(forecast) + for spec in horizon_specs: + idx = int(spec.steps) - 1 + if idx < 0 or idx >= n: + continue + try: + value = float(yhat_col.iloc[idx]) + except (KeyError, IndexError, ValueError): + continue + if not math.isfinite(value): + continue + out[spec.label] = value + return out + + +def _coefficient_of_variation(values: list[float]) -> float: + """Coefficient of variation (stddev / |mean|) across ``values``. + + Used as the model-disagreement score: 0 = perfect agreement, higher + = more disagreement. Returns ``NaN`` when fewer than 2 finite + values are present (no disagreement to measure) or when the mean + is too close to zero for the relative metric to be meaningful. + """ + import math # noqa: PLC0415 + + finite = [v for v in values if math.isfinite(v)] + if len(finite) < _MIN_DISAGREEMENT_VALUES: + return float("nan") + mean = sum(finite) / len(finite) + denom = abs(mean) + if denom < _RELATIVE_METRIC_FLOOR: + return float("nan") + variance = sum((v - mean) ** 2 for v in finite) / len(finite) + return math.sqrt(variance) / denom + + +def _estimate_group_memory_bytes(group_name: str, exporter: Any) -> float: + """Best-effort per-group snapshot memory estimate. + + Sums ``sys.getsizeof`` over every list-valued field on the snapshot + so the resulting gauge tracks "which group's snapshot is dominant" + without pulling in psutil. Fields are discovered via + :func:`dataclasses.fields` so a new emission family that adds a + ``list`` attribute to :class:`GroupSnapshot` is accounted for + without an edit here — the hardcoded-list shape this replaced + silently undercounted the snapshot every time a new family + landed. The number is still a recursive undercount (Python's + object graph isn't fully traversed) and an overcount when point + objects are shared across snapshots; intent is comparative, not + exact. Falls back to 0.0 when the snapshot is missing. + """ + import dataclasses # noqa: PLC0415 + import sys # noqa: PLC0415 + + snapshot = exporter.snapshot_for(group_name) + if snapshot is None: + return 0.0 + total = sys.getsizeof(snapshot) + if not dataclasses.is_dataclass(snapshot): + return float(total) + for snap_field in dataclasses.fields(snapshot): + items = getattr(snapshot, snap_field.name, None) + if not isinstance(items, list) or not items: + continue + total += sys.getsizeof(items) + for item in items: + total += sys.getsizeof(item) + return float(total) diff --git a/forecaster/src/promforecast/runner/_scheduler.py b/forecaster/src/promforecast/runner/_scheduler.py index fbb0a66..f66bc97 100644 --- a/forecaster/src/promforecast/runner/_scheduler.py +++ b/forecaster/src/promforecast/runner/_scheduler.py @@ -20,7 +20,7 @@ from collections.abc import Awaitable, Callable from concurrent.futures import ThreadPoolExecutor from datetime import UTC, datetime, timedelta -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, Any import structlog @@ -34,12 +34,16 @@ from .. import deviation as deviation_mod from .. import discovery as discovery_mod from .. import drift as drift_mod +from .. import drift_explainer as drift_explainer_mod +from .. import health as health_mod from .. import incident_pattern as incident_pattern_mod from .. import maintenance as maintenance_mod from .. import models as model_registry from .. import preprocess as preprocess_mod from .. import quality as quality_mod from .. import regressors as regressors_mod +from .. import revision as revision_mod +from .. import right_sizing as right_sizing_mod from .. import seasonality as seasonality_mod from .. import slo as slo_mod from .. import telemetry as telemetry_mod @@ -57,6 +61,7 @@ GroupConfig, ModelSpec, QueryConfig, + ResourceSizingWorkloadConfig, effective_emission_safeguards, effective_refresh_interval, effective_season_length, @@ -71,8 +76,13 @@ GroupSnapshot, IncidentRecurrencePoint, InvalidMetricNameError, + ModelDisagreementPoint, PreprocessStats, QualityPoint, + ResourceSizingPoint, + RevisionPoint, + ServiceHealthScorePoint, + ThresholdActionCostPoint, validate_metric_name, ) from ..point_factory import BacktestOutcome, SeriesPointFactory @@ -81,9 +91,13 @@ from ..source import PromSource, SeriesFrame from ._budgets import _apply_canary, _apply_overflow, _compute_global_budgets from ._helpers import ( + _coefficient_of_variation, _combine_ensemble, + _estimate_group_memory_bytes, + _extract_per_horizon_yhat, _format_band_window, _pandas_freq, + _query_id_from_metric, _resolve_spec, _run_backtest, _run_fit, @@ -144,6 +158,11 @@ # series budget every refresh. _INCIDENT_RECURRENCE_ROWS = 7 * 24 +# Minimum number of distinct emitted models required to compute the +# per-(series, horizon) model-disagreement coefficient of variation. +# Below two there is no spread to measure; the gauge is suppressed. +_MIN_MODELS_FOR_DISAGREEMENT = 2 + def _build_deploy_tracker(config: Config) -> deploy_risk_mod.DeployTracker: """Construct a :class:`DeployTracker` with per-service caps from config. @@ -238,6 +257,90 @@ def _aggregate_regressor_values( return means, timestamps +def _trailing_regressor_values(regressors: Any | None) -> dict[str, float]: + """Extract the most recent finite value of each regressor column. + + The drift explainer reads "what was loud at the drift moment" from + the latest row of the aligned regressor DataFrame. NaN-trailing + columns walk backwards to find the last finite reading so a + one-step gap at the end of the lookback (common after a deploy) + doesn't blank out the regressor's signal. + + Returns an empty dict when no regressors are configured (the + common case) so the explainer falls back to ``"unknown"`` + cleanly. + """ + if regressors is None: + return {} + out: dict[str, float] = {} + # The frame is expected to be a pandas DataFrame whose columns are + # ``unique_id``, ``ds``, and one column per regressor id. Avoid + # importing pandas at module load — this helper is called only + # when the runner has already imported it via the fit pipeline. + for column in regressors.columns: + if column in ("unique_id", "ds"): + continue + series = regressors[column] + if series.empty: + continue + for value in reversed(list(series)): + try: + fv = float(value) + except (TypeError, ValueError): + continue + if math.isfinite(fv): + out[str(column)] = fv + break + return out + + +def _build_threshold_action_cost_points(group: GroupConfig) -> list[ThresholdActionCostPoint]: + """Project a group's configured thresholds onto action-cost points. + + Pure projection from config — the points carry the operator's + declared cost as the sample value and one row per + (group, query, threshold) with ``action_cost`` configured. + Independent of fit results: a query whose fit failed (or was + skipped this tick) still publishes its action_cost rows so the + Grafana "rank by cost" panel doesn't go blank. + """ + out: list[ThresholdActionCostPoint] = [] + for query in group.queries: + annotations = capacity_mod.build_threshold_action_costs(query.thresholds) + for ann in annotations: + out.append( + ThresholdActionCostPoint( + labels={ + "id": query.id, + "group": group.name, + "name": ann.name, + "type": ann.cost_type, + }, + value=ann.cost_value, + ) + ) + return out + + +def _staleness_max_age_seconds(config: Config) -> float: + """Threshold for the staleness sweep on ``apply_config``. + + Returns the larger of one day and 24x the *shortest* configured + refresh interval (so the sweep keeps pace with sub-hour cadences + without dropping rows that are merely in their second alert + cycle). Picked well past the canonical + ``ForecastSeriesStale`` alert horizon (4x refresh) so an operator + gets multiple alert cycles before a row silently disappears, while + still bounding the memory cost of a noisy fleet. + """ + shortest = config.server.refresh_interval.total_seconds() + for group in config.groups: + for query in group.queries: + if query.refresh_interval is not None: + shortest = min(shortest, query.refresh_interval.total_seconds()) + return max(86_400.0, shortest * 24) + + class Runner: """Schedules and executes per-group forecast runs. @@ -273,6 +376,19 @@ def __init__( # cache; they read the leader's rendered drift gauges via the # shared snapshot. self._drift_cache = drift_mod.DriftCache() + # Forecast-to-forecast revision cache: stores the previous + # central yhat value per (series, model, horizon) so the next + # fit can score how much that horizon's prediction moved. Same + # lifecycle as ``_drift_cache`` — pruned on every reload to + # stay bounded by the live config. + self._revision_cache = revision_mod.RevisionCache() + # Drift-explainer rolling windows. Keyed per + # ``(group, query, series_key, model)`` so the Pearson + # correlation between drift_score and each regressor value + # captures the genuine per-series behaviour rather than + # averaging across the query's whole fan-out. Pruned on + # reload alongside the drift cache. + self._drift_windows: dict[tuple[str, str, str, str], drift_explainer_mod.DriftWindow] = {} # Rolling band-coverage tracker: keeps a bounded ring of # (actual ∈ band) booleans per (series, level, window). Lives on # the Runner so the rolling state survives across runs without @@ -393,6 +509,16 @@ async def apply_config(self, new_config: Config) -> ReloadResult: log = logger.bind(added=added, removed=removed, changed=changed) + # Unconditionally sweep the staleness state by age before + # the fast no-op short-circuit. The sweep is idempotent and + # cheap (dict comprehension over a bounded set), and running + # it on every reload gives operators a deterministic + # "trigger a cleanup" knob even when the config itself + # hasn't changed (the per-group run path also sweeps, but + # an explicit reload is a clearer mental model for + # debugging stale rows). + self._exporter.prune_stale_staleness(_staleness_max_age_seconds(new_config)) + # Fast no-op path: pydantic models compare structurally, so an # equal config means nothing changed and we shouldn't drain. # This matters when a ConfigMap is re-projected with identical @@ -416,6 +542,28 @@ async def apply_config(self, new_config: Config) -> ReloadResult: self._exporter.retain_groups(set(new_by_name)) self._exporter.retain_discovery_groups(set(new_by_name)) self._exporter.retain_incident_recurrence_groups(set(new_by_name)) + self._exporter.retain_staleness_groups(set(new_by_name)) + self._exporter.retain_self_observability_groups(set(new_by_name)) + # Drop fit-success-ratio buckets for models removed from + # ``defaults.models``; without this a dropped model's + # frozen-in-time ratio rows would linger on /metrics until + # process restart, and the ``ForecastModelFitFailing`` alert + # would either misfire on the stale numbers or have to be + # silenced manually. + self._exporter.retain_fit_outcome_models( + {spec.name for spec in new_config.defaults.models} + ) + # Per-query staleness cleanup: a reload that drops a single + # query from a still-present group needs the per-(group, query) + # entries pruned too, mirroring the incident-recurrence + # treatment below. Without this the dropped query's series + # would linger on /metrics with a forever-growing staleness + # value because no fit will ever reset their clock. + live_query_pairs = { + (group.name, query.id) for group in new_config.groups for query in group.queries + } + self._exporter.retain_staleness_queries(live_query_pairs) + self._exporter.retain_regressor_stability_queries(live_query_pairs) # Per-query cleanup for the incident-recurrence heatmap: # the group-level retain above handles whole-group removal, # but a reload that drops a single ``data_profile: events`` @@ -438,7 +586,18 @@ async def apply_config(self, new_config: Config) -> ReloadResult: # reload that drops a query doesn't leak its state forever. live_pairs = {(g.name, q.id) for g in new_config.groups for q in g.queries} self._drift_cache.prune_to(live_pairs) + self._revision_cache.prune_to(live_pairs) self._band_coverage.forget(live_pairs) + # Drift-explainer windows are keyed per + # ``(group, query, series_key, model)`` — the live_pairs + # set is ``(group, query)``; we project the windows to + # those that survive the reload. + self._drift_windows = { + key: window + for key, window in self._drift_windows.items() + if (key[0], key[1]) in live_pairs + } + self._exporter.retain_drift_explained_queries(live_pairs) # Rebuild the deploy tracker against the new spec set. The # rebuild deliberately drops every in-flight active deploy — # a reload mid-rollout starts the next risk-score emission @@ -459,6 +618,19 @@ async def apply_config(self, new_config: Config) -> ReloadResult: # call repopulates from the rebuilt tracker. self._deploy_tracker = _build_deploy_tracker(new_config) self._exporter.replace_deploy_risk_scores([]) + # A reload that turns off ``governance.health_score`` must + # drop the prior aggregated rows immediately rather than + # waiting for the next group run to clear them. The + # opposite case (newly enabled) just lets the next group + # run repopulate via :meth:`_refresh_service_health_scores`. + if new_config.governance.health_score is None: + self._exporter.replace_service_health_scores([]) + # Same treatment for right-sizing — a reload that disables + # the feature (or empties the workload list) must drop the + # prior recommendations. + new_rs = new_config.recommendations.resource_sizing + if not new_rs.enabled or not new_rs.workloads: + self._exporter.replace_resource_sizing_recommendations([]) # Hot-reload the what-if throttle interval. The next-allowed # timestamp inside the throttle is preserved so a reload # can't be used to bypass an in-flight throttle window; @@ -710,6 +882,20 @@ async def run_group( # noqa: PLR0912, PLR0915 ) for cpkey, cpcount in ctx.change_points.items(): merged_changepoints[cpkey] = merged_changepoints.get(cpkey, 0) + cpcount + # Blacklist-window sample counter: monotonic merge keyed on + # (query, reason), pruned to live query IDs so a reload that + # drops a query also drops its row. + merged_blacklisted: dict[tuple[str, str], int] = ( + { + key: count + for key, count in previous_snapshot.blacklisted_samples_cumulative.items() + if key[0] in current_query_ids + } + if previous_snapshot is not None + else {} + ) + for bkey, bcount in ctx.blacklisted_samples.items(): + merged_blacklisted[bkey] = merged_blacklisted.get(bkey, 0) + bcount # Monotonic merge for the emission-clipped counter, pruned # to query IDs that still exist in the live group config so a # reload that drops a query doesn't leak its old label set @@ -745,6 +931,13 @@ async def run_group( # noqa: PLR0912, PLR0915 # tracker on the runner — pull a snapshot now so the next # /metrics scrape sees the most recent rolling fractions. band_coverage_points = self._build_band_coverage_points(group, run_query_ids) + # Threshold action-cost is config-derived (not fit-derived), + # so rebuild the list directly from the current group config + # every run. A threshold removed from YAML disappears from + # /metrics on the next refresh; partial-tick refreshes still + # publish the full set because every configured query is + # walked here regardless of which subset actually ran. + threshold_action_cost_points = _build_threshold_action_cost_points(group) self._exporter.update( GroupSnapshot( group=group.name, @@ -784,11 +977,15 @@ async def run_group( # noqa: PLR0912, PLR0915 gaps_filled_cumulative=merged_gaps, outliers_replaced_cumulative=merged_outliers, change_points_cumulative=merged_changepoints, + blacklisted_samples_cumulative=merged_blacklisted, emission_clipped_cumulative=merged_emission_clipped, cold_start_siblings=dict(ctx.cold_start_siblings), drift_points=list(ctx.drift_points), drift_alerts_cumulative=merged_drift_alerts, band_coverage_points=band_coverage_points, + revision_points=list(ctx.revision_points), + model_disagreement_points=list(ctx.model_disagreement_points), + threshold_action_cost_points=threshold_action_cost_points, ) ) log.info( @@ -807,6 +1004,27 @@ async def run_group( # noqa: PLR0912, PLR0915 regressor_failures=len(ctx.regressor_failures), ) + # Self-observability: per-group resource attribution. + # CPU is the sum of per-(query, model) wall-clock seconds this + # run consumed on the shared executor — the runner's only + # honest CPU number without process-level instrumentation. + # **Crucially** we filter to queries that *ran this tick* — the + # inheritance pass populates ``ctx.fit_durations`` with prior + # values for skipped queries, and those durations already + # contributed to the counter in an earlier cycle. Without the + # filter, partial-tick refreshes would silently re-add the + # same seconds on every refresh. Memory is a best-effort + # snapshot footprint captured here — emitting it before the + # snapshot is updated would miss the current run's contribution. + cpu_this_run = sum( + seconds for (q_id, _), seconds in ctx.fit_durations.items() if q_id in run_query_ids + ) + if cpu_this_run > 0: + self._exporter.add_group_cpu_seconds(group.name, cpu_this_run) + self._exporter.set_group_memory_bytes( + group.name, _estimate_group_memory_bytes(group.name, self._exporter) + ) + # Publish the rendered snapshot to the shared HA cache if a # publisher is wired. We do this *after* updating the local # exporter so the rendered payload reflects the run we just @@ -859,6 +1077,40 @@ def _record_sink_retry(reason: str) -> None: except Exception: log.exception("deploy_risk_refresh_failed") + # Recompute the service-level health composite across all + # current group snapshots. Cheap (a single pass over the + # already-on-exporter quality_points) and idempotent — every + # group run rebuilds the full list rather than diff-merging. + # Best-effort: a defect in the aggregator logs but never + # crashes the run, matching the + # ``_refresh_resource_sizing_recommendations`` / + # ``_refresh_deploy_risk`` failure-isolation pattern. + try: + self._refresh_service_health_scores() + except Exception: + log.exception("service_health_refresh_failed") + + # Refresh per-workload right-sizing recommendations. + # Best-effort: any current_promql fetch failure logs but + # never crashes the run. Idempotent — whole-state + # replacement on the exporter drops workloads whose + # forecast fell below the quality floor this refresh. + try: + await self._refresh_resource_sizing_recommendations() + except Exception: + log.exception("resource_sizing_refresh_failed") + + # Sweep aged staleness rows after each successful group run. + # This is the only path that catches ephemeral-series leaks + # (per-pod metrics, deployment-id labels) without requiring + # an explicit ``/-/reload`` — the apply_config sweep handles + # model removal, but a series that simply stops appearing + # in the source data would otherwise linger forever. The + # operation is a single dict comprehension over a bounded + # dict, so amortising it across every group run adds + # negligible latency to the fit path. + self._exporter.prune_stale_staleness(_staleness_max_age_seconds(self._config)) + async def _refresh_deploy_risk( self, group: GroupConfig, @@ -1022,6 +1274,143 @@ def _publish_deploy_risk_scores(self) -> None: ] self._exporter.replace_deploy_risk_scores(points) + def _refresh_service_health_scores(self) -> None: + """Rebuild ``forecast_service_health_score`` from current snapshots. + + Cheap: walks the per-group quality_points already on the + exporter and feeds them through + :func:`health.compute_service_health_scores`. The runner calls + this after every group run; whole-state replacement on the + exporter means buckets that disappear (e.g. the only series + whose label set matched the dim tuple dropped to NaN this + refresh) leave /metrics on the next scrape. + + No-op when ``governance.health_score`` is unset, so deployments + that don't opt in pay zero cost. + """ + cfg = self._config.governance.health_score + if cfg is None: + # Clear any prior state — a reload that turns the feature + # off must drop the old gauge rows. + self._exporter.replace_service_health_scores([]) + return + quality_points: list[QualityPoint] = [] + for group_name in (g.name for g in self._config.groups): + snap = self._exporter.snapshot_for(group_name) + if snap is None: + continue + quality_points.extend(snap.quality_points) + scores = health_mod.compute_service_health_scores(quality_points, cfg) + self._exporter.replace_service_health_scores( + [ServiceHealthScorePoint(labels=dict(s.labels), score=s.score) for s in scores] + ) + + async def _refresh_resource_sizing_recommendations(self) -> None: + """Recompute right-sizing recommendations from current snapshots. + + Cheap per call: bounded by the operator-configured workload + count (typically a small number). For each workload: + + 1. Pull the forecast snapshot whose query id matches + ``forecast_query`` (walking every group's snapshot — the + workload doesn't bind to a specific group). + 2. Optionally fetch ``current_promql`` via the source. + 3. Delegate the math to + :func:`right_sizing.compute_recommendation` which handles + the quality gate. + 4. Replace the exporter's full recommendation map. + + Empty / disabled config short-circuits at the top — the + ``replace_resource_sizing_recommendations([])`` call there + clears any stale rows from a prior config that had this on. + """ + cfg = self._config.recommendations.resource_sizing + if not cfg.enabled or not cfg.workloads: + self._exporter.replace_resource_sizing_recommendations([]) + return + # Build a flat lookup of forecast / quality points keyed by + # query id so we don't re-walk every group per workload. + forecast_by_id: dict[str, list[ForecastPoint]] = {} + quality_by_id: dict[str, list[QualityPoint]] = {} + for group in self._config.groups: + snap = self._exporter.snapshot_for(group.name) + if snap is None: + continue + for point in snap.points: + qid = _query_id_from_metric(point.metric) + if qid: + forecast_by_id.setdefault(qid, []).append(point) + for qp in snap.quality_points: + qid = qp.labels.get("id", "") + if qid: + quality_by_id.setdefault(qid, []).append(qp) + out_points: list[ResourceSizingPoint] = [] + for workload in cfg.workloads: + forecast_points = forecast_by_id.get(workload.forecast_query, []) + quality_points = quality_by_id.get(workload.forecast_query, []) + current_value = await self._fetch_current_for_workload(workload) + rec = right_sizing_mod.compute_recommendation( + workload=workload, + config=cfg, + forecast_points=forecast_points, + quality_points=quality_points, + current=current_value, + ) + if rec is None: + continue + out_points.append( + ResourceSizingPoint( + labels={ + "workload": rec.workload, + "kind": rec.kind, + "confidence": rec.confidence, + }, + recommended=rec.recommended, + current=rec.current, + savings_pct=rec.savings_pct, + ) + ) + self._exporter.replace_resource_sizing_recommendations(out_points) + + async def _fetch_current_for_workload( + self, + workload: ResourceSizingWorkloadConfig, + ) -> float | None: + """Resolve ``current_promql`` to a single float, or ``None`` on failure. + + A workload without ``current_promql`` skips the fetch entirely. + On error (transport failure, malformed PromQL, no data) we + return ``None`` so the recommendation still emits without the + sibling ``current`` / ``savings_pct`` rows — better than + crashing the whole refresh on one workload's flaky query. + + The lookback window is intentionally narrow: 10 minutes at + ``defaults.step`` gives us enough samples to pick the most + recent finite reading without dragging in stale data. + """ + if workload.current_promql is None: + return None + end = datetime.now(tz=UTC) + step = self._config.defaults.step + start = end - timedelta(minutes=10) + step_seconds = max(1, int(step.total_seconds())) + try: + frames = await self._source.query_range( + workload.current_promql, start, end, step_seconds + ) + except Exception: + logger.exception( + "resource_sizing_current_query_failed", + workload=workload.workload, + kind=workload.kind, + ) + return None + for frame in frames: + for value in reversed(frame.values): + if math.isfinite(value): + return float(value) + return None + def _build_deviation_lookup(self) -> deploy_risk_mod.DeviationLookup: """Return a callable that maps (service, query_id) to mean |ratio|. @@ -1156,6 +1545,25 @@ async def _do_run_query( # noqa: PLR0912, PLR0915 rkey = (query.id, failure.regressor_id, failure.reason) ctx.regressor_failures[rkey] = ctx.regressor_failures.get(rkey, 0) + 1 + # Self-observability: record the per-(group, query, regressor_id) + # stability score derived from the aggregated PromQL values. One + # pass per query (not per series) keeps the bookkeeping cheap; + # calendar regressors are deterministic functions of timestamps so + # they're skipped — a "stability" measurement on a deterministic + # series isn't meaningful. + if prefetch.aggregated: + for regressor_id, agg_df in prefetch.aggregated.items(): + try: + values = agg_df["v"].tolist() + except (KeyError, AttributeError): + continue + self._exporter.record_regressor_stability( + group=group.name, + query=query.id, + regressor_id=regressor_id, + values=[float(v) for v in values], + ) + min_points = self._config.defaults.min_points eff_step = effective_step(query, self._config.defaults) # Cold-start sibling cache: fetched lazily on the first short @@ -1186,6 +1594,13 @@ async def _do_run_query( # noqa: PLR0912, PLR0915 query_stats.max_gap_seconds = max( query_stats.max_gap_seconds, cleaned.max_gap_seconds ) + # Blacklist samples may have driven the drop (the empty- + # series short-circuit when the entire window is + # blacklisted); preserve the per-reason counter so the + # operator can see *why* the series dropped. + for reason, count in cleaned.blacklisted_samples.items(): + bkey = (query.id, reason) + ctx.blacklisted_samples[bkey] = ctx.blacklisted_samples.get(bkey, 0) + count continue self._merge_preprocess_stats( ctx=ctx, @@ -1285,6 +1700,9 @@ def _merge_preprocess_stats( if cleaned.change_points: stats.change_points += cleaned.change_points ctx.change_points[query_id] = ctx.change_points.get(query_id, 0) + cleaned.change_points + for reason, count in cleaned.blacklisted_samples.items(): + bkey = (query_id, reason) + ctx.blacklisted_samples[bkey] = ctx.blacklisted_samples.get(bkey, 0) + count if cleaned.effective_lookback_seconds > 0: stats.effective_lookback_seconds = cleaned.effective_lookback_seconds @@ -1343,8 +1761,69 @@ async def _run_one_series( ctx.failures["fit_error"] = ctx.failures.get("fit_error", 0) + 1 return False _merge_fit_into_ctx(ctx, fit_result, query.id) + self._record_series_fit_successes( + fit_result=fit_result, + group_name=group.name, + query_id=query.id, + frame=frame, + ) return False + def _record_series_fit_successes( + self, + *, + fit_result: _FitResult, + group_name: str, + query_id: str, + frame: SeriesFrame, + ) -> None: + """Stamp ``forecast_data_staleness_seconds`` for every emitted model. + + Iterates ``fit_result.points`` to extract the unique set of + models that produced a snapshot row for this series — that's + the precise success signal we want the staleness clock to + track. Models present in ``fit_result.fit_durations`` but + absent from ``fit_result.points`` (e.g. a backtest-only fit) + intentionally don't reset the clock; the operator-visible + signal is "is this series being forecast", not "did any + fit run". + + The conditional refit family (model names suffixed + ``/conditional``) is filtered out — staleness tracks the + underlying model's emission cadence, and the conditional + emission rides on the same fit success. + + Called from both :meth:`_run_one_series` (the normal path) + and :meth:`_try_cold_start` (the cold-start path) so + synthesised cold-start forecasts reset the clock too. + """ + if not fit_result.points: + return + models: set[str] = set() + for point in fit_result.points: + model = point.labels.get("model", "") + if not model or model.endswith("/conditional"): + continue + models.add(model) + if not models: + return + series_key = _series_key_from_labels(frame.labels) + source_labels = {k: v for k, v in frame.labels.items() if k != "__name__"} + for model_name in models: + labels = { + **source_labels, + "id": query_id, + "group": group_name, + "model": model_name, + } + self._exporter.record_series_fit_success( + group=group_name, + query_id=query_id, + model_name=model_name, + series_key=series_key, + labels=labels, + ) + def _eligible_for_cold_start(self, query: QueryConfig) -> tuple[bool, str | None]: """Decide whether cold-start can run for ``query`` at all. @@ -1570,6 +2049,11 @@ async def _try_cold_start( "group": group.name, "model": cold_start_mod.COLD_START_MODEL_NAME, "cold_start": "true", + # Cold-start has no backtest, so we can't split the + # quality score by horizon — emit the aggregated row + # only (``horizon=""``) for label-shape consistency + # with the steady-state quality emission family. + "horizon": "", } quality_score = cold_start_mod.cold_start_quality_score( sibling_count=outcome.sibling_count, @@ -1580,6 +2064,12 @@ async def _try_cold_start( fit_result.quality.append(QualityPoint(labels=quality_label, score=quality_score)) _merge_fit_into_ctx(ctx, fit_result, query.id) + self._record_series_fit_successes( + fit_result=fit_result, + group_name=group.name, + query_id=query.id, + frame=frame, + ) qlog.info( "cold_start_fit_emitted", siblings=outcome.sibling_count, @@ -2020,6 +2510,20 @@ async def _fit_series( # noqa: PLR0912, PLR0915 rounder=cond_rounder, ) + # Forecast-to-forecast revision and model-disagreement gauges. + # Both need the per-emitted-model forecast frames + the accuracy + # horizons, so they ride next to ``capacity_inputs`` and must + # land before the next line clears that buffer. + self._record_revision_and_disagreement( + query=query, + group_name=group_name, + frame=frame, + inputs=list(result.capacity_inputs), + horizon_specs=accuracy_horizons, + auto_select=auto_select_enabled, + result=result, + ) + # Free the retained per-model dataframes — they can be MB-scale, # and ``_RunContext`` retains them across the per-series loop. result.capacity_inputs.clear() @@ -2106,6 +2610,7 @@ async def _fit_series( # noqa: PLR0912, PLR0915 curve_points=result.curve, factory=factory, result=result, + regressors=regressors_frame, ) # Band coverage: record one (inside / outside) observation per @@ -2137,6 +2642,7 @@ def _record_drift( curve_points: list[ForecastCurvePoint], factory: SeriesPointFactory, result: _FitResult, + regressors: Any | None = None, ) -> None: """Compute drift gauges for every emitted (series, model) pair. @@ -2153,6 +2659,13 @@ def _record_drift( (no prior fit, or both curves all-NaN) skip the gauge — the exporter's render path filters non-finite samples and the alert counter only ticks on finite >= threshold values. + + ``regressors`` (when provided) is the per-series regressor + DataFrame aligned to the fit window; the change-correlation + explainer reads each column's trailing finite value to + identify "what was loud at the drift moment" and to compute + the rolling Pearson correlation between drift_score and + regressor history. """ drift_cfg = query.emission.drift yhat_metric = f"{query.id}_forecast" @@ -2170,6 +2683,7 @@ def _record_drift( if not model_name: continue curves_by_model.setdefault(model_name, []).append(point.value) + current_regressor_values = _trailing_regressor_values(regressors) for model_name, curve in curves_by_model.items(): previous = self._drift_cache.get( group=group_name, @@ -2192,8 +2706,224 @@ def _record_drift( if math.isfinite(score): labels = {**factory.drift_labels(model_name), "method": drift_cfg.method} result.drift_points.append(DriftPoint(labels=labels, score=score)) + # Push the (score, regressor_values) observation into + # the per-(series, model) rolling window every fit — + # not only on threshold crossings — so the + # correlation has enough history to be meaningful by + # the time drift fires. + window = self._drift_window_for( + group_name=group_name, + query_id=query.id, + series_key=series_key, + model_name=model_name, + maxlen=drift_cfg.correlation_window, + ) + window.record(score, current_regressor_values) if drift_cfg.alert_threshold is not None and score >= drift_cfg.alert_threshold: result.drift_alerts += 1 + self._emit_drift_explanation( + group_name=group_name, + query=query, + series_key=series_key, + model_name=model_name, + score=score, + threshold=drift_cfg.alert_threshold, + window=window, + current_regressor_values=current_regressor_values, + ) + + def _drift_window_for( + self, + *, + group_name: str, + query_id: str, + series_key: str, + model_name: str, + maxlen: int, + ) -> drift_explainer_mod.DriftWindow: + """Lazily create the per-(series, model) rolling window.""" + key = (group_name, query_id, series_key, model_name) + window = self._drift_windows.get(key) + if window is None or window.maxlen != maxlen: + # Resize on reload: a new ``correlation_window`` config + # value rebuilds the window so the bound takes effect on + # the next fit. Prior samples drop because the historical + # ones were collected under the old (possibly larger) + # window. + window = drift_explainer_mod.DriftWindow(maxlen=maxlen) + self._drift_windows[key] = window + return window + + def _emit_drift_explanation( + self, + *, + group_name: str, + query: QueryConfig, + series_key: str, + model_name: str, + score: float, + threshold: float, + window: drift_explainer_mod.DriftWindow, + current_regressor_values: dict[str, float], + ) -> None: + """Log a ``forecast_drift_explained`` event and bump the counter. + + Always emits the structured log line — even when no suspects + emerge (no regressors configured, or none correlating / + loud at the moment) — so "drift fired with no candidate + explanation" is itself a visible operator signal rather than + a silent gap. + + Two contextual fields ride alongside the suspects so the + investigation hypothesis isn't limited to regressor changes: + + * ``last_config_reload_timestamp`` — surfaces "did someone + just reload?" alongside the drift event. A recent reload + is often itself the explanation. ``None`` when no reload + has ever been recorded (cold-boot before the first apply). + * ``change_point_count`` — cumulative input-side change-points + detected for the (group, query). Lets an operator + correlate "input-side regime shift detected" with + "output-side drift fired" without scraping a second + dashboard. + """ + explanation = drift_explainer_mod.explain(window, current_regressor_values) + reload_ts, _success, _failures = self._exporter._reload_state() + snap = self._exporter.snapshot_for(group_name) + change_point_count = ( + snap.change_points_cumulative.get(query.id, 0) if snap is not None else 0 + ) + logger.info( + "forecast_drift_explained", + group=group_name, + query=query.id, + model=model_name, + series_key=series_key, + drift_score=round(score, 6), + drift_threshold=threshold, + primary_suspect=explanation.primary_suspect, + top_value_suspect=explanation.top_value_suspect, + top_correlation_suspect=explanation.top_correlation_suspect, + top_regressor_correlation=( + round(explanation.top_regressor_correlation, 6) + if math.isfinite(explanation.top_regressor_correlation) + else None + ), + top_regressor_correlation_confidence=(explanation.top_regressor_correlation_confidence), + correlation_window_size=explanation.correlation_window_size, + regressor_values_at_drift={ + rid: round(v, 6) for rid, v in current_regressor_values.items() + }, + last_config_reload_timestamp=reload_ts if reload_ts > 0 else None, + change_point_count=change_point_count, + ) + self._exporter.increment_drift_explained( + group=group_name, + query=query.id, + primary_suspect=explanation.primary_suspect, + ) + + def _record_revision_and_disagreement( + self, + *, + query: QueryConfig, + group_name: str, + frame: SeriesFrame, + inputs: list[_CapacityInput], + horizon_specs: list[HorizonSpec], + auto_select: bool, + result: _FitResult, + ) -> None: + """Emit ``forecast_revision_magnitude`` + ``forecast_model_disagreement``. + + Revision: per-(series, model, horizon) — relative shift from the + previous fit's central forecast at the same horizon. Always + emitted when at least one model produced a forecast for an + accuracy horizon; the first fit per (series, model, horizon) + records ``NaN`` (filtered at render time) and seeds the cache. + + Disagreement: per-(series, horizon) — coefficient of variation + across emitted models' central forecasts at the same horizon. + Only emitted in explicit mode (``auto_select=False``) with 2+ + models contributing a finite yhat at that horizon; auto-select + modes only ever emit one model's forecast per series so the + gauge would be a constant zero. + """ + if not inputs or not horizon_specs: + return + series_key = _series_key_from_labels(frame.labels) + common = { + **{k: v for k, v in frame.labels.items() if k != "__name__"}, + "id": query.id, + "group": group_name, + } + # Pull each emitted model's per-horizon yhat once and reuse for + # both metrics. The frames retained on ``inputs`` are the raw + # (uncalibrated) per-model forecasts — exactly what the next + # fit will produce, so the revision number reflects the actual + # prediction change rather than a calibration-side shift. + per_horizon: dict[str, dict[str, float]] = {} + for capacity_input in inputs: + yhats = _extract_per_horizon_yhat(capacity_input.forecast, horizon_specs) + if yhats: + per_horizon[capacity_input.model_name] = yhats + if not per_horizon: + return + # Revision: compare current to previous cache; update cache. + for model_name, yhats in per_horizon.items(): + for horizon_label, current in yhats.items(): + previous = self._revision_cache.get( + group=group_name, + query=query.id, + series_key=series_key, + model=model_name, + horizon=horizon_label, + ) + self._revision_cache.put( + group=group_name, + query=query.id, + series_key=series_key, + model=model_name, + horizon=horizon_label, + value=current, + ) + magnitude = ( + revision_mod.compute_revision_magnitude(previous=previous, current=current) + if previous is not None + else float("nan") + ) + if not math.isfinite(magnitude): + continue + result.revision_points.append( + RevisionPoint( + labels={**common, "model": model_name, "horizon": horizon_label}, + magnitude=magnitude, + ) + ) + # Disagreement: per-(series, horizon) coefficient of variation + # across the emitted models. Suppressed in auto-select / ensemble + # modes because only one model's yhat is emitted per series, so + # the cross-model spread is undefined. + if auto_select or len(per_horizon) < _MIN_MODELS_FOR_DISAGREEMENT: + return + horizons_seen: set[str] = set() + for yhats in per_horizon.values(): + horizons_seen.update(yhats.keys()) + for horizon_label in horizons_seen: + values = [ + yhats[horizon_label] for yhats in per_horizon.values() if horizon_label in yhats + ] + if len(values) < _MIN_MODELS_FOR_DISAGREEMENT: + continue + score = _coefficient_of_variation(values) + if not math.isfinite(score): + continue + result.model_disagreement_points.append( + ModelDisagreementPoint( + labels={**common, "horizon": horizon_label}, + score=score, + ) + ) def _record_band_coverage( self, @@ -2704,7 +3434,7 @@ async def _timed(spec: ModelSpec) -> tuple[str, pd.DataFrame, float]: tasks = [_timed(spec) for spec in models] outcomes = await asyncio.gather(*tasks, return_exceptions=True) out: list[ForecastPoint] = [] - for outcome in outcomes: + for spec, outcome in zip(models, outcomes, strict=False): if isinstance(outcome, BaseException): logger.exception( "fit_failed_in_explicit_mode", @@ -2712,6 +3442,10 @@ async def _timed(spec: ModelSpec) -> tuple[str, pd.DataFrame, float]: query=query.id, exc_info=outcome, ) + # Per-model fit failure: record against the rolling success + # ratio so the model-health alert fires regardless of which + # mode (explicit/auto/ensemble) drove the fit. + self._record_fit_outcome(group=group_name, model=spec.name, success=False) continue model_name, forecast, fit_seconds = outcome result.fit_durations[model_name] = fit_seconds @@ -2722,6 +3456,7 @@ async def _timed(spec: ModelSpec) -> tuple[str, pd.DataFrame, float]: model_name=model_name, fit_seconds=fit_seconds, ) + self._record_fit_outcome(group=group_name, model=model_name, success=True) out.extend(factory.forecast_points(forecast, model_name, selection=None)) result.curve.extend(factory.forecast_curve_points(forecast, model_name, selection=None)) result.capacity_inputs.append( @@ -2781,15 +3516,24 @@ async def _auto_select_and_forecast( # params dict in that case so the fallback runs with stock defaults. spec = _resolve_spec(d.models, selection.model) t0 = time.monotonic() - forecast = await self._fit_one( - spec=spec, - df=df, - horizon=horizon, - freq=freq, - levels=levels, - season_length=season_length, - regressors=regressors, - ) + try: + forecast = await self._fit_one( + spec=spec, + df=df, + horizon=horizon, + freq=freq, + levels=levels, + season_length=season_length, + regressors=regressors, + ) + except Exception: + # Narrow on Exception (not BaseException) so a runner cancel + # — asyncio.CancelledError, KeyboardInterrupt — propagates + # without being recorded as a model-fit failure. TimeoutError + # inherits from Exception in 3.11+, so the success-ratio + # gauge still drops on real fit timeouts. + self._record_fit_outcome(group=group_name, model=selection.model, success=False) + raise fit_seconds = time.monotonic() - t0 result.fit_durations[selection.model] = fit_seconds self._record_slow_fit_if_needed( @@ -2799,6 +3543,7 @@ async def _auto_select_and_forecast( model_name=selection.model, fit_seconds=fit_seconds, ) + self._record_fit_outcome(group=group_name, model=selection.model, success=True) result.curve.extend( factory.forecast_curve_points(forecast, selection.model, selection=selection) ) @@ -2853,7 +3598,7 @@ async def _timed(spec: ModelSpec) -> tuple[str, pd.DataFrame, float]: outcomes = await asyncio.gather(*(_timed(spec) for spec in models), return_exceptions=True) component_forecasts: dict[str, pd.DataFrame] = {} out: list[ForecastPoint] = [] - for outcome in outcomes: + for spec, outcome in zip(models, outcomes, strict=False): if isinstance(outcome, BaseException): logger.exception( "fit_failed_in_ensemble_mode", @@ -2861,6 +3606,7 @@ async def _timed(spec: ModelSpec) -> tuple[str, pd.DataFrame, float]: query=query.id, exc_info=outcome, ) + self._record_fit_outcome(group=group_name, model=spec.name, success=False) continue model_name, forecast, fit_seconds = outcome result.fit_durations[model_name] = fit_seconds @@ -2871,6 +3617,7 @@ async def _timed(spec: ModelSpec) -> tuple[str, pd.DataFrame, float]: model_name=model_name, fit_seconds=fit_seconds, ) + self._record_fit_outcome(group=group_name, model=model_name, success=True) component_forecasts[model_name] = forecast out.extend(factory.forecast_points(forecast, model_name, selection=None)) result.curve.extend(factory.forecast_curve_points(forecast, model_name, selection=None)) @@ -3008,6 +3755,23 @@ def _record_slow_fit_if_needed( timeout_seconds=round(timeout_seconds, 3), ) + def _record_fit_outcome(self, *, group: str, model: str, success: bool) -> None: + """Record one per-(group, model) main-forecast fit attempt. + + Wraps the exporter's tracker so the runner doesn't reach into + private attributes. Counts both successes and failures (any + per-model exception or timeout) on the main forecast emission + path — explicit, auto-select, and ensemble modes are all + instrumented. The optional conditional-band refit is **not** + counted here on purpose: a model that succeeds on the + unconditional forecast but happens to fail the conditional + refit (e.g. regressor flat-hold degenerates the search) isn't + "the model is failing"; downstream ``ForecastModelFitFailing`` + alerts would misfire. Conditional failures land separately on + ``forecast_failures_total{reason="conditional_fit_*"}``. + """ + self._exporter.record_fit_outcome(group=group, model=model, success=success) + async def _run_backtests( self, *, diff --git a/forecaster/src/promforecast/runner/_state.py b/forecaster/src/promforecast/runner/_state.py index c9c3dcd..303242a 100644 --- a/forecaster/src/promforecast/runner/_state.py +++ b/forecaster/src/promforecast/runner/_state.py @@ -33,9 +33,11 @@ ForecastPoint, GroupSnapshot, GrowthRatePoint, + ModelDisagreementPoint, PreprocessStats, QualityPoint, RecommendedMaintenanceWindowPoint, + RevisionPoint, ThresholdEtaPoint, TimeToDrainPoint, WillBreachPoint, @@ -163,6 +165,16 @@ class _FitResult: # Drift alert: how many threshold crossings happened for this query # in this run. The runner folds it into the per-(query) total. drift_alerts: int = 0 + # Forecast-to-forecast revision gauges: per (series, model, horizon) + # magnitude of change between this fit's prediction and the prior + # fit's prediction at the same horizon. NaN on the first fit (no + # prior cached); the exporter drops NaN samples. + revision_points: list[RevisionPoint] = field(default_factory=list) + # Per-(series, horizon) model-disagreement gauge — coefficient of + # variation across configured models' central forecasts at the same + # horizon. Only populated in explicit mode with 2+ distinct emitted + # models; auto-select / cold-start leave this empty. + model_disagreement_points: list[ModelDisagreementPoint] = field(default_factory=list) @dataclass(frozen=True) @@ -285,6 +297,8 @@ class _RunContext: outliers_replaced: dict[tuple[str, str], int] = field(default_factory=dict) # Per-query change-point detection count for this run. change_points: dict[str, int] = field(default_factory=dict) + # Per-(query, reason) blacklist-window sample count for this run. + blacklisted_samples: dict[tuple[str, str], int] = field(default_factory=dict) # Per-(query, kind) emission clipping counter for this run. Merged # with the previous snapshot in ``run_group`` so the exposed counter # stays monotonic across runs. @@ -301,6 +315,10 @@ class _RunContext: drift_points: list[DriftPoint] = field(default_factory=list) # Per-(query) drift threshold-crossing counter for this run. drift_alerts: dict[str, int] = field(default_factory=dict) + # Per-(series, model, horizon) revision-magnitude gauges for this run. + revision_points: list[RevisionPoint] = field(default_factory=list) + # Per-(series, horizon) model-disagreement gauges for this run. + model_disagreement_points: list[ModelDisagreementPoint] = field(default_factory=list) # Per-query series count and run duration for the most recent run. # Used to populate the ``query`` label on # ``forecast_series_count`` and ``forecast_run_duration_seconds``. @@ -352,6 +370,8 @@ def _merge_fit_into_ctx(ctx: _RunContext, fit_result: _FitResult, query_id: str) ctx.drift_points.extend(fit_result.drift_points) if fit_result.drift_alerts: ctx.drift_alerts[query_id] = ctx.drift_alerts.get(query_id, 0) + fit_result.drift_alerts + ctx.revision_points.extend(fit_result.revision_points) + ctx.model_disagreement_points.extend(fit_result.model_disagreement_points) # Per-family inheritance registry. Each entry tells @@ -391,6 +411,8 @@ def _query_id_of(point: Any, by: _KeyBy) -> str: _InheritanceSpec("calibration_offsets", "calibration_offsets", "label"), _InheritanceSpec("conditional_calibration_offsets", "conditional_calibration_offsets", "label"), _InheritanceSpec("drift_points", "drift_points", "label"), + _InheritanceSpec("revision_points", "revision_points", "label"), + _InheritanceSpec("model_disagreement_points", "model_disagreement_points", "label"), # ``forecast_recommended_maintenance_window`` is a global metric # name (no per-id prefix); use the ``id`` label to scope inheritance. _InheritanceSpec( diff --git a/forecaster/src/promforecast/self_observability.py b/forecaster/src/promforecast/self_observability.py new file mode 100644 index 0000000..556b558 --- /dev/null +++ b/forecaster/src/promforecast/self_observability.py @@ -0,0 +1,361 @@ +"""Self-observability metrics for the forecaster itself. + +Three operator-facing families: + +* ``forecast_model_fit_success_ratio{group, model, window}`` — the fraction + of attempted fits that completed within ``safety.fit_timeout`` without + raising, computed over a rolling window. Two windows are emitted (``1h`` + and ``24h`` by default). A degrading model surfaces here before its bad + forecasts pollute downstream alerts. +* ``forecast_regressor_stability_score{group, query, regressor_id}`` — + a 0..1 score reflecting how much the regressor's recent value + distribution has drifted from its lookback distribution. Reuses the + same MAPE-style scoring as the forecast-drift module so the + number scales the same way operators already understand. +* ``forecast_group_memory_bytes{group}`` and + ``forecast_group_cpu_seconds_total{group}`` — per-group resource + attribution. CPU is the cumulative sum of per-model fit durations + (every fit that ran on the shared executor counts). Memory is a + best-effort snapshot of the in-flight per-group dataframes plus the + exporter's per-group snapshot, taken at the end of each group run + via ``sys.getsizeof`` summation — no psutil dependency to avoid + pulling in another wheel for one gauge. + +None of these touch the per-series cardinality budget; they're +group / model / query labelled and bounded by configuration size. +""" + +from __future__ import annotations + +import math +import threading +import time +from collections import deque +from dataclasses import dataclass +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from collections.abc import Iterable + + +# Default rolling windows for the success-ratio gauge. The chart's HPA +# threshold language ("ratio < 0.9 for 5m") works against either window; +# the 1h flavour is for spotting a model that just started failing, the +# 24h flavour is the slower trend. +DEFAULT_SUCCESS_WINDOWS_SECONDS: tuple[int, ...] = (3600, 86400) + +# Minimum series length required for a meaningful "recent vs. lookback" +# split. Below this the relative-shift metric is more noise than signal, +# so the stability score returns NaN and the exporter drops the row. +_MIN_STABILITY_SAMPLES = 5 + + +@dataclass(frozen=True) +class FitSuccessSample: + """One (success?, timestamp) observation in the rolling window.""" + + success: bool + timestamp: float + + +@dataclass(frozen=True) +class SuccessRatioPoint: + """Rendered point for ``forecast_model_fit_success_ratio``.""" + + group: str + model: str + window_label: str + ratio: float + + +@dataclass(frozen=True) +class RegressorStabilityPoint: + """Rendered point for ``forecast_regressor_stability_score``.""" + + group: str + query: str + regressor_id: str + score: float + + +@dataclass(frozen=True) +class GroupResourcePoint: + """One ``(group, memory_bytes, cpu_seconds_total)`` triple at render time.""" + + group: str + memory_bytes: float + cpu_seconds_total: float + + +class FitOutcomeTracker: + """Per-(group, model) rolling success/failure ledger. + + A ``deque`` per (group, model) keeps every observation in the + longest configured window; ``ratio_at`` returns the fraction of + successes that fall inside the asked-for window. Observations + older than the longest window are pruned at record time so the + deque stays bounded at the per-(group, model) operational + cardinality the runner already imposes. + """ + + def __init__(self, *, windows_seconds: Iterable[int] = DEFAULT_SUCCESS_WINDOWS_SECONDS) -> None: + self._windows_seconds: tuple[int, ...] = tuple(sorted(set(int(w) for w in windows_seconds))) + if not self._windows_seconds: + raise ValueError("FitOutcomeTracker requires at least one window") + self._max_window = max(self._windows_seconds) + self._lock = threading.Lock() + self._samples: dict[tuple[str, str], deque[FitSuccessSample]] = {} + + @property + def windows_seconds(self) -> tuple[int, ...]: + return self._windows_seconds + + def record(self, *, group: str, model: str, success: bool, now: float | None = None) -> None: + """Record one fit outcome. + + ``now`` defaults to ``time.time()`` and is overridable in tests + so the rolling-window expiry is deterministic without + ``freezegun`` or sleeps. + """ + ts = time.time() if now is None else now + with self._lock: + bucket = self._samples.setdefault((group, model), deque()) + bucket.append(FitSuccessSample(success=success, timestamp=ts)) + self._prune_locked(bucket, ts) + + def render(self, *, now: float | None = None) -> list[SuccessRatioPoint]: + """Snapshot the success ratio at every configured window. + + Returns one point per (group, model, window) that has at least + one observation inside the window. Empty windows are skipped — + a 0/0 ratio is not a "100% success" signal, but emitting NaN + would just confuse the alert path. + """ + ts = time.time() if now is None else now + out: list[SuccessRatioPoint] = [] + with self._lock: + for (group, model), bucket in self._samples.items(): + self._prune_locked(bucket, ts) + for window in self._windows_seconds: + ratio = _ratio_in_window(bucket, ts, window) + if ratio is None: + continue + out.append( + SuccessRatioPoint( + group=group, + model=model, + window_label=_format_window(window), + ratio=ratio, + ) + ) + return out + + def retain_groups(self, names: set[str]) -> None: + """Drop per-(group, model) buckets for groups removed by a reload.""" + with self._lock: + self._samples = { + key: bucket for key, bucket in self._samples.items() if key[0] in names + } + + def retain_models(self, live_models: set[str]) -> None: + """Drop per-(group, model) buckets for models no longer in the live config. + + Mirrors the staleness sweep's "model dropped from defaults.models" + cleanup: without this, a model removed from the config keeps its + success-ratio gauge on /metrics forever (no new observations + arrive to age out the existing samples, and the alert keeps + binding to the stale rows). + """ + with self._lock: + self._samples = { + key: bucket for key, bucket in self._samples.items() if key[1] in live_models + } + + def _prune_locked(self, bucket: deque[FitSuccessSample], now: float) -> None: + cutoff = now - self._max_window + while bucket and bucket[0].timestamp < cutoff: + bucket.popleft() + + +def _ratio_in_window( + bucket: deque[FitSuccessSample], now: float, window_seconds: int +) -> float | None: + cutoff = now - window_seconds + total = 0 + success = 0 + for sample in bucket: + if sample.timestamp < cutoff: + continue + total += 1 + if sample.success: + success += 1 + if total == 0: + return None + return success / total + + +class RegressorStabilityTracker: + """Per-(group, query, regressor_id) drift gauge of recent vs. lookback values. + + The runner pushes the last-seen regressor value series into + :meth:`record` after each fit; :meth:`render` computes a 0..1 + score where 1.0 means the recent tail matches the lookback head + perfectly and 0.0 means it has drifted past the saturating + threshold. The implementation uses the same MAPE-on-quantiles + flavour the forecast-drift module uses for its baseline + comparison so the operator-facing semantics agree across the two + metrics. + """ + + def __init__(self) -> None: + self._lock = threading.Lock() + self._scores: dict[tuple[str, str, str], float] = {} + + def record( + self, + *, + group: str, + query: str, + regressor_id: str, + values: list[float], + recent_fraction: float = 0.2, + ) -> None: + """Record a stability score derived from ``values``. + + The trailing ``recent_fraction`` of the series is compared + against the leading prefix; a small drift (≤ 10% relative) + yields a near-1.0 score, a 100%+ drift yields 0. Series too + short for a meaningful split short-circuit to ``NaN`` (which + the exporter drops) so a thin regressor doesn't spuriously + score as "unstable". + """ + score = compute_stability_score(values, recent_fraction=recent_fraction) + with self._lock: + self._scores[(group, query, regressor_id)] = score + + def render(self) -> list[RegressorStabilityPoint]: + with self._lock: + return [ + RegressorStabilityPoint(group=g, query=q, regressor_id=r, score=score) + for (g, q, r), score in self._scores.items() + ] + + def retain_groups(self, names: set[str]) -> None: + with self._lock: + self._scores = {key: score for key, score in self._scores.items() if key[0] in names} + + def retain_queries(self, live_pairs: set[tuple[str, str]]) -> None: + with self._lock: + self._scores = { + key: score for key, score in self._scores.items() if (key[0], key[1]) in live_pairs + } + + +def compute_stability_score( + values: list[float], *, recent_fraction: float = 0.2, saturation: float = 1.0 +) -> float: + """Map "recent vs. lookback" relative shift to a 0..1 score. + + Returns ``NaN`` when the input is too thin to split, when the + lookback baseline is degenerate (all-zero or non-finite), or + when every value is non-finite. Otherwise the relative shift is + ``|mean(recent) - mean(baseline)| / max(|mean(baseline)|, eps)`` + and the score is ``max(0, 1 - shift / saturation)``: a 10% + relative shift scores 0.9 at ``saturation=1.0``, a 100% shift + scores 0, and any larger shift saturates at 0. + """ + finite = [v for v in values if math.isfinite(v)] + if len(finite) < _MIN_STABILITY_SAMPLES: + return float("nan") + cut = max(1, int(len(finite) * recent_fraction)) + recent = finite[-cut:] + baseline = finite[:-cut] + if not baseline: + return float("nan") + baseline_mean = sum(baseline) / len(baseline) + if not math.isfinite(baseline_mean): + return float("nan") + recent_mean = sum(recent) / len(recent) + denom = max(abs(baseline_mean), 1e-9) + shift = abs(recent_mean - baseline_mean) / denom + if not math.isfinite(shift): + return float("nan") + saturation = max(saturation, 1e-9) + return max(0.0, 1.0 - shift / saturation) + + +class GroupResourceTracker: + """Cumulative CPU + most-recent memory snapshot per group. + + CPU is the sum of per-fit wall-clock seconds reported by the + runner's ``fit_durations`` accumulator. Memory is a cheap + ``sys.getsizeof``-based estimate of the per-group snapshot the + exporter is currently holding — best-effort, intended to surface + "which group's snapshot is dominant" rather than be an exact + process-level breakdown. + """ + + def __init__(self) -> None: + self._lock = threading.Lock() + self._cpu_total: dict[str, float] = {} + self._memory: dict[str, float] = {} + + def add_cpu(self, group: str, seconds: float) -> None: + if seconds <= 0 or not math.isfinite(seconds): + return + with self._lock: + self._cpu_total[group] = self._cpu_total.get(group, 0.0) + float(seconds) + + def set_memory(self, group: str, memory_bytes: float) -> None: + # Drop non-finite, negative, *and* zero values at write time so a + # missing-snapshot probe (which returns 0.0) doesn't populate the + # dict only to be filtered at render. Without this the dict + # accumulates phantom zero entries for every group that hasn't + # produced a snapshot yet. + if not math.isfinite(memory_bytes) or memory_bytes <= 0: + return + with self._lock: + self._memory[group] = float(memory_bytes) + + def render(self) -> list[GroupResourcePoint]: + with self._lock: + groups = set(self._cpu_total) | set(self._memory) + return [ + GroupResourcePoint( + group=group, + memory_bytes=self._memory.get(group, 0.0), + cpu_seconds_total=self._cpu_total.get(group, 0.0), + ) + for group in groups + ] + + def retain_groups(self, names: set[str]) -> None: + with self._lock: + self._cpu_total = {k: v for k, v in self._cpu_total.items() if k in names} + self._memory = {k: v for k, v in self._memory.items() if k in names} + + +def _format_window(seconds: int) -> str: + """Render a duration as the Prometheus-style ```` label.""" + if seconds <= 0: + return "0s" + if seconds % 86_400 == 0: + return f"{seconds // 86_400}d" + if seconds % 3_600 == 0: + return f"{seconds // 3_600}h" + if seconds % 60 == 0: + return f"{seconds // 60}m" + return f"{seconds}s" + + +__all__ = [ + "DEFAULT_SUCCESS_WINDOWS_SECONDS", + "FitOutcomeTracker", + "FitSuccessSample", + "GroupResourcePoint", + "GroupResourceTracker", + "RegressorStabilityPoint", + "RegressorStabilityTracker", + "SuccessRatioPoint", + "compute_stability_score", +] diff --git a/forecaster/src/promforecast/tuner.py b/forecaster/src/promforecast/tuner.py new file mode 100644 index 0000000..3514b4c --- /dev/null +++ b/forecaster/src/promforecast/tuner.py @@ -0,0 +1,626 @@ +"""Hyperparameter recommendation runner. + +The live forecaster pins ``season_length``, ``defaults.models``, and any +per-query overrides at config time and rarely revisits them. As the +underlying signal drifts the pinned values can quietly become wrong; +the only visible symptom is a slowly-degrading +``forecast_quality_score`` or a creeping ``forecast_accuracy_mape``, +both of which read as "the model is noisier" rather than "your config +is stale". + +This module evaluates a small grid of alternative hyperparameter +choices per (group, query, model) against the same backtest harness +the live runner uses (:mod:`promforecast.accuracy`) and emits one +:class:`RecommendedParam` row per configured + candidate pair whose +backtest MAPE beats the configured baseline by a non-trivial margin. + +Operators read the recommendations from a dedicated Grafana panel +(``promforecast_recommended_param`` is a snapshot-style gauge so a +stale recommendation drops as soon as the operator re-runs the +tuner after applying the change), pick the rows with the largest +``expected_mape_improvement_pct``, and apply them by editing the +config. The tuner never auto-applies — the operator stays in +control. + +Runs as a one-shot pod separate from the live forecaster (typical +shape: a weekly ``CronJob``, shipped via the +``promforecast-tuner`` sub-chart) so a long grid walk does not +contend with the scheduled fit budget. The live forecaster's +snapshot is never touched — the tuner is a pure read-side operation +against the long-term TSDB plus an opt-in push of the resulting +gauges via the configured ``sink.remote_write`` target. + +**Scope.** The shipped grid is intentionally small: one alternative +``season_length`` per query (auto-detected from the recent +periodogram), and the full configured ``defaults.models`` list as +the model-choice axis. A larger grid (damping parameters, per-model +hyperparams) is a future expansion; the metric shape and the +operator-visible workflow stay identical. +""" + +from __future__ import annotations + +import asyncio +import math +import sys +import time +from dataclasses import dataclass +from datetime import UTC, datetime, timedelta +from pathlib import Path +from typing import IO, TYPE_CHECKING + +import structlog + +from . import accuracy as accuracy_mod +from . import config as config_module +from . import models as model_registry +from . import preprocess as preprocess_mod +from .accuracy import overall_mape +from .config import Config, GroupConfig, QueryConfig, effective_season_length, effective_step +from .preview import _pandas_freq +from .sink import ForecastCurvePoint, ForecastSink, RemoteWriteSink, SinkError +from .source import PromSource + +if TYPE_CHECKING: + import pandas as pd + +logger = structlog.get_logger(__name__) + + +class TunerError(RuntimeError): + """Raised for any operator-visible tuner failure. + + Carries an exit code so ``main.cli`` can hand it straight to + ``sys.exit``. + """ + + def __init__(self, message: str, *, exit_code: int = 1) -> None: + super().__init__(message) + self.exit_code = exit_code + + +@dataclass(frozen=True) +class RecommendedParam: + """One recommendation row produced by a tuner run. + + Emitted as a sample of ``forecast_recommended_param`` — the value + carries the ``expected_mape_improvement_pct`` so dashboards can + sort by impact without parsing the label. + """ + + group: str + query: str + model: str + name: str + current_value: str + recommended_value: str + expected_mape_improvement_pct: float + + +@dataclass(frozen=True) +class TunerStats: + """Per-run summary returned for logs + the CLI footer.""" + + queries_evaluated: int + candidates_evaluated: int + recommendations_emitted: int + failed_queries: int + elapsed_seconds: float + outcome: str # ``ok | partial | error`` + + +def run_tuner( # noqa: PLR0911 + *, + config_path: Path, + group_selector: list[str] | None, + sink_url_override: str | None, + lookback_override: str | None = None, + min_improvement_pct: float = 5.0, + out: IO[str] | None = None, +) -> int: + """Top-level entry point used by ``main.cli``. + + Returns ``0`` when at least one recommendation was emitted or when + the run completed cleanly with no recommendations (current config + is already optimal); ``1`` for invalid inputs (config issues, + missing sink); ``2`` for TSDB / network failure during the data + fetch. + """ + stream = out if out is not None else sys.stdout + write = stream.write + try: + cfg = config_module.load(config_path) + except FileNotFoundError: + write(f"error: config not found: {config_path}\n") + return 1 + except Exception as exc: + write(f"error: config validation failed: {exc}\n") + return 1 + + sink_url = sink_url_override or cfg.sink.remote_write.url + if not sink_url: + write( + "error: no sink configured; either set sink.remote_write.url in the config " + "or pass --sink-url. The tuner pushes recommendations to the long-term TSDB " + "via the same path the live forecaster uses.\n" + ) + return 1 + + lookback_seconds: int | None = None + if lookback_override is not None: + parsed = config_module.parse_duration(lookback_override) + if not isinstance(parsed, timedelta): + write(f"error: invalid --lookback: {lookback_override!r}\n") + return 1 + lookback_seconds = int(parsed.total_seconds()) + + selected_groups = _select_groups(cfg.groups, group_selector) + if not selected_groups: + write( + "error: no groups selected; check --groups against the configured " + f"group names ({[g.name for g in cfg.groups]!r})\n" + ) + return 1 + + sink = RemoteWriteSink( + url=sink_url, + timeout_seconds=cfg.sink.remote_write.timeout.total_seconds(), + chunk_size=cfg.sink.remote_write.chunk_size, + headers=cfg.sink.remote_write.headers, + max_connections=cfg.sink.remote_write.max_connections, + max_keepalive_connections=cfg.sink.remote_write.max_keepalive_connections, + ) + source = PromSource( + url=cfg.datasource.url, + timeout_seconds=cfg.datasource.timeout.total_seconds(), + ) + + try: + stats = asyncio.run( + _run_async( + cfg=cfg, + source=source, + sink=sink, + groups=selected_groups, + lookback_override_seconds=lookback_seconds, + min_improvement_pct=min_improvement_pct, + ) + ) + except TunerError as exc: + write(f"error: {exc}\n") + return exc.exit_code + except Exception as exc: # pragma: no cover - defensive + logger.exception("tuner_unexpected_error") + write(f"error: {exc}\n") + return 2 + + write( + f"tuner: groups={len(selected_groups)} " + f"queries={stats.queries_evaluated} " + f"candidates={stats.candidates_evaluated} " + f"recommendations={stats.recommendations_emitted} " + f"failed={stats.failed_queries} " + f"elapsed_s={stats.elapsed_seconds:.1f} " + f"outcome={stats.outcome}\n" + ) + return 0 if stats.outcome != "error" else 2 + + +def _select_groups(groups: list[GroupConfig], selector: list[str] | None) -> list[GroupConfig]: + if not selector: + return list(groups) + wanted = set(selector) + return [g for g in groups if g.name in wanted] + + +async def _run_async( + *, + cfg: Config, + source: PromSource, + sink: ForecastSink, + groups: list[GroupConfig], + lookback_override_seconds: int | None, + min_improvement_pct: float, +) -> TunerStats: + start = time.time() + queries_evaluated = 0 + candidates_evaluated = 0 + failed_queries = 0 + recommendations: list[RecommendedParam] = [] + + # Wrap the evaluation + push in a single try/finally so the HTTP + # clients on the source and sink are reliably released even if an + # *unexpected* exception slips past ``_QueryEvaluationError`` + # (e.g. a hard import failure on the pandas branch, an asyncio + # cancellation). Without this, a one-shot tuner pod that crashed + # mid-walk would leak the keepalive pool until process exit. + try: + for group in groups: + for query in group.queries: + queries_evaluated += 1 + try: + cand_count, recs = await _evaluate_query( + cfg=cfg, + source=source, + group=group, + query=query, + lookback_override_seconds=lookback_override_seconds, + min_improvement_pct=min_improvement_pct, + ) + except _QueryEvaluationError as exc: + failed_queries += 1 + logger.warning( + "tuner_query_failed", + group=group.name, + query=query.id, + error=str(exc), + ) + continue + candidates_evaluated += cand_count + recommendations.extend(recs) + + # Always push the tuner run telemetry (timestamp + duration) — even + # an all-failing run produces tracking metrics the operator wants + # to see in Grafana. + try: + await _push_results( + sink=sink, + recommendations=recommendations, + outcome=_run_outcome(failed_queries, len(groups)), + elapsed_seconds=time.time() - start, + groups=[g.name for g in groups], + ) + except SinkError as exc: + logger.error("tuner_sink_push_failed", reason=exc.reason, error=str(exc)) + raise TunerError(f"sink push failed ({exc.reason}): {exc}", exit_code=2) from exc + finally: + # Best-effort cleanup. Swallow per-client errors so a transient + # close failure on one client doesn't mask the actual exception + # raised by the body of the try block. + try: + await sink.aclose() + except Exception: + logger.warning("tuner_sink_aclose_failed") + try: + await source.aclose() + except Exception: + logger.warning("tuner_source_aclose_failed") + + elapsed = time.time() - start + return TunerStats( + queries_evaluated=queries_evaluated, + candidates_evaluated=candidates_evaluated, + recommendations_emitted=len(recommendations), + failed_queries=failed_queries, + elapsed_seconds=elapsed, + outcome=_run_outcome(failed_queries, len(groups)), + ) + + +class _QueryEvaluationError(RuntimeError): + """Internal sentinel — a single-query failure that's worth logging + but mustn't take the whole tuner run down.""" + + +async def _evaluate_query( # noqa: PLR0912 + *, + cfg: Config, + source: PromSource, + group: GroupConfig, + query: QueryConfig, + lookback_override_seconds: int | None, + min_improvement_pct: float, +) -> tuple[int, list[RecommendedParam]]: + """Run the grid over one (group, query) tuple. Returns recommendations.""" + import pandas as pd # noqa: PLC0415 # heavy import: defer until needed + + defaults = cfg.defaults + step_td = effective_step(query, defaults) + lookback_seconds = ( + lookback_override_seconds + if lookback_override_seconds is not None + else int(defaults.lookback.total_seconds()) + ) + end = datetime.now(tz=UTC) + start = end - timedelta(seconds=lookback_seconds) + + try: + frames = await source.query_range( + promql=query.promql, + start=start, + end=end, + step_seconds=max(1, int(step_td.total_seconds())), + ) + except Exception as exc: + raise _QueryEvaluationError(f"source fetch failed: {exc}") from exc + + if not frames: + raise _QueryEvaluationError("source returned no series") + + # Take the first series as the representative — the tuner's job is + # to surface "your configured hyperparams are off for this query", + # not "this specific series of this query is off". A single series + # is sufficient to drive the comparative MAPE and keeps the per-run + # cost bounded; the operator applies any recommendation at the + # query level anyway. + frame = frames[0] + cleaned = preprocess_mod.preprocess_series( + timestamps=list(frame.timestamps), + values=list(frame.values), + step=step_td, + config=group.preprocess, + ) + if cleaned.dropped or len(cleaned.values) < defaults.min_points: + raise _QueryEvaluationError( + f"series too short after preprocessing ({len(cleaned.values)} " + f"points < min_points {defaults.min_points})" + ) + + df = pd.DataFrame( + { + "ds": cleaned.timestamps, + "y": cleaned.values, + "unique_id": ["series"] * len(cleaned.values), + } + ) + freq = _pandas_freq(max(1, int(step_td.total_seconds()))) + season_length = effective_season_length(query, defaults) + horizons = accuracy_mod.horizons_from_durations(defaults.accuracy.horizons, step_td) + if not horizons: + raise _QueryEvaluationError("no usable accuracy horizons configured") + + # Evaluate the baseline for each emitted model first so candidates + # have a number to beat. The baseline is the configured (model, + # season_length) tuple as it stands today. ``QueryConfig`` has no + # per-query ``models`` override today — every query inherits + # ``defaults.models`` — so the grid axis is the defaults list. + configured_model_names: list[str] = [spec.name for spec in defaults.models] + candidates_evaluated = 0 + recommendations: list[RecommendedParam] = [] + for model_name in configured_model_names: + baseline_mape = _fit_and_score( + model_name=model_name, + season_length=season_length, + df=df, + freq=freq, + horizons=horizons, + ) + if baseline_mape is None: + continue + candidates_evaluated += 1 + + # Candidate axis 1: season_length grid (x0.5, x2.0 of configured). + season_candidates = _season_length_grid(season_length) + for candidate_season in season_candidates: + cand_mape = _fit_and_score( + model_name=model_name, + season_length=candidate_season, + df=df, + freq=freq, + horizons=horizons, + ) + if cand_mape is None: + continue + candidates_evaluated += 1 + improvement = _improvement_pct(baseline_mape, cand_mape) + if improvement >= min_improvement_pct: + recommendations.append( + RecommendedParam( + group=group.name, + query=query.id, + model=model_name, + name="season_length", + current_value=str(season_length), + recommended_value=str(candidate_season), + expected_mape_improvement_pct=improvement, + ) + ) + + # Candidate axis 2: alternative model from the configured list. + # Compare each non-baseline model's MAPE; a candidate that beats + # the baseline by ``min_improvement_pct`` becomes a model-choice + # recommendation. + for alt_model in configured_model_names: + if alt_model == model_name: + continue + alt_mape = _fit_and_score( + model_name=alt_model, + season_length=season_length, + df=df, + freq=freq, + horizons=horizons, + ) + if alt_mape is None: + continue + candidates_evaluated += 1 + improvement = _improvement_pct(baseline_mape, alt_mape) + if improvement >= min_improvement_pct: + recommendations.append( + RecommendedParam( + group=group.name, + query=query.id, + model=model_name, + name="model", + current_value=model_name, + recommended_value=alt_model, + expected_mape_improvement_pct=improvement, + ) + ) + + return candidates_evaluated, recommendations + + +def _fit_and_score( + *, + model_name: str, + season_length: int, + df: pd.DataFrame, + freq: str, + horizons: list[accuracy_mod.HorizonSpec], +) -> float | None: + """Fit one candidate and return its overall MAPE. + + Returns ``None`` when the fit / backtest cannot produce a usable + score (model unknown, fit error, all-NaN MAPE on a flat series). + Callers treat ``None`` as "skip this candidate" and never as + "0 improvement"; an all-NaN MAPE is not the same as a perfect fit. + """ + try: + model = model_registry.build(model_name, season_length=season_length) + except model_registry.UnknownModelError: + return None + + max_steps = max((h.steps for h in horizons), default=0) + if max_steps < 1: + return None + try: + bt = accuracy_mod.backtest( + model=model, + df=df, + horizon_steps=max_steps, + freq=freq, + levels=[80], + ) + except Exception: + return None + if bt is None: + return None + test, predictions = bt + train = df.iloc[:-max_steps].reset_index(drop=True) + results = accuracy_mod.evaluate( + test=test, + predictions=predictions, + train=train, + season_length=season_length, + horizons=horizons, + model_name=model_name, + ) + mape = overall_mape(results) + if mape is None or not math.isfinite(mape): + return None + return mape + + +#: Hard ceiling on the candidate ``season_length`` the tuner will fit. +#: Anything above this is rejected as a runaway: a configured value of +#: 10000 (already very large for typical 5-min-step daily seasonality) +#: would otherwise produce a doubled candidate at 20000 and burn +#: per-fit time for no real value. +_MAX_CANDIDATE_SEASON_LENGTH = 10000 + + +def _season_length_grid(current: int) -> list[int]: + """Pick the candidate season_length values to try around ``current``. + + Bounded to two-element grid (half + double) so the tuner stays + cheap per query. A separate auto-seasonality story is the right + home for FFT-driven detection; the tuner's grid is deliberately + small enough to run weekly without dominating the cluster's CPU. + """ + candidates: list[int] = [] + half = max(2, current // 2) + double = current * 2 + if half != current: + candidates.append(half) + if double != current and double <= _MAX_CANDIDATE_SEASON_LENGTH: + candidates.append(double) + return candidates + + +def _improvement_pct(baseline: float, candidate: float) -> float: + """Return the percentage improvement of ``candidate`` over ``baseline``. + + Positive values mean the candidate's MAPE is lower (better). When + the baseline is zero (extremely unusual but possible) we fall back + to 0 — dividing by zero would either explode or sign-flip into a + meaningless recommendation. + """ + if baseline <= 0: + return 0.0 + return float(((baseline - candidate) / baseline) * 100.0) + + +def _run_outcome(failed_queries: int, group_count: int) -> str: + if failed_queries == 0: + return "ok" + # The tuner is best-effort; a single bad query in a multi-query + # group reports ``partial`` (any successful fit + any failure). + # An "all failed" run reports ``error`` so an alert on + # ``forecast_tuner_runs_total{outcome="error"}`` can detect the + # tuner being globally broken (network, auth) rather than a single + # query being unfit for the grid. + if group_count == 0: + return "error" + return "partial" + + +async def _push_results( + *, + sink: ForecastSink, + recommendations: list[RecommendedParam], + outcome: str, + elapsed_seconds: float, + groups: list[str], +) -> None: + """Push the recommendation + run-telemetry rows through ``sink``. + + Uses :class:`ForecastCurvePoint` to ride the existing sink's + Prometheus text-format encoder; each row carries a timestamp at + "now" so the recommendation is treated as the snapshot for the + most recent tuner run. + """ + timestamp_ms = int(time.time() * 1000) + points: list[ForecastCurvePoint] = [] + for rec in recommendations: + points.append( + ForecastCurvePoint( + metric="forecast_recommended_param", + labels={ + "group": rec.group, + "query": rec.query, + "model": rec.model, + "name": rec.name, + "current_value": rec.current_value, + "recommended_value": rec.recommended_value, + }, + value=rec.expected_mape_improvement_pct, + timestamp_ms=timestamp_ms, + ) + ) + # Tuner-internal telemetry. The tuner is a stateless one-shot + # process — pushing a counter sample of ``1.0`` from every run + # produces a flat ``1.0, 1.0, 1.0, ...`` series in the TSDB whose + # ``rate()`` is zero, which is not useful. Instead the tuner emits + # two gauges keyed on the run timestamp: + # + # * ``forecast_tuner_last_run_timestamp_seconds{group, outcome}`` — + # unix timestamp of the most recent run. Operators alert on + # staleness (``time() - max by (group) (forecast_tuner_last_run_timestamp_seconds) > 8d``) + # and split-route on ``outcome`` for "tuner is failing". + # * ``forecast_tuner_last_run_duration_seconds{group}`` — + # wall-clock duration of the most recent run. + # + # This matches the snapshot/pull pattern the rest of the codebase + # uses for last-run telemetry and is the right shape for a + # stateless one-shot pusher. + timestamp_seconds = timestamp_ms / 1000.0 + for group_name in groups: + points.append( + ForecastCurvePoint( + metric="forecast_tuner_last_run_timestamp_seconds", + labels={"group": group_name, "outcome": outcome}, + value=timestamp_seconds, + timestamp_ms=timestamp_ms, + ) + ) + points.append( + ForecastCurvePoint( + metric="forecast_tuner_last_run_duration_seconds", + labels={"group": group_name}, + value=elapsed_seconds, + timestamp_ms=timestamp_ms, + ) + ) + if not points: + return + await sink.write(points) diff --git a/forecaster/src/promforecast/validate.py b/forecaster/src/promforecast/validate.py index 1b15002..63b2ca8 100644 --- a/forecaster/src/promforecast/validate.py +++ b/forecaster/src/promforecast/validate.py @@ -37,6 +37,11 @@ Writer = Callable[[str], None] +# Minimum number of configured models required for the disagreement +# metric to project rows in the cardinality estimate; matches the +# runtime gate in :mod:`promforecast.runner`. +_MIN_MODELS_FOR_DISAGREEMENT_ESTIMATE = 2 + @dataclass(frozen=True) class CardinalityEstimate: @@ -93,6 +98,32 @@ class CardinalityEstimate: # Incident recurrence heatmap — fixed 7 (days) * 24 (hours) per # opted-in query with ``data_profile: events``. incident_recurrence_lines: int + # Forecast-to-forecast revision: per-(series, model, horizon) + # whenever ``accuracy.horizons`` is configured. Independent of + # ``accuracy.evaluate`` because the metric reads the central + # forecast at each horizon directly, no backtest needed. + revision_lines: int + # Model-disagreement: per-(series, horizon) in explicit mode only. + # Same gate-shape as ``revision_lines``. + disagreement_lines: int + # Threshold action-cost: one config-derived gauge row per + # threshold with ``action_cost`` configured. Independent of model + # / series fan-out — the count is the total of opted-in + # thresholds across the live config. + action_cost_lines: int + # Service-level forecast health composite: bounded by the unique + # combinations of ``aggregate_by`` label values across the live + # quality_points. We can't know that combinatoric without running, + # so we estimate the upper bound at ``series_cap * queries`` (one + # bucket per series in the worst case, when every series has a + # distinct ``aggregate_by`` tuple) and let the operator narrow + # via ``quality_floor`` if it overshoots. + service_health_lines: int + # Per-workload right-sizing: up to three rows per configured + # workload (recommended + optional current + optional savings_pct). + # Bounded by the operator's configured workload count; the actual + # row count is smaller when ``current_promql`` is unset. + resource_sizing_lines: int operational_lines: int @property @@ -118,16 +149,22 @@ def total(self) -> int: + self.recommended_window_lines + self.deploy_risk_lines + self.incident_recurrence_lines + + self.revision_lines + + self.disagreement_lines + + self.action_cost_lines + + self.service_health_lines + + self.resource_sizing_lines + self.operational_lines ) -def run_validate( # noqa: PLR0915 +def run_validate( # noqa: PLR0911, PLR0915 config_path: Path, *, dry_run: bool = False, datasource_url_override: str | None = None, strict_cardinality: bool = False, + estimate_cost: bool = False, out: IO[str] | None = None, ) -> int: """Validate ``config_path`` and optionally run a sample fit. @@ -141,6 +178,13 @@ def run_validate( # noqa: PLR0915 on a config PR can reject configs whose projected emission exceeds ``safety.max_total_series`` before the change ships. The knob is opt-in so existing callers keep their advisory behaviour. + + ``estimate_cost`` runs a cheap synthetic-data fit per (group, + configured model) pair and prints the expected per-refresh CPU / + fit-duration totals. The estimator is deliberately approximate + (synthetic data doesn't capture every real-world fit-time + distribution); the goal is order-of-magnitude headroom checks at + review time rather than a benchmark. """ stream: IO[str] = out if out is not None else sys.stdout @@ -192,6 +236,11 @@ def err(line: str) -> None: write(f" recommended wins {estimate.recommended_window_lines:>8}") write(f" deploy risk lines {estimate.deploy_risk_lines:>8}") write(f" incident recur {estimate.incident_recurrence_lines:>8}") + write(f" revision lines {estimate.revision_lines:>8}") + write(f" disagreement lines {estimate.disagreement_lines:>8}") + write(f" action cost lines {estimate.action_cost_lines:>8}") + write(f" service health {estimate.service_health_lines:>8}") + write(f" resource sizing {estimate.resource_sizing_lines:>8}") write(f" operational lines {estimate.operational_lines:>8}") write(f" total {estimate.total:>8}") rc = _handle_cardinality_overshoot( @@ -204,6 +253,14 @@ def err(line: str) -> None: if rc != 0: return rc + if estimate_cost: + write("") + try: + _run_cost_estimate(cfg, write) + except Exception as exc: + err(f"error: cost estimate failed: {exc}") + return 2 + if not dry_run: return 0 @@ -301,7 +358,18 @@ def _estimate_cardinality(cfg: Config) -> CardinalityEstimate: # noqa: PLR0912, if cfg.defaults.emission.deviation else 0 ) - quality_lines = queries * series_cap * emitted_models if cfg.defaults.emission.quality else 0 + # Per-horizon segmentation: each (series, model) emits the aggregated + # ``horizon=""`` row plus one row per configured accuracy horizon. + # The aggregate row is always emitted (preserves the historical + # ``horizon``-less shape so existing alerts continue to bind); the + # per-horizon rows only show up when ``accuracy.evaluate: true`` + # produces backtest results. + per_horizon_quality_factor = 1 + (n_horizons if cfg.defaults.accuracy.evaluate else 0) + quality_lines = ( + queries * series_cap * emitted_models * per_horizon_quality_factor + if cfg.defaults.emission.quality + else 0 + ) # Contribution: only emitted for queries that actually configure # regressors; the count is the *worst case* assuming every series of # those queries fans out one row per regressor. Calendar regressors are @@ -402,6 +470,19 @@ def _estimate_cardinality(cfg: Config) -> CardinalityEstimate: # noqa: PLR0912, preprocess_lines += queries * n_outlier_methods if any_changepoint_on: preprocess_lines += queries + # Blacklist windows emit ``forecast_input_blacklisted_samples_total`` + # per (group, query, reason) where ``reason`` is bounded by the + # operator's config (one distinct reason per configured window, plus + # the empty-string reason for unreasoned windows). The per-query + # multiplier is the count of distinct reasons across the group's + # blacklist windows — accounted only on groups that actually + # configured one. + for group in cfg.groups: + if not group.preprocess.blacklist_windows: + continue + distinct_reasons = {w.reason for w in group.preprocess.blacklist_windows} + if distinct_reasons: + preprocess_lines += len(group.queries) * len(distinct_reasons) # Count-aware emission clipping: a per-(query, kind=negative|rounded) # counter, bounded to 2 lines per query. Charged only against queries @@ -443,6 +524,24 @@ def _estimate_cardinality(cfg: Config) -> CardinalityEstimate: # noqa: PLR0912, drift_lines += series_cap * emitted_models drift_lines += 1 # the per-(group, query) alert counter row + # Revision magnitude: per-(series, model, horizon). The runtime + # only requires ``accuracy.horizons`` to be configured — backtest + # ``evaluate`` is *not* a gate, because the metric reads the + # central forecast at each horizon directly, without needing + # backtest results. First-fit rows are NaN and dropped at render, + # so the worst case is the steady-state row count. + revision_lines = queries * series_cap * emitted_models * n_horizons if n_horizons else 0 + # Model disagreement: per-(series, horizon). Only emitted in + # explicit mode (auto_select == False) with 2+ configured models. + # Same gate-shape as revision: needs configured horizons, not + # ``accuracy.evaluate``. ``auto_select == "ensemble"`` counts as + # active (only the ensemble row is emitted, so there's no + # cross-model spread to score). + auto_select_active = cfg.defaults.accuracy.auto_select is not False + disagreement_lines = 0 + if n_horizons and not auto_select_active and n_models >= _MIN_MODELS_FOR_DISAGREEMENT_ESTIMATE: + disagreement_lines = queries * series_cap * n_horizons + # Empirical band coverage. ``forecast_band_empirical_coverage`` is # per (series, model, level, window) for opted-in queries. band_coverage_lines = 0 @@ -491,6 +590,40 @@ def _estimate_cardinality(cfg: Config) -> CardinalityEstimate: # noqa: PLR0912, if q.data_profile == "events": incident_recurrence_lines += event_buckets + # Threshold action-cost: one config-derived line per threshold + # with ``action_cost`` configured. Bounded by the operator's + # configured threshold count; no model / series fan-out. + action_cost_lines = 0 + for group in cfg.groups: + for q in group.queries: + action_cost_lines += sum(1 for t in q.thresholds if t.action_cost is not None) + + # Service-level forecast health composite. Worst-case upper bound: + # one bucket per quality_point (which is per (series, model, + # horizon)). The realistic count is far smaller (operators + # aggregate by 1-2 labels with low cardinality) but we can't + # introspect the live label values from config alone. + service_health_lines = 0 + if cfg.governance.health_score is not None: + # ``per_horizon_quality_factor`` was computed earlier — one + # quality row per emitted (model, horizon) per series. Upper + # bound is the quality_lines total itself. + service_health_lines = quality_lines + + # Right-sizing: up to three rows per configured workload entry — + # the primary recommended gauge plus the optional current / + # savings_pct siblings (only emitted when ``current_promql`` is set + # on the entry). Bounded by the operator's configured workload count. + resource_sizing_lines = 0 + rs_cfg = cfg.recommendations.resource_sizing + if rs_cfg.enabled: + for workload in rs_cfg.workloads: + # One row for the recommended gauge plus up to two more + # for the optional sibling pair. + resource_sizing_lines += 1 + if workload.current_promql is not None: + resource_sizing_lines += 2 # current + savings_pct + # Operational: 7 gauges + N reasons per group; we ballpark 7 per group. # The runner also emits per-(group, query) rows of # ``forecast_query_run_duration_seconds`` and @@ -524,6 +657,11 @@ def _estimate_cardinality(cfg: Config) -> CardinalityEstimate: # noqa: PLR0912, recommended_window_lines=recommended_window_lines, deploy_risk_lines=deploy_risk_lines, incident_recurrence_lines=incident_recurrence_lines, + revision_lines=revision_lines, + disagreement_lines=disagreement_lines, + action_cost_lines=action_cost_lines, + service_health_lines=service_health_lines, + resource_sizing_lines=resource_sizing_lines, operational_lines=operational_lines, ) @@ -703,3 +841,169 @@ def _pandas_freq_seconds(seconds: int) -> str: if seconds % 60 == 0: return f"{seconds // 60}min" return f"{seconds}s" + + +@dataclass(frozen=True) +class CostEstimate: + """One group's projected per-refresh resource budget. + + All numbers are best-effort: the per-(group, model) fit time is + measured against a synthetic sinusoid + trend series, which is + smoother than typical production input but captures the AutoARIMA + /AutoETS / SeasonalNaive cost order-of-magnitude. The estimator's + purpose is to catch *config-level* surprises (e.g. AutoARIMA on + 14 days of 5-minute data over 500 series) at review time, not to + benchmark a deployment. + """ + + group: str + refresh_interval_seconds: float + estimated_series: int + per_fit_seconds_by_model: dict[str, float] + total_fit_seconds_per_refresh: float + peak_memory_bytes: int + + +def _run_cost_estimate(cfg: Config, write: Writer) -> None: + """Print a per-group fit-time + memory budget.""" + write("Estimated runtime cost (synthetic-data fit per model):") + write( + " Note: estimator is order-of-magnitude only — synthetic input doesn't " + "capture every real fit-time shape. Treat as a sanity check, not a benchmark." + ) + write("") + estimates = list(_estimate_runtime_cost(cfg)) + if not estimates: + write(" (no groups to estimate)") + return + total_seconds = 0.0 + for est in estimates: + write(f" group {est.group}:") + write(f" refresh_interval {est.refresh_interval_seconds:>10.1f}s") + write(f" estimated series {est.estimated_series:>10d}") + for model_name, seconds in sorted(est.per_fit_seconds_by_model.items()): + write(f" per-fit {model_name:<20}{seconds:>10.3f}s") + write(f" total fit time / refresh {est.total_fit_seconds_per_refresh:>10.2f}s") + write(f" peak memory (tracemalloc){est.peak_memory_bytes:>10d} bytes") + if ( + est.refresh_interval_seconds > 0 + and est.total_fit_seconds_per_refresh > est.refresh_interval_seconds + ): + write( + " WARN: projected fit time exceeds refresh_interval — " + "the group cannot keep up at the configured cadence." + ) + total_seconds += est.total_fit_seconds_per_refresh + write("") + write(f" total fit time across groups (per refresh): {total_seconds:.2f}s") + + +def _estimate_runtime_cost(cfg: Config) -> list[CostEstimate]: + """Time each configured model against synthetic data; project per-refresh totals. + + ``defaults.models`` is config-global, so each model is fit exactly + once against the synthetic data and the resulting timings are reused + across every group. Without this, an N-group x M-model config would + run NxM identical fits where M would suffice — turning a 30-second + validate into multiple minutes on realistic configs. + + Per-fit peak memory is captured via :mod:`tracemalloc` (Python's + allocator-level peak), which gives an honest per-block number + instead of the process-wide RSS that monotonically grows across + fits and makes the second group's "peak" always meaningless. + """ + import tracemalloc # noqa: PLC0415 + + import numpy as np # noqa: PLC0415 + import pandas as pd # noqa: PLC0415 + + from . import models as model_registry # noqa: PLC0415 + + series_cap = cfg.safety.max_series_per_query + # Synthetic dataset: 720 hourly points (~30d) — enough for AutoARIMA + # to do real work but small enough to keep the estimator cheap. The + # signal is a daily sinusoid + linear trend + light noise so the + # statsforecast fit isn't trivially degenerate. + n_points = 720 + rng = np.random.default_rng(seed=42) + timestamps = pd.date_range("2024-01-01", periods=n_points, freq="1h") + base = ( + np.sin(np.arange(n_points) * 2 * math.pi / 24) * 10.0 + + np.arange(n_points) * 0.01 + + rng.normal(0, 0.5, n_points) + + 100.0 + ) + df = pd.DataFrame({"unique_id": ["s"] * n_points, "ds": timestamps, "y": base}) + levels = list(cfg.defaults.confidence_levels) + horizon = 24 + freq = "1h" + + # Fit each model once and capture the per-fit wall-clock and the + # tracemalloc peak attributable to that fit alone. + per_fit_times: dict[str, float] = {} + per_fit_memory: dict[str, int] = {} + tracemalloc.start() + try: + for spec in cfg.defaults.models: + try: + model = model_registry.build( + spec.name, + season_length=cfg.defaults.season_length, + params=spec.params, + ) + except Exception: + # Model wasn't registered (typo, missing plug-in). Skip + # without crashing — validate also reports config-level + # errors elsewhere. + continue + tracemalloc.reset_peak() + t0 = time.monotonic() + try: + _ = model.fit_predict(df=df, horizon=horizon, freq=freq, levels=levels) + except Exception: + # Real fit failures are noisy — store NaN so the + # operator sees "couldn't measure" rather than a + # spurious zero. + per_fit_times[spec.name] = float("nan") + per_fit_memory[spec.name] = 0 + continue + per_fit_times[spec.name] = time.monotonic() - t0 + _, peak = tracemalloc.get_traced_memory() + per_fit_memory[spec.name] = int(peak) + finally: + tracemalloc.stop() + + out: list[CostEstimate] = [] + for group in cfg.groups: + # Refresh interval can come from the per-query override; if no + # queries exist we fall back to the server default. + if group.queries: + refresh_interval = config_module.effective_refresh_interval( + group.queries[0], cfg.server + ).total_seconds() + else: + refresh_interval = cfg.server.refresh_interval.total_seconds() + # Each query fits every configured model once per series; in + # auto-select mode only the winner emits but every candidate is + # still backtested. The estimate sums per-fit cost across + # configured models and projects to the per-query series cap. + per_query_fit_time = sum(v for v in per_fit_times.values() if math.isfinite(v)) + queries = len(group.queries) + total = per_query_fit_time * queries * series_cap + # Peak memory attributable to a fit in this group: the max + # per-fit allocation peak across the configured models. Since + # ``defaults.models`` is global, the per-model peak is the same + # in every group, but the field shape preserves the per-group + # report so a downstream UI can render it per row. + peak_memory = max(per_fit_memory.values(), default=0) + out.append( + CostEstimate( + group=group.name, + refresh_interval_seconds=refresh_interval, + estimated_series=series_cap * queries, + per_fit_seconds_by_model=dict(per_fit_times), + total_fit_seconds_per_refresh=total, + peak_memory_bytes=peak_memory, + ) + ) + return out diff --git a/forecaster/src/promforecast/warmup.py b/forecaster/src/promforecast/warmup.py new file mode 100644 index 0000000..21978c1 --- /dev/null +++ b/forecaster/src/promforecast/warmup.py @@ -0,0 +1,441 @@ +"""Forecast warm-up status. + +Reports per-(group, query) "what is still loading?" status for a fresh +install. The output answers the four questions an operator asks during +the first cycle after ``helm install``: + +* How many lookback samples does this query already have? +* How many does it need before the runner will fit a model? +* What is currently blocking it (insufficient lookback, missing + regressor, ...)? +* Roughly how long until the first usable forecast? + +The status is computed on demand — there is no background poll. Each +call issues one cheap PromQL probe per configured query (a small +``query_range`` over the lookback window, returning only the +timestamps the source actually has) plus one additional probe per +required custom regressor when the primary's point count has cleared +``min_points``. Required-regressor probes fan out concurrently with +the primary and each other. + +The endpoint is opt-in via ``server.expose_warmup_endpoint`` because +even the "one cheap probe per query" cost is wasteful on the steady- +state path; the operator workflow is "hit it a few times during the +first refresh, then never again". For the same reason there is no +caching layer — a stale cached answer is more confusing than a fresh +500ms probe. +""" + +from __future__ import annotations + +import asyncio +import math +from dataclasses import dataclass +from datetime import UTC, datetime, timedelta +from typing import TYPE_CHECKING, Any, Literal + +import structlog + +from .config import Config, GroupConfig, QueryConfig, RegressorConfig, effective_step + +if TYPE_CHECKING: + from .source import PromSource, SeriesFrame + +logger = structlog.get_logger(__name__) + + +# Bounded enum of ``blocked_by`` reasons the runner can plausibly +# report from a probe. ``regressor:`` is the templated case — we +# pin the prefix here so callers can match on it without scraping. +BlockedBy = Literal[ + "none", + "min_points", + "lookback", + "change_point", + "discovery", +] + + +@dataclass(frozen=True) +class QueryWarmupStatus: + """Per-query warm-up status row in the warmup report.""" + + id: str + status: Literal["ready", "warming", "error"] + points_available: int + points_needed: int + blocked_by: str # one of ``BlockedBy`` or ``regressor:`` + eta_seconds: float + + def to_dict(self) -> dict[str, Any]: + """Render as a JSON-friendly dict.""" + return { + "id": self.id, + "status": self.status, + "points_available": self.points_available, + "points_needed": self.points_needed, + "blocked_by": self.blocked_by, + "eta_seconds": self.eta_seconds, + } + + +@dataclass(frozen=True) +class GroupWarmupStatus: + """Per-group warm-up status.""" + + name: str + queries: list[QueryWarmupStatus] + + def to_dict(self) -> dict[str, Any]: + return { + "name": self.name, + "queries": [q.to_dict() for q in self.queries], + } + + +@dataclass(frozen=True) +class WarmupReport: + """Full warm-up report.""" + + groups: list[GroupWarmupStatus] + + def to_dict(self) -> dict[str, Any]: + return {"groups": [g.to_dict() for g in self.groups]} + + +async def compute_warmup( + *, + config: Config, + source: PromSource, + query_timeout_seconds: float | None = None, +) -> WarmupReport: + """Probe every configured query and return its warm-up status. + + Fan-out is concurrent across queries — the per-query probe is a + cheap ``query_range`` that returns at most ``lookback / step`` + timestamps, so total wall-time is bounded by ``max(per-probe + latency)`` rather than the sum. + + ``query_timeout_seconds`` overrides the per-probe timeout; without + it the call inherits ``config.safety.query_timeout``. Probe + failures degrade the row to ``status: "error"`` so a single + bad PromQL never sinks the rest of the report. + """ + timeout = ( + query_timeout_seconds + if query_timeout_seconds is not None + else config.safety.query_timeout.total_seconds() + ) + tasks = [ + _probe_group(config=config, group=group, source=source, query_timeout_seconds=timeout) + for group in config.groups + ] + groups = await asyncio.gather(*tasks) if tasks else [] + return WarmupReport(groups=list(groups)) + + +async def _probe_group( + *, + config: Config, + group: GroupConfig, + source: PromSource, + query_timeout_seconds: float, +) -> GroupWarmupStatus: + """Probe every query in ``group`` concurrently.""" + tasks = [ + _probe_query( + config=config, + query=query, + source=source, + query_timeout_seconds=query_timeout_seconds, + ) + for query in group.queries + ] + queries = await asyncio.gather(*tasks) + return GroupWarmupStatus(name=group.name, queries=list(queries)) + + +async def _probe_query( # noqa: PLR0911 + *, + config: Config, + query: QueryConfig, + source: PromSource, + query_timeout_seconds: float, +) -> QueryWarmupStatus: + """Run one probe and translate the result into a ``QueryWarmupStatus``. + + Event-profile queries (``data_profile: events``) and queries with + discovery variables take separate runtime paths in the runner; + here we report them as ``ready`` with no probe — they don't use + ``min_points`` and a substitution-failing template would just + surface as a probe error anyway. + """ + eff_step = effective_step(query, config.defaults) + step_seconds = max(1, int(eff_step.total_seconds())) + min_points = config.defaults.min_points + + if query.data_profile == "events": + # Event-profile queries don't have a min_points contract; they + # produce the recurrence heatmap on every refresh once the + # counter has at least one tick. + return QueryWarmupStatus( + id=query.id, + status="ready", + points_available=0, + points_needed=0, + blocked_by="none", + eta_seconds=0.0, + ) + + if query.discover: + # Templated queries depend on the discovery variable resolving + # before we can probe the rendered string. Probing the raw + # template (with the substitution placeholder still in it) + # would just surface as a probe error and confuse the + # operator, so we short-circuit. Surface the dependency + # explicitly via ``blocked_by="discovery"`` rather than + # ``none``: a templated query reporting ``ready`` + ``none`` + # implies "actively forecasting" which is misleading until + # discovery has resolved. The status is still ``ready`` + # (callers that gate on status alone — including the CLI's + # exit-code loop — continue to treat templated queries as + # non-blocking) but the operator can read ``blocked_by`` as + # "I can't probe this; check the discovery counters". + return QueryWarmupStatus( + id=query.id, + status="ready", + points_available=0, + points_needed=0, + blocked_by="discovery", + eta_seconds=0.0, + ) + + # Probe the primary PromQL with a short query_range. Reusing the + # configured lookback gives us the most-points-the-fit-would-see + # number without a second query path. + end = datetime.now(tz=UTC) + start = end - config.defaults.lookback + try: + frames = await asyncio.wait_for( + source.query_range(query.promql, start, end, step_seconds), + timeout=query_timeout_seconds, + ) + except TimeoutError: + logger.warning("warmup_probe_timeout", query=query.id) + return _error_status(query.id, min_points) + except Exception as exc: + logger.warning("warmup_probe_failed", query=query.id, error=str(exc)) + return _error_status(query.id, min_points) + + points_available = _points_in_longest_series(frames) + + if points_available < min_points: + eta = _estimate_eta( + points_available=points_available, + points_needed=min_points, + step_seconds=step_seconds, + ) + return QueryWarmupStatus( + id=query.id, + status="warming", + points_available=points_available, + points_needed=min_points, + blocked_by="min_points" if points_available > 0 else "lookback", + eta_seconds=eta, + ) + + # Primary has enough points — check required PromQL regressors next. + # A required regressor that the TSDB can't yet resolve (no samples in + # the lookback) would cause the runner's per-series fit to skip with + # ``forecast_failures_total{reason="required_regressor"}``. The + # warmup endpoint exists to give operators that signal *before* the + # fit attempt, so we issue one probe per required PromQL regressor + # and surface the first one that comes back empty as the block. + blocking_regressor = await _probe_required_regressors( + query=query, + source=source, + step_seconds=step_seconds, + lookback=config.defaults.lookback, + query_timeout_seconds=query_timeout_seconds, + ) + if blocking_regressor is not None: + return QueryWarmupStatus( + id=query.id, + status="warming", + points_available=points_available, + points_needed=min_points, + blocked_by=f"regressor:{blocking_regressor}", + eta_seconds=float(step_seconds), + ) + + return QueryWarmupStatus( + id=query.id, + status="ready", + points_available=points_available, + points_needed=min_points, + blocked_by="none", + eta_seconds=0.0, + ) + + +async def _probe_required_regressors( + *, + query: QueryConfig, + source: PromSource, + step_seconds: int, + lookback: timedelta, + query_timeout_seconds: float, +) -> str | None: + """Probe each required PromQL regressor; return the first empty one's id. + + Calendar / holiday regressors are skipped — they don't query the + TSDB and so can't be "missing data". Only ``custom`` regressors + with a PromQL string are probed. + + A probe failure (timeout / 5xx) is treated as "blocking" so the + operator sees the dependency surface; the runner would log the + same failure under ``forecast_regressor_failures_total`` so the + decision is self-consistent. + + Probes are fanned out concurrently — total wall time is bounded by + the slowest probe rather than the sum. The "blocking" result is + deterministic across runs: ties resolve by config order (the first + required regressor that came back empty/failed), independent of + which network response landed first. + """ + required = [ + reg for reg in query.regressors if reg.required and reg.type == "custom" and reg.promql + ] + if not required: + return None + end = datetime.now(tz=UTC) + start = end - lookback + + async def _probe_one(reg: RegressorConfig) -> bool: + """``True`` when the regressor is blocking (empty/failed).""" + try: + frames = await asyncio.wait_for( + source.query_range(reg.promql or "", start, end, step_seconds), + timeout=query_timeout_seconds, + ) + except Exception: + logger.warning( + "warmup_required_regressor_probe_failed", + query=query.id, + regressor=reg.id, + ) + return True + return not frames or _points_in_longest_series(frames) == 0 + + blocked_flags = await asyncio.gather(*(_probe_one(reg) for reg in required)) + for reg, blocked in zip(required, blocked_flags, strict=True): + if blocked: + return reg.id + return None + + +def _error_status(query_id: str, min_points: int) -> QueryWarmupStatus: + """Stable error row for a probe that timed out or raised.""" + return QueryWarmupStatus( + id=query_id, + status="error", + points_available=0, + points_needed=min_points, + blocked_by="lookback", + eta_seconds=math.inf, + ) + + +def _points_in_longest_series(frames: list[SeriesFrame]) -> int: + """Pick the series with the most samples and return its length. + + A multi-series response is reported by the longest series rather + than the mean — the runner's per-series fit treats each frame + independently, so the "best off" series is what unblocks the + query first. Using the mean would silently smear over a single + slow-to-warm peer. + """ + if not frames: + return 0 + return max(len(frame.values) for frame in frames) + + +def _estimate_eta( + *, + points_available: int, + points_needed: int, + step_seconds: int, +) -> float: + """Rough seconds-to-ready estimate from the current point count. + + Uses the simplest possible model — "each ``step`` adds one + sample". That underestimates if the underlying series is sparse + (TSDB downsampling, scrape misses) and overestimates the typical + case slightly because backfill jobs may insert points faster + than the scrape cadence. The estimate is meant to answer + "minutes, hours, or days?" — not to be a hard deadline. + """ + if points_available >= points_needed: + return 0.0 + missing = points_needed - points_available + return float(missing * step_seconds) + + +def render_table(report: WarmupReport) -> str: + """Render the report as a plain-text aligned table for the CLI. + + Kept simple — three columns per query row, columns padded by + rjust/ljust so the typical 80-column terminal is comfortable. + """ + if not report.groups: + return "no groups configured\n" + rows: list[tuple[str, str, str, str, str, str, str]] = [ + ("group", "query", "status", "points", "needed", "eta", "blocked_by") + ] + for group in report.groups: + for query in group.queries: + eta_str = _format_eta(query.eta_seconds) + rows.append( + ( + group.name, + query.id, + query.status, + str(query.points_available), + str(query.points_needed), + eta_str, + query.blocked_by, + ) + ) + widths = [max(len(row[i]) for row in rows) for i in range(7)] + lines: list[str] = [] + for i, row in enumerate(rows): + # left-align text, right-align numbers + formatted = " ".join( + cell.ljust(widths[j]) if j in {0, 1, 2, 6} else cell.rjust(widths[j]) + for j, cell in enumerate(row) + ) + lines.append(formatted) + if i == 0: + lines.append(" ".join("-" * w for w in widths)) + return "\n".join(lines) + "\n" + + +_SECONDS_PER_MINUTE = 60 +_SECONDS_PER_HOUR = 3_600 +_SECONDS_PER_DAY = 86_400 + + +def _format_eta(eta_seconds: float) -> str: + """Format an ETA value as a compact Prometheus-style duration.""" + if not math.isfinite(eta_seconds): + return "∞" + if eta_seconds <= 0: + return "-" + seconds = round(eta_seconds) + if seconds >= _SECONDS_PER_DAY: + return f"{seconds // _SECONDS_PER_DAY}d" + if seconds >= _SECONDS_PER_HOUR: + return f"{seconds // _SECONDS_PER_HOUR}h" + if seconds >= _SECONDS_PER_MINUTE: + return f"{seconds // _SECONDS_PER_MINUTE}m" + return f"{seconds}s" diff --git a/forecaster/tests/test_backfill.py b/forecaster/tests/test_backfill.py new file mode 100644 index 0000000..e052b2d --- /dev/null +++ b/forecaster/tests/test_backfill.py @@ -0,0 +1,487 @@ +"""Tests for the backfill CLI's pure helpers. + +The full backfill walk requires the model registry + statsforecast +and is exercised end-to-end via docker-compose. This module covers +the parsing helpers, the window-cap guard, the per-group timestamp +walk, and the curve-rebasing math, plus a synthetic happy-path test +that wires ``_run_query_at`` against a fake source and a fake model +without touching the real statsforecast import surface. +""" + +from __future__ import annotations + +import io +from collections.abc import Sequence +from datetime import UTC, datetime, timedelta +from pathlib import Path +from typing import Any + +import pandas as pd +import pytest + +from promforecast import backfill as backfill_module +from promforecast import models as model_registry +from promforecast.backfill import ( + BackfillError, + _check_window, + _iter_backfill_timestamps, + _parse_duration_seconds, + _parse_to_timestamp, + _rebase_curve_to_as_of, + _run_query_at, + _select_groups, + run_backfill, +) +from promforecast.config import ( + AccuracyConfig, + Config, + DatasourceConfig, + DefaultsConfig, + EmissionConfig, + GroupConfig, + QueryConfig, + SafetyConfig, + ServerConfig, +) +from promforecast.sink import ForecastCurvePoint +from promforecast.source import PromSource, SeriesFrame + + +def _config( + *, + refresh_interval: timedelta = timedelta(hours=1), + backfill_max_window: timedelta = timedelta(days=90), +) -> Config: + return Config( + datasource=DatasourceConfig(url="http://x"), + server=ServerConfig(refresh_interval=refresh_interval), + safety=SafetyConfig(backfill_max_window=backfill_max_window), + defaults=DefaultsConfig(), + groups=[ + GroupConfig(name="g1", queries=[QueryConfig(id="m", promql="up")]), + GroupConfig(name="g2", queries=[QueryConfig(id="m", promql="up")]), + ], + ) + + +def test_parse_duration_seconds_accepts_compact_form() -> None: + assert _parse_duration_seconds("5m", flag="--from") == 300.0 + assert _parse_duration_seconds("1h", flag="--from") == 3600.0 + assert _parse_duration_seconds("30d", flag="--from") == 30 * 86_400 + + +def test_parse_duration_seconds_rejects_garbage() -> None: + with pytest.raises(BackfillError): + _parse_duration_seconds("nope", flag="--from") + + +def test_parse_to_timestamp_now() -> None: + before = datetime.now(tz=UTC) + dt = _parse_to_timestamp(None) + after = datetime.now(tz=UTC) + assert before <= dt <= after + + +def test_parse_to_timestamp_iso8601() -> None: + dt = _parse_to_timestamp("2026-05-24T12:00:00Z") + assert dt == datetime(2026, 5, 24, 12, 0, 0, tzinfo=UTC) + + +def test_parse_to_timestamp_rejects_garbage() -> None: + with pytest.raises(BackfillError): + _parse_to_timestamp("yesterday") + + +def test_check_window_passes_under_cap() -> None: + cfg = _config(backfill_max_window=timedelta(days=30)) + _check_window(cfg, from_seconds=14 * 86_400) + + +def test_check_window_rejects_over_cap() -> None: + cfg = _config(backfill_max_window=timedelta(days=30)) + with pytest.raises(BackfillError) as exc: + _check_window(cfg, from_seconds=60 * 86_400) + assert "backfill_max_window" in str(exc.value) + + +def test_iter_backfill_timestamps_walks_at_refresh_cadence() -> None: + cfg = _config(refresh_interval=timedelta(hours=1)) + group = cfg.groups[0] + to_time = datetime(2026, 5, 24, 12, 0, 0, tzinfo=UTC) + timestamps = list(_iter_backfill_timestamps(cfg, group, 3 * 3600.0, to_time)) + # 3h window at 1h cadence: 4 timestamps (3h ago, 2h ago, 1h ago, now). + assert len(timestamps) == 4 + assert timestamps[0] == to_time - timedelta(hours=3) + assert timestamps[-1] == to_time + + +def test_iter_backfill_timestamps_floors_cadence_at_60s() -> None: + """A misconfigured sub-minute refresh interval doesn't explode timestamp count.""" + cfg = _config(refresh_interval=timedelta(seconds=10)) + group = cfg.groups[0] + to_time = datetime(2026, 5, 24, 12, 0, 0, tzinfo=UTC) + timestamps = list(_iter_backfill_timestamps(cfg, group, 600.0, to_time)) + # 10 minutes / 60s cadence = 11 timestamps. + assert len(timestamps) == 11 + + +def test_select_groups_returns_all_when_no_selector() -> None: + cfg = _config() + assert [g.name for g in _select_groups(cfg, None)] == ["g1", "g2"] + + +def test_select_groups_filters_to_selector() -> None: + cfg = _config() + assert [g.name for g in _select_groups(cfg, ["g2"])] == ["g2"] + assert _select_groups(cfg, ["missing"]) == [] + + +def test_rebase_curve_shifts_to_target_anchor() -> None: + """Curve points keep their relative spacing but anchor at as_of.""" + as_of = datetime(2026, 1, 1, tzinfo=UTC) + raw_ts_ms = int(datetime(2026, 5, 24, tzinfo=UTC).timestamp() * 1000) + step_ms = 60_000 # 60s + points = [ + ForecastCurvePoint( + metric="up_forecast", + labels={"id": "up", "horizon": "next"}, + value=1.0, + timestamp_ms=raw_ts_ms + i * step_ms, + ) + for i in range(3) + ] + rebased = list(_rebase_curve_to_as_of(points, as_of=as_of, step_seconds=60)) + target_first_ms = int(as_of.timestamp() * 1000) + assert rebased[0].timestamp_ms == target_first_ms + assert rebased[1].timestamp_ms == target_first_ms + step_ms + assert rebased[2].timestamp_ms == target_first_ms + 2 * step_ms + + +def test_rebase_curve_handles_empty_list() -> None: + assert list(_rebase_curve_to_as_of([], as_of=datetime.now(tz=UTC), step_seconds=60)) == [] + + +def test_run_backfill_returns_2_on_missing_config() -> None: + rc = run_backfill( + config_path=Path("/nonexistent/foo.yaml"), + from_duration="1h", + to_timestamp=None, + group_selector=None, + sink_url_override=None, + out=io.StringIO(), + ) + assert rc == 2 + + +def test_run_backfill_returns_1_when_no_sink_configured(tmp_path: Path) -> None: + cfg_body = """ +apiVersion: promforecast.io/v1 +datasource: + url: http://127.0.0.1:1 +groups: + - name: g + queries: + - id: existing + promql: up +""" + path = tmp_path / "cfg.yaml" + path.write_text(cfg_body) + rc = run_backfill( + config_path=path, + from_duration="1h", + to_timestamp=None, + group_selector=None, + sink_url_override=None, + out=io.StringIO(), + ) + assert rc == 1 + + +class _FakeSource(PromSource): + """Synthetic PromSource: returns canned frames, never opens sockets.""" + + def __init__(self, frames: Sequence[SeriesFrame]) -> None: + self._frames = list(frames) + self.calls: list[str] = [] + + async def query_range( + self, + promql: str, + start: datetime, + end: datetime, + step_seconds: int, + ) -> list[SeriesFrame]: + self.calls.append(promql) + return list(self._frames) + + async def aclose(self) -> None: + return None + + +class _FakeModel: + """Drop-in for a real statsforecast model used by the backfill path.""" + + def __init__(self, name: str, value: float = 1.5) -> None: + self.name = name + self._value = value + + def fit_predict( + self, + df: pd.DataFrame, + horizon_steps: int, + freq: str, + levels: list[int], + regressors: pd.DataFrame | None, + ) -> pd.DataFrame: + start_ts = df["ds"].iloc[-1] + pd.Timedelta(freq) + ts = pd.date_range(start=start_ts, periods=horizon_steps, freq=freq) + out: dict[str, Any] = {"ds": ts, "yhat": [self._value] * horizon_steps} + for level in levels: + out[f"yhat_lower_{level}"] = [self._value - 1.0] * horizon_steps + out[f"yhat_upper_{level}"] = [self._value + 1.0] * horizon_steps + return pd.DataFrame(out) + + +@pytest.mark.asyncio +async def test_run_query_at_happy_path_emits_anchored_curve( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """End-to-end ``_run_query_at`` covers fit + safeguard + rebase. + + Wires a fake model registry entry so the real statsforecast import + stays out of the path; covers the dominant code path (preprocess + accepts → frame fits → curve points emit anchored at ``as_of``). + Without this the per-fit pipeline — which is the only functionally + novel code in the module — has no unit coverage. + """ + monkeypatch.setattr( + model_registry, + "build", + lambda name, season_length, params=None: _FakeModel(name, value=2.0), + ) + as_of = datetime(2026, 1, 15, 12, 0, 0, tzinfo=UTC) + step = timedelta(minutes=1) + timestamps = [as_of - step * (n + 1) for n in reversed(range(60))] + frame = SeriesFrame( + labels={"instance": "host-a", "__name__": "up"}, + timestamps=timestamps, + values=[float(i) for i in range(60)], + ) + cfg = Config( + datasource=DatasourceConfig(url="http://x"), + server=ServerConfig(), + safety=SafetyConfig(), + defaults=DefaultsConfig( + min_points=10, + models=["AutoARIMA"], + confidence_levels=[80], + step=step, + lookback=timedelta(hours=1), + horizon=timedelta(minutes=10), + accuracy=AccuracyConfig( + evaluate=False, auto_select=False, horizons=[timedelta(minutes=5)] + ), + emission=EmissionConfig(deviation=False, quality=False, contribution=False), + ), + groups=[GroupConfig(name="g", queries=[QueryConfig(id="m", promql="up")])], + ) + source = _FakeSource([frame]) + produced = await _run_query_at( + config=cfg, + group=cfg.groups[0], + query=cfg.groups[0].queries[0], + source=source, + as_of=as_of, + ) + assert produced.fit_count == ["AutoARIMA"], "exactly one fit per configured model" + assert produced.curve, "fit must produce a non-empty curve" + + # Pure forecast curve points are anchored at as_of (in ms). + target_first_ms = int(as_of.timestamp() * 1000) + forecast_points = [p for p in produced.curve if p.metric.endswith("_forecast")] + assert forecast_points, "expected at least one ``..._forecast`` curve point" + assert min(p.timestamp_ms for p in forecast_points) == target_first_ms + + +class _RecordingSink: + """Records every batch passed to ``write`` so the audit metric is observable.""" + + def __init__(self) -> None: + self.batches: list[list[ForecastCurvePoint]] = [] + + async def write( + self, + points: list[ForecastCurvePoint], + on_retry: object | None = None, + ) -> None: + del on_retry + self.batches.append(list(points)) + + async def aclose(self) -> None: + return None + + +@pytest.mark.asyncio +async def test_walk_backfill_emits_audit_row_with_gauge_name( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """The audit row uses ``_last`` (gauge) not ``_total`` (counter). + + Backfill runs are independent one-shot operations with no + monotonic-counter semantics; a second run with fewer fits than + the first would false-fire ``rate()`` / ``increase()`` if the + metric carried the ``_total`` suffix. This test pins the gauge + naming so a refactor can't silently regress it. + """ + monkeypatch.setattr( + model_registry, + "build", + lambda name, season_length, params=None: _FakeModel(name, value=1.0), + ) + as_of = datetime(2026, 1, 15, 12, 0, 0, tzinfo=UTC) + step = timedelta(minutes=1) + timestamps = [as_of - step * (n + 1) for n in reversed(range(60))] + frame = SeriesFrame( + labels={"instance": "host-a"}, + timestamps=timestamps, + values=[float(i) for i in range(60)], + ) + cfg = Config( + datasource=DatasourceConfig(url="http://x"), + server=ServerConfig(refresh_interval=timedelta(hours=1)), + safety=SafetyConfig(), + defaults=DefaultsConfig( + min_points=10, + models=["AutoARIMA"], + confidence_levels=[80], + step=step, + lookback=timedelta(hours=1), + horizon=timedelta(minutes=10), + accuracy=AccuracyConfig( + evaluate=False, auto_select=False, horizons=[timedelta(minutes=5)] + ), + emission=EmissionConfig(deviation=False, quality=False, contribution=False), + ), + groups=[GroupConfig(name="g", queries=[QueryConfig(id="m", promql="up")])], + ) + source = _FakeSource([frame]) + sink = _RecordingSink() + await backfill_module._walk_backfill( # type: ignore[attr-defined] + config=cfg, + source=source, + sink=sink, # type: ignore[arg-type] + from_seconds=3 * 3600.0, + to_time=as_of, + groups=cfg.groups, + out=io.StringIO(), + ) + # Find the audit batch — exactly one ForecastCurvePoint with + # metric name "forecast_backfill_fits_last". + audit_points = [ + p for batch in sink.batches for p in batch if p.metric == "forecast_backfill_fits_last" + ] + assert len(audit_points) == 1 + audit = audit_points[0] + assert audit.labels == {"group": "g"} + assert audit.value > 0 + # And the old (incorrect) name must NOT appear anywhere. + assert not any( + p.metric == "forecast_backfill_fits_total" for batch in sink.batches for p in batch + ), "old counter-style name should be gone" + + +@pytest.mark.asyncio +async def test_run_query_at_isolates_per_series_failures( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A bad series doesn't abort the rest of the query's walk. + + Mirrors the live runner's per-series failure isolation contract. + Force the preprocess helper to raise on one frame; the other + must still produce a non-empty curve. + """ + monkeypatch.setattr( + model_registry, + "build", + lambda name, season_length, params=None: _FakeModel(name, value=1.0), + ) + + real_preprocess = backfill_module.preprocess_mod.preprocess_series + call_count = {"n": 0} + + def explosive_preprocess(*args: Any, **kwargs: Any) -> Any: + call_count["n"] += 1 + if call_count["n"] == 1: + raise RuntimeError("synthetic preprocess failure") + return real_preprocess(*args, **kwargs) + + monkeypatch.setattr(backfill_module.preprocess_mod, "preprocess_series", explosive_preprocess) + + as_of = datetime(2026, 1, 15, 12, 0, 0, tzinfo=UTC) + step = timedelta(minutes=1) + timestamps = [as_of - step * (n + 1) for n in reversed(range(60))] + bad_frame = SeriesFrame(labels={"instance": "host-a"}, timestamps=timestamps, values=[1.0] * 60) + good_frame = SeriesFrame( + labels={"instance": "host-b"}, + timestamps=timestamps, + values=[float(i) for i in range(60)], + ) + cfg = Config( + datasource=DatasourceConfig(url="http://x"), + server=ServerConfig(), + safety=SafetyConfig(), + defaults=DefaultsConfig( + min_points=10, + models=["AutoARIMA"], + confidence_levels=[80], + step=step, + lookback=timedelta(hours=1), + horizon=timedelta(minutes=10), + accuracy=AccuracyConfig( + evaluate=False, auto_select=False, horizons=[timedelta(minutes=5)] + ), + emission=EmissionConfig(deviation=False, quality=False, contribution=False), + ), + groups=[GroupConfig(name="g", queries=[QueryConfig(id="m", promql="up")])], + ) + source = _FakeSource([bad_frame, good_frame]) + produced = await _run_query_at( + config=cfg, + group=cfg.groups[0], + query=cfg.groups[0].queries[0], + source=source, + as_of=as_of, + ) + # The good frame still fit; the bad one didn't take the query down. + assert produced.fit_count == ["AutoARIMA"] + + +def test_run_backfill_returns_1_when_window_over_cap(tmp_path: Path) -> None: + """An over-cap window fails before any TSDB I/O happens.""" + cfg_body = """ +apiVersion: promforecast.io/v1 +datasource: + url: http://127.0.0.1:1 +safety: + backfill_max_window: 7d +sink: + remote_write: + enabled: true + url: http://127.0.0.1:1/api/v1/import/prometheus +groups: + - name: g + queries: + - id: existing + promql: up +""" + path = tmp_path / "cfg.yaml" + path.write_text(cfg_body) + rc = run_backfill( + config_path=path, + from_duration="30d", + to_timestamp=None, + group_selector=None, + sink_url_override=None, + out=io.StringIO(), + ) + assert rc == 1 diff --git a/forecaster/tests/test_capacity.py b/forecaster/tests/test_capacity.py index de8ec96..054268a 100644 --- a/forecaster/tests/test_capacity.py +++ b/forecaster/tests/test_capacity.py @@ -8,11 +8,12 @@ import pandas as pd from promforecast.capacity import ( + build_threshold_action_costs, compute_growth_rates, compute_threshold_etas, compute_will_breach, ) -from promforecast.config import ThresholdConfig +from promforecast.config import ActionCostConfig, ThresholdConfig def _forecast_frame(values: list[float], *, step_seconds: int = 60) -> pd.DataFrame: @@ -348,3 +349,84 @@ def test_will_breach_emits_per_level_row() -> None: band = next(b for b in breaches if b.level == 80) assert point.value == 0.0 assert band.value == 1.0 + + +def test_action_cost_skips_thresholds_without_annotation() -> None: + """Only thresholds with ``action_cost`` produce annotations. + + Most thresholds carry no cost. ``build_threshold_action_costs`` + must filter them out so the gauge family only ever materialises for + the opted-in subset. + """ + out = build_threshold_action_costs( + [ + ThresholdConfig(name="warn", value=80.0, comparator="gt"), + ThresholdConfig(name="critical", value=95.0, comparator="gt"), + ] + ) + assert out == [] + + +def test_action_cost_projects_value_and_type() -> None: + """The projection carries the configured value and type label verbatim.""" + out = build_threshold_action_costs( + [ + ThresholdConfig( + name="warn", + value=80.0, + comparator="gt", + action_cost=ActionCostConfig(type="usd", value=50.0), + ), + ThresholdConfig( + name="critical", + value=95.0, + comparator="gt", + action_cost=ActionCostConfig(type="usd", value=50_000.0), + ), + ] + ) + assert [(a.name, a.cost_type, a.cost_value) for a in out] == [ + ("warn", "usd", 50.0), + ("critical", "usd", 50_000.0), + ] + + +def test_action_cost_preserves_config_order() -> None: + """Stable ordering keeps the published gauge sequence deterministic. + + A reshuffle each run would churn the per-(group, query, name) label + set Grafana caches against the metric — operators expect rows to + stay in their configured order so successive renders look identical + on dashboards. + """ + out = build_threshold_action_costs( + [ + ThresholdConfig( + name="z_last", + value=1.0, + action_cost=ActionCostConfig(type="usd", value=3.0), + ), + ThresholdConfig( + name="a_first", + value=1.0, + action_cost=ActionCostConfig(type="usd", value=1.0), + ), + ] + ) + assert [a.name for a in out] == ["z_last", "a_first"] + + +def test_action_cost_non_currency_type_is_accepted() -> None: + """``type`` is free text so non-currency cost framings work too.""" + out = build_threshold_action_costs( + [ + ThresholdConfig( + name="warn", + value=80.0, + action_cost=ActionCostConfig(type="engineer_hours", value=8.0), + ), + ] + ) + assert out == [ + type(out[0])(name="warn", cost_type="engineer_hours", cost_value=8.0), + ] diff --git a/forecaster/tests/test_config.py b/forecaster/tests/test_config.py index 1e36733..730c852 100644 --- a/forecaster/tests/test_config.py +++ b/forecaster/tests/test_config.py @@ -776,6 +776,248 @@ def test_load_rejects_zero_alert_within(tmp_path: Path) -> None: config_module.load(path) +def test_load_accepts_threshold_with_action_cost(tmp_path: Path) -> None: + """A threshold with ``action_cost`` round-trips through the loader.""" + path = _write( + tmp_path, + """ +apiVersion: promforecast.io/v1 +datasource: + url: http://vm:8428/ +groups: + - name: g + queries: + - id: disk_used_pct + promql: up + thresholds: + - name: critical + value: 95 + action_cost: + type: usd + value: 50000 +""", + ) + cfg = config_module.load(path) + threshold = cfg.groups[0].queries[0].thresholds[0] + assert threshold.action_cost is not None + assert threshold.action_cost.type == "usd" + assert threshold.action_cost.value == 50_000 + + +def test_load_rejects_action_cost_with_empty_type(tmp_path: Path) -> None: + """An empty / whitespace ``type`` label is rejected at load time. + + Empty labels render as ``type=""`` on /metrics and Grafana templating + typically drops them — silently breaking the rank-by-cost panel. + Better to fail fast. + """ + path = _write( + tmp_path, + """ +apiVersion: promforecast.io/v1 +datasource: + url: http://vm:8428/ +groups: + - name: g + queries: + - id: m + promql: up + thresholds: + - name: warn + value: 80 + action_cost: + type: " " + value: 1.0 +""", + ) + with pytest.raises(ValueError, match=r"action_cost\.type must be non-empty"): + config_module.load(path) + + +def test_load_rejects_action_cost_with_non_positive_value(tmp_path: Path) -> None: + """``value <= 0`` is rejected so a misconfig can't silently rank below every meaningful cost.""" + path = _write( + tmp_path, + """ +apiVersion: promforecast.io/v1 +datasource: + url: http://vm:8428/ +groups: + - name: g + queries: + - id: m + promql: up + thresholds: + - name: warn + value: 80 + action_cost: + type: usd + value: 0 +""", + ) + with pytest.raises(ValueError, match="greater than 0"): + config_module.load(path) + + +def test_load_accepts_governance_health_score(tmp_path: Path) -> None: + """A ``governance.health_score`` block round-trips through the loader.""" + path = _write( + tmp_path, + """ +apiVersion: promforecast.io/v1 +datasource: + url: http://vm:8428/ +governance: + health_score: + aggregate_by: [service, environment] + aggregator: weighted_mean + weight_by: forecast_series_count + quality_floor: 0.6 +groups: + - name: g + queries: + - id: m + promql: up +""", + ) + cfg = config_module.load(path) + hs = cfg.governance.health_score + assert hs is not None + assert hs.aggregate_by == ["service", "environment"] + assert hs.aggregator == "weighted_mean" + assert hs.weight_by == "forecast_series_count" + assert hs.quality_floor == 0.6 + + +def test_load_rejects_governance_health_score_with_empty_aggregate_by( + tmp_path: Path, +) -> None: + """An empty ``aggregate_by`` is rejected — the bucket key would be ``()``.""" + path = _write( + tmp_path, + """ +apiVersion: promforecast.io/v1 +datasource: + url: http://vm:8428/ +governance: + health_score: + aggregate_by: [] +groups: + - name: g + queries: + - id: m + promql: up +""", + ) + with pytest.raises(ValueError, match="aggregate_by"): + config_module.load(path) + + +# --------------------------------------------------------------------------- +# Right-sizing recommendation validator paths. +# --------------------------------------------------------------------------- + + +_BASE_RESOURCE_SIZING_HEADER = """ +apiVersion: promforecast.io/v1 +datasource: + url: http://vm:8428/ +groups: + - name: g + queries: + - id: svc + promql: up +recommendations: + resource_sizing: + enabled: true + workloads: +""" + + +def test_load_accepts_valid_resource_sizing_block(tmp_path: Path) -> None: + """A well-formed right-sizing block round-trips through the loader.""" + path = _write( + tmp_path, + _BASE_RESOURCE_SIZING_HEADER + + """ - workload: api + kind: cpu + forecast_query: svc + current_promql: kube_pod_container_resource_requests +""", + ) + cfg = config_module.load(path) + rs = cfg.recommendations.resource_sizing + assert rs.enabled is True + assert len(rs.workloads) == 1 + assert rs.workloads[0].workload == "api" + assert rs.workloads[0].kind == "cpu" + + +def test_load_rejects_resource_sizing_unknown_forecast_query(tmp_path: Path) -> None: + """``forecast_query`` must reference a configured query id.""" + path = _write( + tmp_path, + _BASE_RESOURCE_SIZING_HEADER + + """ - workload: api + kind: cpu + forecast_query: nonexistent +""", + ) + with pytest.raises(ValueError, match="does not match any configured query id"): + config_module.load(path) + + +def test_load_rejects_resource_sizing_duplicate_workload_kind(tmp_path: Path) -> None: + """Two entries with the same (workload, kind) would race on /metrics — reject.""" + path = _write( + tmp_path, + _BASE_RESOURCE_SIZING_HEADER + + """ - workload: api + kind: cpu + forecast_query: svc + - workload: api + kind: cpu + forecast_query: svc +""", + ) + with pytest.raises(ValueError, match="duplicate"): + config_module.load(path) + + +def test_load_rejects_resource_sizing_empty_current_promql(tmp_path: Path) -> None: + """Empty / whitespace ``current_promql`` is the half-configured shape; reject. + + Treating an empty string as "configured" would issue a doomed + PromQL fetch every refresh and silently never produce the + current / savings_pct siblings the operator was after. + """ + path = _write( + tmp_path, + _BASE_RESOURCE_SIZING_HEADER + + """ - workload: api + kind: cpu + forecast_query: svc + current_promql: " " +""", + ) + with pytest.raises(ValueError, match="current_promql must be a non-empty PromQL"): + config_module.load(path) + + +def test_load_rejects_resource_sizing_empty_workload_name(tmp_path: Path) -> None: + """An empty ``workload`` label would render as ``workload=""`` on /metrics.""" + path = _write( + tmp_path, + _BASE_RESOURCE_SIZING_HEADER + + """ - workload: "" + kind: cpu + forecast_query: svc +""", + ) + with pytest.raises(ValueError, match="workload must be non-empty"): + config_module.load(path) + + def test_load_rejects_growth_rate_enabled_without_windows(tmp_path: Path) -> None: """``growth_rate.enabled: true`` with empty windows must fail at load time.""" path = _write( diff --git a/forecaster/tests/test_cookbook_configs.py b/forecaster/tests/test_cookbook_configs.py new file mode 100644 index 0000000..69a7aa5 --- /dev/null +++ b/forecaster/tests/test_cookbook_configs.py @@ -0,0 +1,68 @@ +"""CI guard: every cookbook YAML loads cleanly and yields a finite cardinality. + +The docs/cookbook recipes ship copy-pasteable YAML in +``examples/configs/cookbook/``. If one of those snippets drifts from +the schema (new validator added, field renamed, type changed) operators +hit a confusing failure right after copy-pasting the recipe — exactly +the friction the cookbook is meant to remove. This test loads every +snippet through ``promforecast.config.load`` *and* the cardinality +estimator so a schema or estimator regression catches the cookbook on +the same PR as the rest of the test suite. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from promforecast import config as config_module +from promforecast.validate import _estimate_cardinality + +REPO_ROOT = Path(__file__).resolve().parents[2] +COOKBOOK_DIR = REPO_ROOT / "examples" / "configs" / "cookbook" + +# Enumerated so a recipe that someone forgot to ship under +# ``examples/configs/cookbook/`` is a hard failure rather than a +# silently-skipped test. +EXPECTED_RECIPES = frozenset( + { + "node-filesystem-avail-bytes.yaml", + "http-requests-total.yaml", + "kube-deployment-status-replicas.yaml", + "jvm-gc-pause-seconds.yaml", + "redis-connected-clients.yaml", + "aws-cloudwatch-estimated-charges-usd.yaml", + "kafka-consumer-lag.yaml", + } +) + + +def test_cookbook_directory_has_every_expected_recipe() -> None: + """Every recipe listed in the cookbook README ships a YAML snippet.""" + present = {path.name for path in COOKBOOK_DIR.glob("*.yaml")} + missing = EXPECTED_RECIPES - present + assert not missing, ( + f"cookbook YAMLs missing for recipes: {sorted(missing)}. " + f"Add the file under {COOKBOOK_DIR}/." + ) + + +@pytest.mark.parametrize("filename", sorted(EXPECTED_RECIPES)) +def test_cookbook_config_loads_and_estimates(filename: str) -> None: + """Each cookbook YAML loads through the schema validator without error. + + Then the cardinality estimator must produce a finite, non-zero + total. A zero total means no forecast / accuracy / quality rows + will be emitted — which is almost certainly a recipe bug + (everything disabled). + """ + path = COOKBOOK_DIR / filename + cfg = config_module.load(path) + estimate = _estimate_cardinality(cfg) + assert estimate.total > 0, ( + f"cookbook recipe {filename} produced zero estimated rows — " + "check the model / horizon / emission config." + ) + # Every recipe carries at least one query. + assert sum(len(g.queries) for g in cfg.groups) >= 1 diff --git a/forecaster/tests/test_drift_explainer.py b/forecaster/tests/test_drift_explainer.py new file mode 100644 index 0000000..8402a20 --- /dev/null +++ b/forecaster/tests/test_drift_explainer.py @@ -0,0 +1,351 @@ +"""Tests for the drift change-correlation explainer + runner wiring.""" + +from __future__ import annotations + +import math +from datetime import UTC, datetime +from typing import Any + +import pandas as pd +import pytest + +from promforecast.config import ( + AccuracyConfig, + Config, + DatasourceConfig, + DefaultsConfig, + DriftEmissionConfig, + EmissionConfig, + GroupConfig, + QueryConfig, + QueryEmissionConfig, + SafetyConfig, + ServerConfig, +) +from promforecast.drift_explainer import ( + DriftWindow, + classify_confidence, + compute_pearson, + explain, +) +from promforecast.exporter import Exporter +from promforecast.runner import Runner +from promforecast.source import PromSource, SeriesFrame + +# ---- Pure-function tests ---------------------------------------------------- + + +def test_pearson_perfect_positive_correlation() -> None: + """Two identical signals correlate at +1.0.""" + coef, n = compute_pearson([1.0, 2.0, 3.0, 4.0, 5.0], [1.0, 2.0, 3.0, 4.0, 5.0]) + assert math.isclose(coef, 1.0) + assert n == 5 + + +def test_pearson_perfect_negative_correlation() -> None: + """Mirrored signals correlate at -1.0.""" + coef, n = compute_pearson([1.0, 2.0, 3.0, 4.0, 5.0], [5.0, 4.0, 3.0, 2.0, 1.0]) + assert math.isclose(coef, -1.0) + assert n == 5 + + +def test_pearson_drops_non_finite_pairs() -> None: + """A single NaN doesn't poison the coefficient. + + Operators see NaN regressor readings when a fetch fails — we + can't let that take down the correlation for the rest of the + window. + """ + coef, n = compute_pearson( + [1.0, 2.0, float("nan"), 4.0, 5.0], + [1.0, 2.0, 3.0, 4.0, 5.0], + ) + assert n == 4 + assert math.isclose(coef, 1.0) + + +def test_pearson_insufficient_samples_returns_nan() -> None: + """Below the minimum pair count, the coefficient is NaN. + + The explainer treats this as ``insufficient_data`` and falls + back to the value-based suspect. + """ + coef, n = compute_pearson([1.0, 2.0], [1.0, 2.0]) + assert math.isnan(coef) + assert n == 2 + + +def test_pearson_zero_variance_returns_nan() -> None: + """A flat regressor can't explain anything → NaN coefficient. + + Without this guard, a constant regressor would divide by zero + inside the closed-form correlation; we surface NaN so the caller + can fall back to the value suspect cleanly. + """ + coef, n = compute_pearson([1.0, 2.0, 3.0, 4.0, 5.0], [7.0, 7.0, 7.0, 7.0, 7.0]) + assert math.isnan(coef) + assert n == 5 + + +def test_pearson_rejects_mismatched_lengths() -> None: + """Programmer-error guard so a length bug fails loud.""" + with pytest.raises(ValueError): + compute_pearson([1.0, 2.0], [1.0]) + + +def test_classify_confidence_named_bands() -> None: + """Confidence band matches the documented cutoffs.""" + assert classify_confidence(0.9, n_pairs=25) == "high" + # Strong correlation but too few samples → drops to medium. + assert classify_confidence(0.9, n_pairs=10) == "medium" + # Medium correlation with enough samples → medium. + assert classify_confidence(0.5, n_pairs=10) == "medium" + # Weak correlation → low. + assert classify_confidence(0.2, n_pairs=10) == "low" + # Insufficient samples → distinct band, not "low". + assert classify_confidence(0.9, n_pairs=3) == "insufficient_data" + # NaN coefficient → insufficient_data. + assert classify_confidence(float("nan"), n_pairs=100) == "insufficient_data" + + +def test_explain_picks_correlation_suspect_when_confidence_high() -> None: + """High-confidence correlation wins over the loud-but-uncorrelated suspect. + + Captures the story 20 distinction: a regressor that's loud right + now may be one-shot noise. The rolling-window correlation surfaces + the genuine driver instead. + """ + window = DriftWindow(maxlen=30) + # Drift trends up over 20 samples — ``deploys`` regressor also + # trends up (Pearson ≈ 1.0); ``noise`` regressor is random. + for i in range(25): + window.record( + drift_score=i * 0.05, + regressor_values={"deploys": float(i), "noise": float((i * 7) % 3)}, + ) + # Current values: "loud" is highest right now but doesn't track + # the drift history. + explanation = explain( + window, + current_regressor_values={"deploys": 24.0, "noise": 0.0, "loud": 9999.0}, + ) + assert explanation.top_correlation_suspect == "deploys" + assert explanation.top_value_suspect == "loud" + # Primary picks correlation when confidence is high. + assert explanation.primary_suspect == "deploys" + assert explanation.top_regressor_correlation_confidence == "high" + + +def test_explain_falls_back_to_value_suspect_when_insufficient_data() -> None: + """A fresh-install drift event with no history uses the value suspect. + + The first time drift fires there's no rolling history to compute + correlation against; primary_suspect picks the loud one so the + operator gets *some* hypothesis to start with. + """ + window = DriftWindow(maxlen=30) + # No samples in the window — every correlation is undefined. + explanation = explain( + window, + current_regressor_values={"deploys": 5.0, "noise": 1.0}, + ) + assert explanation.top_regressor_correlation_confidence == "insufficient_data" + assert explanation.primary_suspect == "deploys" # the loud one + assert explanation.top_value_suspect == "deploys" + + +def test_explain_emits_unknown_when_no_regressors() -> None: + """A query without regressors emits ``unknown`` rather than crashing.""" + window = DriftWindow(maxlen=30) + explanation = explain(window, current_regressor_values={}) + assert explanation.primary_suspect == "unknown" + assert explanation.top_value_suspect == "unknown" + assert explanation.top_correlation_suspect == "unknown" + + +def test_drift_window_evicts_oldest_when_full() -> None: + """The ring buffer respects ``maxlen``.""" + window = DriftWindow(maxlen=3) + for i in range(5): + window.record(drift_score=float(i), regressor_values={"r": float(i)}) + assert len(window) == 3 + samples = window.samples() + assert [s.drift_score for s in samples] == [2.0, 3.0, 4.0] + + +# ---- Runner-level wiring --------------------------------------------------- + + +class _FakeSource(PromSource): + def __init__(self) -> None: + pass + + async def query_range( + self, + promql: str, + start: datetime, + end: datetime, + step_seconds: int, + ) -> list[SeriesFrame]: + return [] + + async def aclose(self) -> None: + return None + + +def _runner_config(alert_threshold: float = 0.25, correlation_window: int = 30) -> Config: + return Config( + datasource=DatasourceConfig(url="http://x"), + server=ServerConfig(), + safety=SafetyConfig(), + defaults=DefaultsConfig( + min_points=5, + models=["SeasonalNaive"], + accuracy=AccuracyConfig(evaluate=False, auto_select=False), + emission=EmissionConfig(quality=False, deviation=False), + ), + groups=[ + GroupConfig( + name="g", + queries=[ + QueryConfig( + id="m", + promql="up", + emission=QueryEmissionConfig( + drift=DriftEmissionConfig( + enabled=True, + alert_threshold=alert_threshold, + correlation_window=correlation_window, + ), + ), + ) + ], + ) + ], + ) + + +def _curve(model: str, query_id: str, values: list[float]) -> list[Any]: + """Build minimal sink.ForecastCurvePoint shapes for the drift recorder.""" + from promforecast.sink import ForecastCurvePoint # noqa: PLC0415 + + points = [] + for i, v in enumerate(values): + points.append( + ForecastCurvePoint( + metric=f"{query_id}_forecast", + labels={"model": model, "instance": "host-a"}, + value=v, + timestamp_ms=int((datetime(2026, 5, 21, tzinfo=UTC).timestamp() + i * 60) * 1000), + ) + ) + return points + + +@pytest.mark.asyncio +async def test_record_drift_emits_explained_counter_and_log( + capsys: pytest.CaptureFixture[str], +) -> None: + """A drift crossing bumps the counter and emits the structured log event. + + Threads a synthetic curve through ``_record_drift`` twice — the + first call seeds the cache; the second produces a drift score + that crosses the configured threshold. + """ + cfg = _runner_config(alert_threshold=0.1) + runner = Runner(config=cfg, source=_FakeSource(), exporter=(exporter := Exporter())) + from promforecast.point_factory import SeriesPointFactory # noqa: PLC0415 + + factory = SeriesPointFactory( + metric_id="m", + base_labels={"instance": "host-a"}, + group="g", + levels=[], + ) + from promforecast.runner._state import _FitResult # noqa: PLC0415 + + # Seed the cache with the first curve so the next call has a + # baseline to compare against. + seed = _FitResult() + runner._record_drift( + query=cfg.groups[0].queries[0], + group_name="g", + series_key="host-a", + curve_points=_curve("m1", "m", [10.0, 10.0, 10.0]), + factory=factory, + result=seed, + regressors=pd.DataFrame( + { + "unique_id": ["s"] * 3, + "ds": pd.date_range("2026-01-01", periods=3, freq="5min"), + "deploys": [0.0, 0.0, 1.0], + } + ), + ) + # Second call: a curve that diverges sharply — score above the + # 0.1 threshold so the explainer fires. + second = _FitResult() + runner._record_drift( + query=cfg.groups[0].queries[0], + group_name="g", + series_key="host-a", + curve_points=_curve("m1", "m", [100.0, 100.0, 100.0]), + factory=factory, + result=second, + regressors=pd.DataFrame( + { + "unique_id": ["s"] * 3, + "ds": pd.date_range("2026-01-01", periods=3, freq="5min"), + "deploys": [0.0, 0.0, 5.0], + } + ), + ) + assert second.drift_alerts == 1 + body = exporter.render().decode() + assert "forecast_drift_explained_total{" in body + assert 'group="g"' in body + assert 'query="m"' in body + # Primary suspect is "deploys" — the only configured regressor. + assert 'primary_suspect="deploys"' in body + # The structured log line ride structlog's configured renderer + # (set up in main.run / tests indirectly); capsys catches the + # stdout emission deterministically. + captured = capsys.readouterr() + assert "forecast_drift_explained" in (captured.out + captured.err) + + +@pytest.mark.asyncio +async def test_record_drift_with_no_regressors_emits_unknown_suspect() -> None: + """A query without configured regressors still gets the counter row. + + The structured log line carries enough context that ``unknown`` + is itself a useful signal — "drift fired but I have no + hypothesis for you". Without this, regressor-free deployments + would never see the counter at all. + """ + cfg = _runner_config(alert_threshold=0.1) + runner = Runner(config=cfg, source=_FakeSource(), exporter=(exporter := Exporter())) + from promforecast.point_factory import SeriesPointFactory # noqa: PLC0415 + from promforecast.runner._state import _FitResult # noqa: PLC0415 + + factory = SeriesPointFactory(metric_id="m", base_labels={}, group="g", levels=[]) + runner._record_drift( + query=cfg.groups[0].queries[0], + group_name="g", + series_key="host-a", + curve_points=_curve("m1", "m", [10.0, 10.0]), + factory=factory, + result=_FitResult(), + regressors=None, + ) + runner._record_drift( + query=cfg.groups[0].queries[0], + group_name="g", + series_key="host-a", + curve_points=_curve("m1", "m", [50.0, 50.0]), + factory=factory, + result=_FitResult(), + regressors=None, + ) + body = exporter.render().decode() + assert 'primary_suspect="unknown"' in body diff --git a/forecaster/tests/test_example_alerts_labelset.py b/forecaster/tests/test_example_alerts_labelset.py index 5eae7c5..8c4fd78 100644 --- a/forecaster/tests/test_example_alerts_labelset.py +++ b/forecaster/tests/test_example_alerts_labelset.py @@ -150,7 +150,15 @@ def _expand_levels(base: list[dict[str, str]], levels: tuple[str, ...]) -> list[ ), ], "forecast_quality_score": [ - _with(_SNAPSHOT_OP_LABELS, group=g, id=i, model=m, model_fallback=mf, cold_start="false") + _with( + _SNAPSHOT_OP_LABELS, + group=g, + id=i, + model=m, + model_fallback=mf, + cold_start="false", + horizon=h, + ) for g, i, m in [ ("node_capacity", "node_cpu_busy_pct", "AutoARIMA"), ("node_capacity", "node_memory_used_bytes", "SeasonalNaive"), @@ -160,6 +168,13 @@ def _expand_levels(base: list[dict[str, str]], levels: tuple[str, ...]) -> list[ # (filters on ``model_fallback="true"``) and the cold-start / # quality alerts (filter on ``cold_start!="true"``) both resolve. for mf in ("false", "true") + # Aggregated (``horizon=""``) + per-horizon ("1h"/"24h") rows + # match the per-horizon emission shape. Example alerts gate + # explicitly on ``horizon=""`` for back-compat with the + # historical horizon-less behaviour, but the fixture carries + # the per-horizon rows too so any rule that opts into a + # specific horizon resolves cleanly. + for h in ("", "1h", "24h") ], "forecast_deviation_outside_band": [ _with(_SNAPSHOT_OP_LABELS, group=g, id=i, model=m, level=lvl) @@ -317,6 +332,139 @@ def _expand_levels(base: list[dict[str, str]], levels: tuple[str, ...]) -> list[ for d in (0, 1, 2) for h in (0, 12, 23) ], + # Per-(series, model) staleness gauge. Snapshot-only, so it carries + # the forecaster scrape labels alongside the source-side labels the + # forecast emission picked up. + "forecast_data_staleness_seconds": [ + _with( + _SNAPSHOT_OP_LABELS, + id=i, + group=g, + model=m, + instance="node-exporter:9100", + ) + for g, i, m in [ + ("node_capacity", "node_cpu_busy_pct", "AutoARIMA"), + ("node_capacity", "node_memory_used_bytes", "SeasonalNaive"), + ("node_storage", "node_filesystem_avail_bytes", "AutoARIMA"), + ] + ], + # Forecast-stability families. Both carry the forecaster scrape + # labels alongside the source-side labels. + "forecast_revision_magnitude": [ + _with( + _SNAPSHOT_OP_LABELS, + id=i, + group=g, + model=m, + horizon=h, + instance="node-exporter:9100", + ) + for g, i, m in [ + ("node_capacity", "node_cpu_busy_pct", "AutoARIMA"), + ("node_storage", "node_filesystem_avail_bytes", "AutoARIMA"), + ] + for h in ("1h", "24h") + ], + "forecast_model_disagreement": [ + _with( + _SNAPSHOT_OP_LABELS, + id=i, + group=g, + horizon=h, + instance="node-exporter:9100", + ) + for g, i in [ + ("node_capacity", "node_cpu_busy_pct"), + ("node_storage", "node_filesystem_avail_bytes"), + ] + for h in ("1h", "24h") + ], + # Self-observability families. + "forecast_model_fit_success_ratio": [ + _with(_SNAPSHOT_OP_LABELS, group=g, model=m, window=w) + for g, m in [ + ("node_capacity", "AutoARIMA"), + ("node_capacity", "SeasonalNaive"), + ("node_storage", "AutoARIMA"), + ] + for w in ("1h", "24h") + ], + "forecast_regressor_stability_score": [ + _with(_SNAPSHOT_OP_LABELS, group=g, query=q, regressor_id=r) + for g, q, r in [ + ("node_capacity", "node_cpu_busy_pct", "deploys"), + ("node_storage", "node_filesystem_avail_bytes", "deploys"), + ] + ], + # Degraded-mode families. Source retries and sink-failure aggregate + # ship on /metrics only (snapshot path), so they carry the forecaster + # scrape labels. The snapshot-age gauge mirrors + # ``forecast_last_run_timestamp_seconds`` and is one row per group. + "forecast_source_retries_total": [ + _with(_SNAPSHOT_OP_LABELS, reason=r) for r in ("transport", "http_5xx") + ], + "forecast_sink_write_failures_total": [ + _with(_SNAPSHOT_OP_LABELS, reason=r) for r in ("timeout", "http_error", "unexpected") + ], + "forecast_snapshot_age_seconds": [ + _with(_SNAPSHOT_OP_LABELS, group=g) for g in ("node_capacity", "node_storage") + ], + "forecast_failures_total": [ + _with(_SNAPSHOT_OP_LABELS, group=g, reason=r) + for g in ("node_capacity", "node_storage") + for r in ("query_error", "query_timeout", "fit_error", "fit_timeout") + ], + # Governance / drift-explainer families. Snapshot-only emission + # so they carry the scrape-side labels. + "forecast_service_health_score": [ + _with( + _SNAPSHOT_OP_LABELS, + service=s, + environment=e, + aggregator="weighted_mean", + weight_by="forecast_series_count", + series_count=str(c), + ) + for s, e, c in [ + ("checkout", "prod", 42), + ("payments", "prod", 18), + ] + ], + "forecast_drift_explained_total": [ + _with(_SNAPSHOT_OP_LABELS, group=g, query=q, primary_suspect=s) + for g, q, s in [ + ("node_capacity", "node_cpu_busy_pct", "deploys"), + ("node_storage", "node_filesystem_avail_bytes", "unknown"), + ] + ], + # Cost-of-action gauge for capacity thresholds. Config-derived; + # ``type`` is free-text per spec, but the fixture pins it to USD + # to keep the binding deterministic. + "forecast_threshold_action_cost": [ + _with(_SNAPSHOT_OP_LABELS, id=i, group=g, name=n, type="usd") + for g, i, n in [ + ("node_storage", "node_filesystem_avail_bytes", "critical"), + ("kafka", "kafka_consumer_lag", "backlog_warn"), + ] + ], + # Right-sizing recommendation family — three sibling gauges, + # all keyed on (workload, kind). ``confidence`` rides only on + # the recommended + savings_pct emissions; the ``current`` + # sibling doesn't carry it because operators may want to + # graph current size without filtering by confidence band. + "forecast_recommended_resource_size": [ + _with(_SNAPSHOT_OP_LABELS, workload=w, kind=k, confidence="high") + for w, k in [("api-server", "cpu"), ("api-server", "memory")] + ], + "forecast_recommended_resource_current": [ + _with(_SNAPSHOT_OP_LABELS, workload=w, kind=k) + for w, k in [("api-server", "cpu"), ("api-server", "memory")] + ], + "forecast_recommended_resource_savings_pct": [ + _with(_SNAPSHOT_OP_LABELS, workload=w, kind=k, confidence="high") + for w, k in [("api-server", "cpu"), ("api-server", "memory")] + ], } # --------------------------------------------------------------------------- diff --git a/forecaster/tests/test_exporter_render.py b/forecaster/tests/test_exporter_render.py index 15f8e3f..1dd8e5a 100644 --- a/forecaster/tests/test_exporter_render.py +++ b/forecaster/tests/test_exporter_render.py @@ -3,6 +3,7 @@ from __future__ import annotations import math +import time as time_mod from promforecast.exporter import ( AccuracyPoint, @@ -256,3 +257,192 @@ def test_render_escapes_label_values() -> None: ) body = exporter.render().decode() assert 'weird="has\\"quotes\\\\and\\nnewline"' in body + + +def test_render_omits_staleness_family_when_no_fits_recorded() -> None: + """A fresh deployment shouldn't carry a zero-row staleness family. + + Mirrors the convention every other opt-in family uses (no rows -> + omit entirely) so dashboards can confirm the gauge starts emitting + on the first refresh rather than from process boot. + """ + body = Exporter().render().decode() + assert "forecast_data_staleness_seconds" not in body + + +def test_render_includes_staleness_after_recording_a_fit() -> None: + """A recorded fit emits ``forecast_data_staleness_seconds`` with elapsed time.""" + exporter = Exporter() + before = time_mod.time() + exporter.record_series_fit_success( + group="node_capacity", + query_id="node_cpu_busy_pct", + model_name="AutoARIMA", + series_key="instance=host-a", + labels={ + "id": "node_cpu_busy_pct", + "group": "node_capacity", + "model": "AutoARIMA", + "instance": "host-a", + }, + ) + body = exporter.render().decode() + after = time_mod.time() + assert "# TYPE forecast_data_staleness_seconds gauge" in body + assert 'id="node_cpu_busy_pct"' in body + assert 'model="AutoARIMA"' in body + assert 'group="node_capacity"' in body + assert 'instance="host-a"' in body + # The value should be a small non-negative elapsed time; the bound + # is generous to absorb test scheduler jitter. + assert "forecast_data_staleness_seconds{" in body + # The recorded sample is bounded by the time we spent between + # recording the fit and rendering — well under a second on any + # remotely modern machine. + assert (after - before) < 5.0 + + +def test_staleness_resets_when_fit_is_re_recorded() -> None: + """A second successful fit overwrites the timestamp, dropping the elapsed time.""" + exporter = Exporter() + exporter.record_series_fit_success( + group="g", + query_id="q", + model_name="m", + series_key="k", + labels={"id": "q", "group": "g", "model": "m"}, + ) + # Simulate "time passes". We don't actually sleep — we manually + # rewrite the recorded timestamp to an older one so the next render + # observes a large elapsed value, then re-recording resets the + # clock. + key = ("g", "q", "m", "k") + timestamp, labels = exporter._series_last_success[key] + exporter._series_last_success[key] = (timestamp - 3600.0, labels) + body = exporter.render().decode() + line = next(ln for ln in body.splitlines() if ln.startswith("forecast_data_staleness_seconds{")) + # Last value on the metric line is the gauge value. + value = float(line.rsplit(" ", 1)[-1]) + assert value >= 3600.0 + + # Re-recording resets the clock. + exporter.record_series_fit_success( + group="g", + query_id="q", + model_name="m", + series_key="k", + labels={"id": "q", "group": "g", "model": "m"}, + ) + body2 = exporter.render().decode() + line2 = next( + ln for ln in body2.splitlines() if ln.startswith("forecast_data_staleness_seconds{") + ) + value2 = float(line2.rsplit(" ", 1)[-1]) + assert value2 < 5.0 + # The reset must happen monotonically — the new sample is strictly + # smaller than the artificially-aged one. + assert value2 < value + assert time_mod.time() > 0 # silence "unused import" if assertion sequence shifts + + +def test_staleness_retain_groups_prunes_removed_groups() -> None: + exporter = Exporter() + exporter.record_series_fit_success( + group="keep", + query_id="q", + model_name="m", + series_key="k", + labels={"id": "q", "group": "keep", "model": "m"}, + ) + exporter.record_series_fit_success( + group="drop", + query_id="q", + model_name="m", + series_key="k", + labels={"id": "q", "group": "drop", "model": "m"}, + ) + exporter.retain_staleness_groups({"keep"}) + body = exporter.render().decode() + assert 'group="keep"' in body + assert 'group="drop"' not in body + + +def test_staleness_retain_queries_prunes_removed_queries() -> None: + exporter = Exporter() + exporter.record_series_fit_success( + group="g", + query_id="keep", + model_name="m", + series_key="k", + labels={"id": "keep", "group": "g", "model": "m"}, + ) + exporter.record_series_fit_success( + group="g", + query_id="drop", + model_name="m", + series_key="k", + labels={"id": "drop", "group": "g", "model": "m"}, + ) + exporter.retain_staleness_queries({("g", "keep")}) + body = exporter.render().decode() + assert 'id="keep"' in body + assert 'id="drop"' not in body + + +def test_prune_stale_staleness_drops_rows_older_than_threshold() -> None: + """A model dropped from config (or an ephemeral series) gets pruned by age. + + Guards the two leaks the per-group / per-query retention helpers + can't reach: a model name removed from ``defaults.models`` (its + rows never refresh again) and ephemeral series whose label set + rotates (per-pod metrics with short pod lifetimes). Both manifest + as a staleness row that's never re-touched by ``record_series_fit_success``; + the time-based sweep is the only mechanism that cleans them up. + """ + exporter = Exporter() + exporter.record_series_fit_success( + group="g", + query_id="q", + model_name="ghost", + series_key="k", + labels={"id": "q", "group": "g", "model": "ghost"}, + ) + exporter.record_series_fit_success( + group="g", + query_id="q", + model_name="live", + series_key="k", + labels={"id": "q", "group": "g", "model": "live"}, + ) + # Age the ``ghost`` row past the threshold and leave the ``live`` + # row fresh. + ghost_key = ("g", "q", "ghost", "k") + live_key = ("g", "q", "live", "k") + ghost_ts, ghost_labels = exporter._series_last_success[ghost_key] + exporter._series_last_success[ghost_key] = (ghost_ts - 100_000.0, ghost_labels) + + pruned = exporter.prune_stale_staleness(86_400.0) + + assert pruned == 1 + assert ghost_key not in exporter._series_last_success + assert live_key in exporter._series_last_success + + +def test_prune_stale_staleness_no_op_when_threshold_zero_or_negative() -> None: + """A zero/negative threshold disables pruning so callers can opt out.""" + exporter = Exporter() + exporter.record_series_fit_success( + group="g", + query_id="q", + model_name="m", + series_key="k", + labels={"id": "q", "group": "g", "model": "m"}, + ) + # Age the row well past any sane threshold. + key = ("g", "q", "m", "k") + ts, labels = exporter._series_last_success[key] + exporter._series_last_success[key] = (ts - 10_000_000.0, labels) + + assert exporter.prune_stale_staleness(0.0) == 0 + assert exporter.prune_stale_staleness(-1.0) == 0 + assert key in exporter._series_last_success diff --git a/forecaster/tests/test_health.py b/forecaster/tests/test_health.py new file mode 100644 index 0000000..78da621 --- /dev/null +++ b/forecaster/tests/test_health.py @@ -0,0 +1,353 @@ +"""Unit tests for the service-level forecast health composite.""" + +from __future__ import annotations + +import math +from datetime import datetime + +import pytest + +from promforecast.config import ( + AccuracyConfig, + Config, + DatasourceConfig, + DefaultsConfig, + EmissionConfig, + GovernanceConfig, + GroupConfig, + HealthScoreConfig, + QueryConfig, + SafetyConfig, + ServerConfig, +) +from promforecast.exporter import Exporter, GroupSnapshot, QualityPoint +from promforecast.health import compute_service_health_scores +from promforecast.runner import Runner +from promforecast.source import PromSource, SeriesFrame + + +def _qp(score: float, **labels: str) -> QualityPoint: + return QualityPoint(labels=labels, score=score) + + +def test_aggregate_buckets_quality_by_single_label() -> None: + """A single aggregate_by label produces one row per distinct value.""" + cfg = HealthScoreConfig(aggregate_by=["service"]) + points = [ + _qp(0.8, service="checkout", id="latency", group="g"), + _qp(0.6, service="checkout", id="errors", group="g"), + _qp(0.4, service="search", id="latency", group="g"), + ] + scores = compute_service_health_scores(points, cfg) + assert len(scores) == 2 + by_service = {s.labels["service"]: s for s in scores} + assert math.isclose(by_service["checkout"].score, 0.7) + assert by_service["checkout"].series_count == 2 + assert math.isclose(by_service["search"].score, 0.4) + assert by_service["search"].series_count == 1 + + +def test_aggregate_by_multiple_labels_distinguishes_buckets() -> None: + """Two aggregate_by labels produce one row per (service, environment) pair.""" + cfg = HealthScoreConfig(aggregate_by=["service", "environment"]) + points = [ + _qp(0.9, service="checkout", environment="prod"), + _qp(0.8, service="checkout", environment="prod"), + _qp(0.5, service="checkout", environment="staging"), + ] + scores = compute_service_health_scores(points, cfg) + assert len(scores) == 2 + prod = next(s for s in scores if s.labels["environment"] == "prod") + staging = next(s for s in scores if s.labels["environment"] == "staging") + assert math.isclose(prod.score, 0.85) + assert math.isclose(staging.score, 0.5) + + +def test_aggregate_drops_points_missing_required_label() -> None: + """A series missing one of the aggregate_by labels is dropped. + + Without the drop, an unlabelled forecast would slip into every + bucket and skew the report. + """ + cfg = HealthScoreConfig(aggregate_by=["service", "environment"]) + points = [ + _qp(0.9, service="checkout", environment="prod"), + _qp(0.1, service="checkout"), # missing environment + _qp(0.5, environment="prod"), # missing service + ] + scores = compute_service_health_scores(points, cfg) + assert len(scores) == 1 + assert math.isclose(scores[0].score, 0.9) + + +def test_aggregate_drops_empty_string_label_values() -> None: + """``label=""`` is treated as missing — likely a labelling bug.""" + cfg = HealthScoreConfig(aggregate_by=["service"]) + points = [ + _qp(0.9, service="checkout"), + _qp(0.1, service=""), + ] + scores = compute_service_health_scores(points, cfg) + assert len(scores) == 1 + assert scores[0].labels["service"] == "checkout" + + +def test_aggregate_drops_non_finite_scores() -> None: + """A NaN quality score doesn't poison its bucket.""" + cfg = HealthScoreConfig(aggregate_by=["service"]) + points = [ + _qp(0.8, service="checkout"), + _qp(float("nan"), service="checkout"), + _qp(float("inf"), service="checkout"), + ] + scores = compute_service_health_scores(points, cfg) + assert len(scores) == 1 + # Only the 0.8 row contributed. + assert math.isclose(scores[0].score, 0.8) + assert scores[0].series_count == 1 + + +def test_quality_floor_drops_low_scores_before_aggregation() -> None: + """``quality_floor`` filters individual scores before the bucket mean. + + Useful when cold-start / low-quality forecasts should not drag a + service's composite below the alert gate (typically 0.6). + """ + cfg = HealthScoreConfig(aggregate_by=["service"], quality_floor=0.5) + points = [ + _qp(0.9, service="checkout"), + _qp(0.4, service="checkout"), # dropped — below floor + _qp(0.7, service="checkout"), + ] + scores = compute_service_health_scores(points, cfg) + assert len(scores) == 1 + assert math.isclose(scores[0].score, 0.8) + assert scores[0].series_count == 2 + + +def test_quality_floor_can_eliminate_a_bucket_entirely() -> None: + """A bucket whose every score is below the floor disappears. + + The story: when every series for a service is too low-quality to + be trustworthy, surfacing a zero-row sample would mislead readers + of the executive dashboard. Better to omit the bucket and let the + "bucket missing" be the signal. + """ + cfg = HealthScoreConfig(aggregate_by=["service"], quality_floor=0.5) + points = [ + _qp(0.4, service="checkout"), + _qp(0.3, service="checkout"), + _qp(0.9, service="payments"), + ] + scores = compute_service_health_scores(points, cfg) + assert len(scores) == 1 + assert scores[0].labels["service"] == "payments" + + +def test_emitted_labels_carry_aggregator_and_weight_by() -> None: + """The aggregator / weight_by config rides as documentation labels. + + A Grafana panel can read which interpretation produced the number + alongside the bucket dimensions. + """ + cfg = HealthScoreConfig( + aggregate_by=["service"], + weight_by="forecast_series_count", + ) + points = [_qp(0.9, service="checkout"), _qp(0.7, service="checkout")] + scores = compute_service_health_scores(points, cfg) + assert scores[0].labels["aggregator"] == "weighted_mean" + assert scores[0].labels["weight_by"] == "forecast_series_count" + assert scores[0].labels["series_count"] == "2" + + +def test_empty_inputs_produce_no_rows() -> None: + """An empty quality_points list emits nothing, not a NaN row.""" + cfg = HealthScoreConfig(aggregate_by=["service"]) + assert compute_service_health_scores([], cfg) == [] + + +def test_output_is_deterministic_across_runs() -> None: + """Bucket ordering is stable so Grafana series colours don't churn.""" + cfg = HealthScoreConfig(aggregate_by=["service"]) + points = [ + _qp(0.8, service="zeta"), + _qp(0.6, service="alpha"), + _qp(0.4, service="mu"), + ] + first = compute_service_health_scores(points, cfg) + second = compute_service_health_scores(points, cfg) + assert [s.labels["service"] for s in first] == [s.labels["service"] for s in second] + assert [s.labels["service"] for s in first] == ["alpha", "mu", "zeta"] + + +# ---- Runner-level wiring --------------------------------------------------- + + +class _FakeSource(PromSource): + def __init__(self, by_query: dict[str, list[SeriesFrame]]) -> None: + self._by_query = by_query + + async def query_range( + self, + promql: str, + start: datetime, + end: datetime, + step_seconds: int, + ) -> list[SeriesFrame]: + return self._by_query.get(promql, []) + + async def aclose(self) -> None: + return None + + +def _runner_config(governance: GovernanceConfig | None = None) -> Config: + """Minimal config — the runner test mutates exporter state directly.""" + return Config( + datasource=DatasourceConfig(url="http://x"), + server=ServerConfig(), + safety=SafetyConfig(), + defaults=DefaultsConfig( + min_points=5, + models=["SeasonalNaive"], + accuracy=AccuracyConfig(evaluate=False, auto_select=False), + emission=EmissionConfig(quality=True, deviation=False), + ), + groups=[ + GroupConfig(name="g", queries=[QueryConfig(id="m", promql="up")]), + ], + governance=governance or GovernanceConfig(), + ) + + +def _qsnap(group_name: str, quality_points: list[QualityPoint]) -> GroupSnapshot: + return GroupSnapshot(group=group_name, quality_points=quality_points) + + +@pytest.mark.asyncio +async def test_runner_refresh_aggregates_across_groups() -> None: + """``_refresh_service_health_scores`` aggregates across every live group. + + Bypasses ``run_group`` (which would overwrite our seeded snapshots + with empty fits) so we can isolate the cross-group aggregation + semantics. The runner's read-path walks every configured group's + snapshot, then publishes one row per bucket. + """ + cfg = _runner_config(GovernanceConfig(health_score=HealthScoreConfig(aggregate_by=["service"]))) + # Two groups in the live config so the aggregator sweeps both + # snapshots; the runner's source is unused on this path. + cfg.groups.append(GroupConfig(name="g2", queries=[QueryConfig(id="x", promql="up")])) + exporter = Exporter() + exporter.update( + _qsnap( + "g", + [ + _qp(0.9, service="checkout", id="latency"), + _qp(0.7, service="checkout", id="errors"), + _qp(0.5, service="payments", id="latency"), + ], + ) + ) + exporter.update(_qsnap("g2", [_qp(0.3, service="payments", id="errors")])) + + runner = Runner(config=cfg, source=_FakeSource({"up": []}), exporter=exporter) + runner._refresh_service_health_scores() + + body = exporter.render().decode() + assert "forecast_service_health_score{" in body + # Per-service mean across both groups: + # checkout = (0.9 + 0.7) / 2 = 0.8; + # payments = (0.5 + 0.3) / 2 = 0.4. + checkout_line = next( + line + for line in body.splitlines() + if line.startswith("forecast_service_health_score{") and 'service="checkout"' in line + ) + assert " 0.8" in checkout_line + payments_line = next( + line + for line in body.splitlines() + if line.startswith("forecast_service_health_score{") and 'service="payments"' in line + ) + assert " 0.4" in payments_line + + +@pytest.mark.asyncio +async def test_runner_clears_health_score_when_feature_disabled() -> None: + """``apply_config`` clears the gauge when ``governance.health_score`` is removed.""" + cfg = _runner_config(GovernanceConfig(health_score=HealthScoreConfig(aggregate_by=["service"]))) + exporter = Exporter() + exporter.update(_qsnap("g", [_qp(0.9, service="checkout", id="latency")])) + + runner = Runner(config=cfg, source=_FakeSource({"up": []}), exporter=exporter) + runner._refresh_service_health_scores() + initial = exporter.render().decode() + assert "forecast_service_health_score{" in initial + + # Reload with the feature turned off — the gauge must disappear. + disabled = _runner_config(GovernanceConfig(health_score=None)) + await runner.apply_config(disabled) + after = exporter.render().decode() + assert "forecast_service_health_score{" not in after + + +@pytest.mark.asyncio +async def test_runner_skips_health_refresh_when_unconfigured() -> None: + """A config without ``governance.health_score`` emits no gauge rows.""" + cfg = _runner_config() # no governance block + exporter = Exporter() + exporter.update(_qsnap("g", [_qp(0.9, service="checkout", id="latency")])) + + runner = Runner(config=cfg, source=_FakeSource({"up": []}), exporter=exporter) + runner._refresh_service_health_scores() + body = exporter.render().decode() + assert "forecast_service_health_score{" not in body + + +def test_exporter_health_bucket_key_is_order_stable() -> None: + """``replace_service_health_scores`` deduplicates regardless of label-insertion order. + + Without sorting (key, value) pairs by key, two emissions for the + same logical bucket but with different insertion orders would + produce two distinct rows on /metrics — a silent regression that + would only surface as an unexplained doubled row count under + load. Sort-on-key keeps the dedup deterministic. + """ + from promforecast.exporter import ServiceHealthScorePoint # noqa: PLC0415 + + exporter = Exporter() + # Two points for the same conceptual bucket — same label values + # but inserted in different orders. The second emission must + # REPLACE the first, not add a second row. + first = ServiceHealthScorePoint( + labels={ + "service": "checkout", + "environment": "prod", + "aggregator": "weighted_mean", + "weight_by": "forecast_series_count", + "series_count": "10", + }, + score=0.8, + ) + second = ServiceHealthScorePoint( + labels={ + # Same labels, different insertion order. + "environment": "prod", + "service": "checkout", + "weight_by": "forecast_series_count", + "aggregator": "weighted_mean", + "series_count": "12", + }, + score=0.9, + ) + exporter.replace_service_health_scores([first]) + exporter.replace_service_health_scores([second]) + state = exporter._service_health_state() + assert len(state) == 1 + assert state[0].score == 0.9 + # The render path emits exactly one row for the bucket. + body = exporter.render().decode() + bucket_lines = [ + line for line in body.splitlines() if line.startswith("forecast_service_health_score{") + ] + assert len(bucket_lines) == 1 diff --git a/forecaster/tests/test_inspect.py b/forecaster/tests/test_inspect.py new file mode 100644 index 0000000..c5282fc --- /dev/null +++ b/forecaster/tests/test_inspect.py @@ -0,0 +1,343 @@ +"""Tests for the inspector CLI's pure helpers. + +The full inspector pipeline (config load -> PromQL fetch -> model +fit) is exercised end-to-end via docker-compose; here we cover the +pure parsing / rendering helpers + a synthetic-source happy path so +CI doesn't need network egress to verify the table shape. +""" + +from __future__ import annotations + +import io +import math +from pathlib import Path + +import pytest + +from promforecast.inspect import ( + BacktestFoldRow, + BacktestSummary, + InputStats, + InspectError, + InspectReport, + parse_series_filter, + render_text, + run_inspect, +) + + +def test_parse_series_filter_handles_braces() -> None: + assert parse_series_filter('{instance="host-a",device="sda1"}') == { + "instance": "host-a", + "device": "sda1", + } + + +def test_parse_series_filter_accepts_bare_pairs() -> None: + assert parse_series_filter('instance="host-a"') == {"instance": "host-a"} + + +def test_parse_series_filter_returns_none_on_empty() -> None: + assert parse_series_filter(None) is None + assert parse_series_filter("") is None + assert parse_series_filter("{}") is None + + +def test_parse_series_filter_rejects_missing_equals() -> None: + with pytest.raises(InspectError): + parse_series_filter("{instance}") + + +def test_render_text_includes_each_section() -> None: + report = InspectReport( + query_id="node_cpu_busy_pct", + group="node_capacity", + model="SeasonalNaive", + series_labels={"instance": "host-a"}, + step_seconds=60, + season_length=1440, + horizon_steps=12, + input_stats=InputStats( + point_count=200, + gap_count=2, + largest_gap_seconds=120.0, + lookback_start="2026-05-23T00:00:00+00:00", + lookback_end="2026-05-24T00:00:00+00:00", + ), + forecast_timestamps=["2026-05-24T01:00:00+00:00"], + forecast_yhat=[42.0], + forecast_bands={"95": {"lower": [30.0], "upper": [50.0]}}, + preprocessing={"gaps_filled": {"forward_fill": 2}, "outliers_replaced": 0}, + backtest=BacktestSummary( + model="SeasonalNaive", + n_holdout=12, + mape=4.5, + mase=0.8, + rows=[ + BacktestFoldRow( + ds="2026-05-24T00:00:00+00:00", + actual=41.0, + yhat=42.5, + bands={"95": {"lower": 30.0, "upper": 50.0}}, + ) + ], + ), + ) + text = render_text(report) + assert "Query: node_cpu_busy_pct (group node_capacity)" in text + assert "Model: SeasonalNaive" in text + assert "instance=host-a" in text + assert "Forecast (first 1 of 1 rows):" in text + assert "yhat=42.0000" in text + assert "[95: 30.0000..50.0000]" in text + assert "Backtest" in text + assert "MAPE : 4.5000%" in text + assert "MASE : 0.8000" in text + + +def test_render_text_handles_nan_values() -> None: + report = InspectReport( + query_id="m", + group="g", + model="X", + series_labels={}, + step_seconds=60, + season_length=1, + horizon_steps=1, + input_stats=InputStats( + point_count=0, + gap_count=0, + largest_gap_seconds=0.0, + lookback_start="", + lookback_end="", + ), + forecast_timestamps=[], + forecast_yhat=[], + forecast_bands={}, + preprocessing={}, + backtest=BacktestSummary( + model="X", + n_holdout=0, + mape=math.nan, + mase=math.nan, + rows=[], + ), + ) + text = render_text(report) + assert "MAPE : NaN%" in text + assert "MASE : NaN" in text + assert "(no forecast rows)" in text + + +def test_inspect_report_to_dict_round_trip() -> None: + """The JSON envelope strips non-finite numbers to ``None`` so jq is happy.""" + report = InspectReport( + query_id="m", + group="g", + model="X", + series_labels={"instance": "h"}, + step_seconds=60, + season_length=1, + horizon_steps=1, + input_stats=InputStats( + point_count=1, + gap_count=0, + largest_gap_seconds=0.0, + lookback_start="t0", + lookback_end="t1", + ), + forecast_timestamps=["t2"], + forecast_yhat=[1.0], + forecast_bands={"95": {"lower": [0.5], "upper": [1.5]}}, + preprocessing={}, + backtest=BacktestSummary( + model="X", + n_holdout=1, + mape=math.nan, + mase=2.0, + rows=[ + BacktestFoldRow( + ds="t2", + actual=math.nan, + yhat=1.0, + bands={"95": {"lower": 0.5, "upper": 1.5}}, + ) + ], + ), + ) + payload = report.to_dict() + assert payload["query_id"] == "m" + assert payload["forecast"]["bands"]["95"]["lower"] == [0.5] + assert payload["backtest"]["mape"] is None # NaN -> None + assert payload["backtest"]["mase"] == 2.0 + assert payload["backtest"]["rows"][0]["actual"] is None + + +def test_inspector_applies_count_aware_safeguards( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + """A ``data_profile: count`` query must clip negatives in inspector output. + + Guards the runner-parity contract spelled out in + ``docs/operations/inspect.md``: the inspector exists to debug "what + the live forecaster would do right now", so count-aware safeguards + (``clip_non_negative``, ``round_to_integer``) MUST be applied + before the inspector report renders. Without this, an operator + inspecting a count-shaped query would see raw model output + (possibly negative or fractional) that the live exporter would + have rejected. + """ + from datetime import UTC # noqa: PLC0415 + from datetime import datetime as _dt # noqa: PLC0415 + from datetime import timedelta as _td # noqa: PLC0415 + + import pandas as pd # noqa: PLC0415 + + from promforecast import inspect as inspect_module # noqa: PLC0415 + from promforecast import models as model_registry # noqa: PLC0415 + from promforecast.source import PromSource, SeriesFrame # noqa: PLC0415 + + class _NegativeModel: + name = "AutoARIMA" + + def fit_predict( + self, + df: pd.DataFrame, + horizon_steps: int, + freq: str, + levels: list[int], + regressors: pd.DataFrame | None, + ) -> pd.DataFrame: + start_ts = df["ds"].iloc[-1] + pd.Timedelta(freq) + ts = pd.date_range(start=start_ts, periods=horizon_steps, freq=freq) + out: dict[str, object] = {"ds": ts, "yhat": [-5.0] * horizon_steps} + for level in levels: + out[f"yhat_lower_{level}"] = [-10.0] * horizon_steps + out[f"yhat_upper_{level}"] = [0.5] * horizon_steps + return pd.DataFrame(out) + + class _FakeInspectSource(PromSource): + def __init__(self) -> None: + now = _dt.now(tz=UTC) + step = _td(minutes=1) + self._frames = [ + SeriesFrame( + labels={"instance": "host-a"}, + timestamps=[now - step * (60 - i) for i in range(60)], + values=[float(i) for i in range(60)], + ) + ] + + async def query_range( + self, + promql: str, + start: _dt, + end: _dt, + step_seconds: int, + ) -> list[SeriesFrame]: + return list(self._frames) + + async def aclose(self) -> None: + return None + + monkeypatch.setattr( + model_registry, + "build", + lambda name, season_length, params=None: _NegativeModel(), + ) + monkeypatch.setattr(inspect_module, "PromSource", lambda **_kw: _FakeInspectSource()) + + cfg_body = """ +apiVersion: promforecast.io/v1 +datasource: + url: http://127.0.0.1:1 +defaults: + min_points: 10 + models: [AutoARIMA] + confidence_levels: [80] + step: 1m + lookback: 1h + horizon: 5m + accuracy: + evaluate: false + auto_select: false + horizons: [5m] + emission: + deviation: false + quality: false + contribution: false +groups: + - name: g + queries: + - id: q + promql: up + data_profile: count +""" + path = tmp_path / "cfg.yaml" + path.write_text(cfg_body) + out = io.StringIO() + rc = run_inspect( + config_path=path, + query_id="q", + series_filter=None, + horizon_seconds=None, + lookback_seconds=None, + model=None, + backtest=False, + json_output=True, + out=out, + ) + assert rc == 0, out.getvalue() + import json # noqa: PLC0415 + + payload = json.loads(out.getvalue()) + # ``data_profile: count`` implies ``clip_non_negative`` — every + # forecast yhat must be >= 0, even though the model returned -5. + assert all(v >= 0.0 for v in payload["forecast"]["yhat"]), payload["forecast"]["yhat"] + # Bands must clip too. + assert all(v >= 0.0 for v in payload["forecast"]["bands"]["80"]["lower"]) + + +def test_run_inspect_returns_2_on_missing_config() -> None: + """A missing config file is a 2 (datasource/I/O error).""" + rc = run_inspect( + config_path=Path("/nonexistent/forecaster/config.yaml"), + query_id="m", + series_filter=None, + horizon_seconds=None, + lookback_seconds=None, + model=None, + backtest=False, + json_output=False, + out=io.StringIO(), + ) + assert rc == 2 + + +def test_run_inspect_returns_1_on_unknown_query(tmp_path: Path) -> None: + """An unknown query id fails at find_query, returning exit 1.""" + cfg_body = """ +apiVersion: promforecast.io/v1 +datasource: + url: http://127.0.0.1:1 +groups: + - name: g + queries: + - id: existing + promql: up +""" + path = tmp_path / "cfg.yaml" + path.write_text(cfg_body) + rc = run_inspect( + config_path=path, + query_id="missing", + series_filter=None, + horizon_seconds=None, + lookback_seconds=None, + model=None, + backtest=False, + json_output=False, + out=io.StringIO(), + ) + assert rc == 1 diff --git a/forecaster/tests/test_point_factory.py b/forecaster/tests/test_point_factory.py index 7fe09ad..869d41d 100644 --- a/forecaster/tests/test_point_factory.py +++ b/forecaster/tests/test_point_factory.py @@ -89,9 +89,16 @@ def test_quality_points_skip_models_without_backtest() -> None: coverage=1.0, ) ) - assert len(points) == 1 - assert points[0].labels["model"] == "AutoARIMA" - assert math.isfinite(points[0].score) + # Aggregated (``horizon=""``) row + one per-horizon row from the + # single backtest result. SeasonalNaive contributes nothing + # because it has no backtest data. + assert len(points) == 2 + models = {p.labels["model"] for p in points} + assert models == {"AutoARIMA"} + horizons = {p.labels.get("horizon", "") for p in points} + assert horizons == {"", "1h"} + for p in points: + assert math.isfinite(p.score) def test_deviation_points_skip_models_without_band_at_level() -> None: diff --git a/forecaster/tests/test_preprocess.py b/forecaster/tests/test_preprocess.py index 92f552a..d918d18 100644 --- a/forecaster/tests/test_preprocess.py +++ b/forecaster/tests/test_preprocess.py @@ -6,8 +6,10 @@ from datetime import UTC, datetime, timedelta import numpy as np +import pytest from promforecast.config import ( + BlacklistWindowConfig, ChangePointConfig, GapImputationConfig, OutlierConfig, @@ -293,3 +295,169 @@ def test_preprocess_result_imputed_fraction_zero_clean() -> None: assert isinstance(result, PreprocessResult) assert result.imputed_fraction == 0.0 assert result.values == [] + + +# ---------- blacklist windows --------------------------------------------- + + +def _config_with_blacklist(windows: list[BlacklistWindowConfig]) -> PreprocessConfig: + base = _config() + return PreprocessConfig( + gaps=base.gaps, + outliers=base.outliers, + change_point=base.change_point, + blacklist_windows=windows, + ) + + +def test_blacklist_absolute_window_drops_samples() -> None: + # 60 1-minute samples; drop the middle 20. + ts, vs = _regular_series(60) + start = ts[20].isoformat().replace("+00:00", "Z") + end = ts[40].isoformat().replace("+00:00", "Z") + window = BlacklistWindowConfig(start=start, end=end, reason="incident") + result = preprocess_series( + timestamps=ts, + values=vs, + step=timedelta(minutes=1), + config=_config_with_blacklist([window]), + ) + assert not result.dropped + assert result.blacklisted_samples == {"incident": 20} + # The default ``forward_fill`` gap strategy imputes the blacklisted + # span back to a regular grid so the series length is unchanged; + # what changes is the counter (20 reasoned samples removed) and the + # effective-lookback gauge, which discounts those samples. + assert len(result.values) == 60 + expected_total = (ts[-1] - ts[0]).total_seconds() + assert result.effective_lookback_seconds < expected_total + # ``20 samples x 60s step`` worth of discount: ~1200s. + assert result.effective_lookback_seconds == expected_total - 20 * 60 + + +def test_blacklist_default_empty_reason() -> None: + ts, vs = _regular_series(60) + start = ts[10].isoformat().replace("+00:00", "Z") + end = ts[15].isoformat().replace("+00:00", "Z") + window = BlacklistWindowConfig(start=start, end=end) + result = preprocess_series( + timestamps=ts, + values=vs, + step=timedelta(minutes=1), + config=_config_with_blacklist([window]), + ) + assert result.blacklisted_samples == {"": 5} + + +def test_blacklist_annual_recurring_fires_per_year() -> None: + # 3-year lookback at 1h step (around 365*3*24 ≈ 26280 samples is too + # heavy for a unit test). Build a sparse-but-representative two-year + # input covering January in each year so the annual window fires twice. + points: list[datetime] = [] + for year in (2024, 2025): + for day in range(1, 31): + points.append(datetime(year, 1, day, 12, 0, tzinfo=UTC)) + values = [float(i) for i in range(len(points))] + window = BlacklistWindowConfig( + start="01-01T00:00", + end="01-02T00:00", + reason="newyear", + recurring="annual", + ) + result = preprocess_series( + timestamps=points, + values=values, + step=timedelta(hours=24), + config=_config_with_blacklist([window]), + ) + # One sample per year falls inside the New Year's Day window. + assert result.blacklisted_samples == {"newyear": 2} + + +def test_blacklist_consuming_all_samples_drops_series() -> None: + ts, vs = _regular_series(30) + start = (ts[0] - timedelta(hours=1)).isoformat().replace("+00:00", "Z") + end = (ts[-1] + timedelta(hours=1)).isoformat().replace("+00:00", "Z") + window = BlacklistWindowConfig(start=start, end=end, reason="freeze") + result = preprocess_series( + timestamps=ts, + values=vs, + step=timedelta(minutes=1), + config=_config_with_blacklist([window]), + ) + assert result.dropped + assert result.drop_reason == "blacklist_removed_all_samples" + assert result.blacklisted_samples == {"freeze": 30} + + +def test_blacklist_rejects_malformed_window() -> None: + with pytest.raises(ValueError, match="ISO-8601"): + BlacklistWindowConfig(start="2024-11-29", end="2024-11-30") + + with pytest.raises(ValueError, match="MM-DDTHH"): + BlacklistWindowConfig( + start="11-29", + end="11-30", + recurring="annual", + ) + + +def test_blacklist_annual_window_anchored_in_local_timezone() -> None: + """An annual window configured with ``timezone: America/New_York`` + should flip at local midnight, not UTC-midnight. A hard test: a + sample at 04:00 UTC on Jan 1 is 23:00 on Dec 31 in NYC and must + be EXCLUDED from a ``01-01T00:00`` → ``01-02T00:00`` NYC window; + a sample at 06:00 UTC (01:00 NYC) is INSIDE the window. + """ + # Pick two samples around midnight-NYC on New Year's Day 2025. + # NYC is UTC-5 in January (EST). + ts = [ + datetime(2025, 1, 1, 4, 0, tzinfo=UTC), # 23:00 EST on Dec 31 - OUTSIDE + datetime(2025, 1, 1, 6, 0, tzinfo=UTC), # 01:00 EST on Jan 1 - INSIDE + datetime(2025, 1, 2, 4, 0, tzinfo=UTC), # 23:00 EST on Jan 1 - INSIDE + datetime(2025, 1, 2, 6, 0, tzinfo=UTC), # 01:00 EST on Jan 2 - OUTSIDE + ] + values = [1.0, 2.0, 3.0, 4.0] + window = BlacklistWindowConfig( + start="01-01T00:00", + end="01-02T00:00", + recurring="annual", + timezone="America/New_York", + reason="ny_newyear", + ) + result = preprocess_series( + timestamps=ts, + values=values, + step=timedelta(hours=2), + config=_config_with_blacklist([window]), + ) + # Two samples (the 06:00 UTC and the 04:00 UTC the next day) fall + # inside the local-midnight-to-local-midnight window. + assert result.blacklisted_samples == {"ny_newyear": 2} + + +def test_blacklist_annual_unknown_timezone_is_skipped() -> None: + """An unparseable timezone should not crash the fit pipeline. + + The pydantic validator does not (yet) check that the timezone is + an IANA name; an operator typo would otherwise reach the runtime + parser. ``_parse_annual_window`` returns ``None`` on + ``ZoneInfoNotFoundError`` and the window is silently skipped so + the rest of the series is still fit. + """ + ts, vs = _regular_series(60) + window = BlacklistWindowConfig( + start="01-01T00:00", + end="01-02T00:00", + recurring="annual", + timezone="Not/A_Real_Zone", + ) + result = preprocess_series( + timestamps=ts, + values=vs, + step=timedelta(minutes=1), + config=_config_with_blacklist([window]), + ) + # The window failed to expand; no samples dropped, no counter ticks. + assert result.blacklisted_samples == {} + assert not result.dropped diff --git a/forecaster/tests/test_revision.py b/forecaster/tests/test_revision.py new file mode 100644 index 0000000..e6cdd3e --- /dev/null +++ b/forecaster/tests/test_revision.py @@ -0,0 +1,94 @@ +"""Unit tests for the forecast-to-forecast revision module.""" + +from __future__ import annotations + +import math + +import pytest + +from promforecast.revision import RevisionCache, compute_revision_magnitude + + +def test_compute_revision_magnitude_zero_when_unchanged() -> None: + assert compute_revision_magnitude(previous=10.0, current=10.0) == 0.0 + + +def test_compute_revision_magnitude_relative_shift() -> None: + # 30% increase + assert compute_revision_magnitude(previous=100.0, current=130.0) == pytest.approx(0.3) + # Sign-insensitive + assert compute_revision_magnitude(previous=100.0, current=70.0) == pytest.approx(0.3) + + +def test_compute_revision_magnitude_handles_zero_previous() -> None: + # Should not blow up; uses floor 1e-9 + magnitude = compute_revision_magnitude(previous=0.0, current=0.5) + assert math.isfinite(magnitude) + # 0.5 / 1e-9 = 5e8 — very large, indicating "huge swing from zero" + assert magnitude > 1e6 + + +def test_compute_revision_magnitude_nan_on_non_finite() -> None: + assert math.isnan(compute_revision_magnitude(previous=float("nan"), current=1.0)) + assert math.isnan(compute_revision_magnitude(previous=1.0, current=float("inf"))) + + +def test_revision_cache_round_trip() -> None: + cache = RevisionCache() + cache.put(group="g", query="q", series_key="s", model="m", horizon="1h", value=12.5) + assert cache.get(group="g", query="q", series_key="s", model="m", horizon="1h") == 12.5 + # Missing entry + assert cache.get(group="g", query="q", series_key="s", model="m", horizon="24h") is None + + +def test_revision_cache_skips_nan_writes() -> None: + cache = RevisionCache() + cache.put( + group="g", + query="q", + series_key="s", + model="m", + horizon="1h", + value=float("nan"), + ) + # NaN entries are dropped so the next fit sees "no prior data" + # rather than poisoning the diff. + assert cache.get(group="g", query="q", series_key="s", model="m", horizon="1h") is None + + +def test_revision_cache_prune_to_drops_dead_pairs() -> None: + cache = RevisionCache() + cache.put(group="g", query="q1", series_key="s", model="m", horizon="1h", value=1.0) + cache.put(group="g", query="q2", series_key="s", model="m", horizon="1h", value=2.0) + cache.prune_to({("g", "q1")}) + assert cache.get(group="g", query="q1", series_key="s", model="m", horizon="1h") == 1.0 + assert cache.get(group="g", query="q2", series_key="s", model="m", horizon="1h") is None + + +def test_revision_cache_evicts_oldest_when_over_capacity() -> None: + # Guard against series-key churn (ephemeral pod IPs, container IDs): + # without the LRU cap the cache would grow forever between reloads. + cache = RevisionCache(max_entries=3) + for i in range(5): + cache.put(group="g", query="q", series_key=f"s{i}", model="m", horizon="1h", value=float(i)) + assert len(cache) == 3 + # The three most recent writes survive; the older two are evicted. + assert cache.get(group="g", query="q", series_key="s0", model="m", horizon="1h") is None + assert cache.get(group="g", query="q", series_key="s1", model="m", horizon="1h") is None + assert cache.get(group="g", query="q", series_key="s4", model="m", horizon="1h") == 4.0 + + +def test_revision_cache_touch_on_read_keeps_hot_entries() -> None: + # A reader-touch on the oldest key promotes it to most-recent, so + # subsequent eviction takes a now-older entry instead. + cache = RevisionCache(max_entries=3) + cache.put(group="g", query="q", series_key="s0", model="m", horizon="1h", value=0.0) + cache.put(group="g", query="q", series_key="s1", model="m", horizon="1h", value=1.0) + cache.put(group="g", query="q", series_key="s2", model="m", horizon="1h", value=2.0) + # Touch s0 — it's now the most-recent entry. + assert cache.get(group="g", query="q", series_key="s0", model="m", horizon="1h") == 0.0 + # Adding s3 should evict s1 (the new oldest after the touch), not s0. + cache.put(group="g", query="q", series_key="s3", model="m", horizon="1h", value=3.0) + assert cache.get(group="g", query="q", series_key="s0", model="m", horizon="1h") == 0.0 + assert cache.get(group="g", query="q", series_key="s1", model="m", horizon="1h") is None + assert cache.get(group="g", query="q", series_key="s3", model="m", horizon="1h") == 3.0 diff --git a/forecaster/tests/test_right_sizing.py b/forecaster/tests/test_right_sizing.py new file mode 100644 index 0000000..8f72589 --- /dev/null +++ b/forecaster/tests/test_right_sizing.py @@ -0,0 +1,452 @@ +"""Tests for the right-sizing recommendation primitive + runner wiring.""" + +from __future__ import annotations + +import math +from datetime import UTC, datetime + +import pytest + +from promforecast.config import ( + AccuracyConfig, + Config, + DatasourceConfig, + DefaultsConfig, + EmissionConfig, + GroupConfig, + QueryConfig, + RecommendationsConfig, + ResourceSizingConfig, + ResourceSizingWorkloadConfig, + SafetyConfig, + ServerConfig, +) +from promforecast.exporter import Exporter, ForecastPoint, GroupSnapshot, QualityPoint +from promforecast.right_sizing import ( + compute_predicted_peak, + compute_recommendation, + confidence_for, +) +from promforecast.runner import Runner +from promforecast.source import PromSource, SeriesFrame + +# ---- Pure-function tests ---------------------------------------------------- + + +def _fp(metric: str, value: float, **labels: str) -> ForecastPoint: + return ForecastPoint(metric=metric, labels=labels, value=value) + + +def _qp(score: float, **labels: str) -> QualityPoint: + return QualityPoint(labels=labels, score=score) + + +def test_confidence_buckets_quality_score() -> None: + """Confidence label maps quality score into a categorical band.""" + assert confidence_for(0.9) == "high" + assert confidence_for(0.8) == "high" + assert confidence_for(0.7) == "medium" + assert confidence_for(0.6) == "medium" + assert confidence_for(0.5) == "low" + # Non-finite => low so we don't accidentally render a "high" label + # on a degenerate forecast. + assert confidence_for(float("nan")) == "low" + + +def test_peak_prefers_upper_band_at_highest_level() -> None: + """The peak uses ``yhat_upper`` at the highest configured level. + + Operators read the recommendation as a conservative cap; the + central forecast would systematically under-provision for the + variability the upper band represents. + """ + points = [ + _fp("svc_forecast", 10.0, model="m"), + _fp("svc_forecast_upper", 12.0, model="m", level="80"), + _fp("svc_forecast_upper", 15.0, model="m", level="95"), + _fp("svc_forecast_upper", 14.0, model="m", level="95"), + ] + assert compute_predicted_peak(points, "svc") == 15.0 + + +def test_peak_falls_back_to_central_when_no_band() -> None: + """Without confidence bands, the central forecast peak is the recommendation.""" + points = [ + _fp("svc_forecast", 5.0, model="m"), + _fp("svc_forecast", 7.0, model="m"), + _fp("svc_forecast", 6.0, model="m"), + ] + assert compute_predicted_peak(points, "svc") == 7.0 + + +def test_peak_returns_none_when_no_finite_samples() -> None: + """All-NaN forecast => no recommendation.""" + points = [ + _fp("svc_forecast", float("nan"), model="m"), + _fp("svc_forecast_upper", float("inf"), model="m", level="95"), + ] + assert compute_predicted_peak(points, "svc") is None + + +def test_peak_ignores_other_query_ids() -> None: + """A peak for ``svc`` must not be polluted by another query's forecast.""" + points = [ + _fp("svc_forecast", 5.0, model="m"), + _fp("other_forecast", 999.0, model="m"), + ] + assert compute_predicted_peak(points, "svc") == 5.0 + + +def _config_with_floor(floor: float = 0.6, headroom: float = 30.0) -> ResourceSizingConfig: + return ResourceSizingConfig( + enabled=True, + headroom_pct=headroom, + quality_floor=floor, + workloads=[ResourceSizingWorkloadConfig(workload="api", kind="cpu", forecast_query="svc")], + ) + + +def test_recommendation_applies_headroom_to_peak() -> None: + """Recommended = peak * (1 + headroom_pct/100).""" + cfg = _config_with_floor(floor=0.0, headroom=30.0) + rec = compute_recommendation( + workload=cfg.workloads[0], + config=cfg, + forecast_points=[_fp("svc_forecast_upper", 10.0, level="95")], + quality_points=[_qp(0.9, id="svc")], + ) + assert rec is not None + assert math.isclose(rec.recommended, 13.0) + assert rec.confidence == "high" + + +def test_recommendation_skipped_below_quality_floor() -> None: + """Mean quality below floor => no row emitted.""" + cfg = _config_with_floor(floor=0.6) + rec = compute_recommendation( + workload=cfg.workloads[0], + config=cfg, + forecast_points=[_fp("svc_forecast_upper", 10.0, level="95")], + quality_points=[_qp(0.5, id="svc")], + ) + assert rec is None + + +def test_recommendation_quality_gate_uses_mean_not_max() -> None: + """A single high-quality emission alongside bad ones must NOT pass the gate. + + The peak we compute can come from any emitted model — gating on + the max-quality model wouldn't constrain which model actually + contributed the peak. Mean-quality keeps the gate honest: "the + forecast is broadly trustworthy", not "at least one model is + trustworthy". + """ + cfg = _config_with_floor(floor=0.6) + # Mean of (0.9, 0.3, 0.3) = 0.5 — below the 0.6 floor — even + # though one model is well above it. + rec = compute_recommendation( + workload=cfg.workloads[0], + config=cfg, + forecast_points=[_fp("svc_forecast_upper", 10.0, level="95")], + quality_points=[ + _qp(0.9, id="svc"), + _qp(0.3, id="svc"), + _qp(0.3, id="svc"), + ], + ) + assert rec is None + + +def test_recommendation_skipped_when_quality_missing() -> None: + """A query with no quality_points yet => no recommendation. + + Right-sizing on an unmeasured forecast would systematically + surprise operators when the score eventually catches up. + """ + cfg = _config_with_floor() + rec = compute_recommendation( + workload=cfg.workloads[0], + config=cfg, + forecast_points=[_fp("svc_forecast_upper", 10.0, level="95")], + quality_points=[], + ) + assert rec is None + + +def test_recommendation_with_current_computes_savings_pct() -> None: + """``current`` enables the sibling savings_pct emission.""" + cfg = _config_with_floor(floor=0.0, headroom=30.0) + rec = compute_recommendation( + workload=cfg.workloads[0], + config=cfg, + forecast_points=[_fp("svc_forecast_upper", 10.0, level="95")], + quality_points=[_qp(0.9, id="svc")], + current=20.0, + ) + assert rec is not None + # current=20, recommended=13 → savings = (20-13)/20*100 = 35 + assert math.isclose(rec.savings_pct, 35.0) + assert rec.current == 20.0 + + +def test_recommendation_negative_savings_means_upsize() -> None: + """A recommended value higher than current => negative savings. + + Operators read negative savings as "upsize, don't downsize" — + the forecast predicts a peak above the current limit. + """ + cfg = _config_with_floor(floor=0.0, headroom=30.0) + rec = compute_recommendation( + workload=cfg.workloads[0], + config=cfg, + forecast_points=[_fp("svc_forecast_upper", 20.0, level="95")], + quality_points=[_qp(0.9, id="svc")], + current=10.0, + ) + assert rec is not None + # current=10, recommended=26 → savings = (10-26)/10*100 = -160 + assert rec.savings_pct is not None + assert rec.savings_pct < 0 + + +def test_recommendation_drops_current_when_non_positive() -> None: + """``current <= 0`` is treated as missing — the percentage is undefined.""" + cfg = _config_with_floor(floor=0.0) + rec = compute_recommendation( + workload=cfg.workloads[0], + config=cfg, + forecast_points=[_fp("svc_forecast_upper", 10.0, level="95")], + quality_points=[_qp(0.9, id="svc")], + current=0.0, + ) + assert rec is not None + # current=0 is meaningless; savings_pct must be unset. + assert rec.savings_pct is None + assert rec.current is None + + +# ---- Runner-level wiring --------------------------------------------------- + + +class _FakeSource(PromSource): + def __init__(self, by_query: dict[str, list[SeriesFrame]] | None = None) -> None: + self._by_query = by_query or {} + + async def query_range( + self, + promql: str, + start: datetime, + end: datetime, + step_seconds: int, + ) -> list[SeriesFrame]: + return self._by_query.get(promql, []) + + async def aclose(self) -> None: + return None + + +def _runner_config( + *, + workloads: list[ResourceSizingWorkloadConfig] | None = None, + enabled: bool = True, +) -> Config: + return Config( + datasource=DatasourceConfig(url="http://x"), + server=ServerConfig(), + safety=SafetyConfig(), + defaults=DefaultsConfig( + min_points=5, + models=["SeasonalNaive"], + accuracy=AccuracyConfig(evaluate=False, auto_select=False), + emission=EmissionConfig(quality=True, deviation=False), + ), + groups=[GroupConfig(name="g", queries=[QueryConfig(id="svc", promql="up")])], + recommendations=RecommendationsConfig( + resource_sizing=ResourceSizingConfig( + enabled=enabled, + headroom_pct=30.0, + quality_floor=0.6, + workloads=workloads or [], + ) + ), + ) + + +@pytest.mark.asyncio +async def test_runner_emits_recommendation_after_refresh() -> None: + """The runner publishes a recommendation when the forecast snapshot is healthy.""" + cfg = _runner_config( + workloads=[ResourceSizingWorkloadConfig(workload="api", kind="cpu", forecast_query="svc")] + ) + exporter = Exporter() + exporter.update( + GroupSnapshot( + group="g", + points=[ForecastPoint(metric="svc_forecast_upper", labels={"level": "95"}, value=10.0)], + quality_points=[_qp(0.9, id="svc", group="g", model="SeasonalNaive")], + ) + ) + runner = Runner(config=cfg, source=_FakeSource(), exporter=exporter) + await runner._refresh_resource_sizing_recommendations() + + body = exporter.render().decode() + assert "forecast_recommended_resource_size{" in body + # 10 * 1.3 = 13.0 (no current_promql configured → no sibling rows). + rec_line = next( + line for line in body.splitlines() if line.startswith("forecast_recommended_resource_size{") + ) + assert " 13.0" in rec_line + assert 'workload="api"' in rec_line + assert 'kind="cpu"' in rec_line + assert 'confidence="high"' in rec_line + # Sibling current/savings_pct must NOT appear without current_promql. + assert "forecast_recommended_resource_current{" not in body + assert "forecast_recommended_resource_savings_pct{" not in body + + +@pytest.mark.asyncio +async def test_runner_omits_recommendation_when_quality_too_low() -> None: + """Recommendations gated by ``quality_floor`` aren't published. + + Better to publish nothing than to mislead an operator into + downsizing on an untrustworthy forecast. + """ + cfg = _runner_config( + workloads=[ResourceSizingWorkloadConfig(workload="api", kind="cpu", forecast_query="svc")] + ) + exporter = Exporter() + exporter.update( + GroupSnapshot( + group="g", + points=[ForecastPoint(metric="svc_forecast_upper", labels={"level": "95"}, value=10.0)], + # Below the 0.6 floor configured in _runner_config. + quality_points=[_qp(0.4, id="svc", group="g", model="SeasonalNaive")], + ) + ) + runner = Runner(config=cfg, source=_FakeSource(), exporter=exporter) + await runner._refresh_resource_sizing_recommendations() + + body = exporter.render().decode() + assert "forecast_recommended_resource_size{" not in body + + +@pytest.mark.asyncio +async def test_runner_clears_recommendations_when_feature_disabled() -> None: + """Disabling resource_sizing on reload drops the prior rows immediately.""" + cfg = _runner_config( + workloads=[ResourceSizingWorkloadConfig(workload="api", kind="cpu", forecast_query="svc")] + ) + exporter = Exporter() + exporter.update( + GroupSnapshot( + group="g", + points=[ForecastPoint(metric="svc_forecast_upper", labels={"level": "95"}, value=10.0)], + quality_points=[_qp(0.9, id="svc", group="g", model="SeasonalNaive")], + ) + ) + runner = Runner(config=cfg, source=_FakeSource(), exporter=exporter) + await runner._refresh_resource_sizing_recommendations() + initial = exporter.render().decode() + assert "forecast_recommended_resource_size{" in initial + + disabled = _runner_config(enabled=False) + await runner.apply_config(disabled) + after = exporter.render().decode() + assert "forecast_recommended_resource_size{" not in after + + +@pytest.mark.asyncio +async def test_runner_current_promql_success_emits_all_three_siblings() -> None: + """A resolved ``current_promql`` produces recommended + current + savings_pct. + + End-to-end verification of the merged sibling family: all three + metrics share the (workload, kind) label set; current/savings_pct + only emit when the operator wires the optional fetch. + """ + cfg = _runner_config( + workloads=[ + ResourceSizingWorkloadConfig( + workload="api", + kind="cpu", + forecast_query="svc", + current_promql="kube_pod_container_resource_requests", + ) + ] + ) + exporter = Exporter() + exporter.update( + GroupSnapshot( + group="g", + points=[ForecastPoint(metric="svc_forecast_upper", labels={"level": "95"}, value=10.0)], + quality_points=[_qp(0.9, id="svc", group="g", model="SeasonalNaive")], + ) + ) + # Source resolves the current_promql to a single sample of 20.0. + source = _FakeSource( + { + "kube_pod_container_resource_requests": [ + SeriesFrame( + labels={"workload": "api"}, + timestamps=[datetime(2026, 5, 25, tzinfo=UTC)], + values=[20.0], + ) + ] + } + ) + runner = Runner(config=cfg, source=source, exporter=exporter) + await runner._refresh_resource_sizing_recommendations() + + body = exporter.render().decode() + # Primary recommendation gauge: 10 * 1.3 = 13.0. + assert any( + line.startswith("forecast_recommended_resource_size{") and " 13.0" in line + for line in body.splitlines() + ) + # Current value from the source: 20.0. + assert any( + line.startswith("forecast_recommended_resource_current{") and " 20.0" in line + for line in body.splitlines() + ) + # Savings %: (20 - 13) / 20 * 100 = 35.0. + assert any( + line.startswith("forecast_recommended_resource_savings_pct{") and " 35.0" in line + for line in body.splitlines() + ) + + +@pytest.mark.asyncio +async def test_runner_current_promql_failure_still_emits_recommendation() -> None: + """A failed ``current_promql`` fetch must not suppress the recommended row. + + The recommendation is the primary signal; the current / savings_pct + siblings are nice-to-haves. A flaky current query shouldn't take + the whole row down. + """ + cfg = _runner_config( + workloads=[ + ResourceSizingWorkloadConfig( + workload="api", + kind="cpu", + forecast_query="svc", + current_promql="nonexistent_promql_returns_nothing", + ) + ] + ) + exporter = Exporter() + exporter.update( + GroupSnapshot( + group="g", + points=[ForecastPoint(metric="svc_forecast_upper", labels={"level": "95"}, value=10.0)], + quality_points=[_qp(0.9, id="svc", group="g", model="SeasonalNaive")], + ) + ) + # Source returns nothing for the current PromQL — the fetch is a + # no-op rather than an error, but the resulting None must still + # produce a recommended row. + runner = Runner(config=cfg, source=_FakeSource(), exporter=exporter) + await runner._refresh_resource_sizing_recommendations() + + body = exporter.render().decode() + assert "forecast_recommended_resource_size{" in body + # Without a current value, savings_pct must be absent. + assert "forecast_recommended_resource_savings_pct{" not in body diff --git a/forecaster/tests/test_runner_capacity.py b/forecaster/tests/test_runner_capacity.py index e08c01e..799d273 100644 --- a/forecaster/tests/test_runner_capacity.py +++ b/forecaster/tests/test_runner_capacity.py @@ -14,6 +14,7 @@ from promforecast.config import ( AccuracyConfig, + ActionCostConfig, Config, DatasourceConfig, DefaultsConfig, @@ -589,3 +590,106 @@ async def test_runner_emits_recommended_maintenance_window_family( assert 'id="cpu_busy"' in body assert 'rank="1"' in body assert 'rank="2"' in body + + +@pytest.mark.asyncio +async def test_runner_emits_threshold_action_cost_family( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A threshold with ``action_cost`` produces the cost gauge on /metrics. + + Independent of model fan-out — one row per (group, query, threshold). + The cost lands as the sample value, ``type`` rides the label set. + """ + from promforecast import models as model_registry # noqa: PLC0415 + + monkeypatch.setattr(model_registry, "build", lambda *a, **kw: _RampForecastModel()) + + cfg = _config_with_query( + QueryConfig( + id="disk_used_pct", + promql="up", + thresholds=[ + ThresholdConfig( + name="warn", + value=60.0, + comparator="gt", + action_cost=ActionCostConfig(type="usd", value=50.0), + ), + ThresholdConfig( + name="critical", + value=70.0, + comparator="gt", + action_cost=ActionCostConfig(type="usd", value=50_000.0), + ), + # A threshold without action_cost must NOT show up in + # the action_cost family — only the opted-in subset is + # published. + ThresholdConfig(name="info", value=55.0, comparator="gt"), + ], + ) + ) + source = _FakeSource({"up": [_frame(20)]}) + runner = Runner(config=cfg, source=source, exporter=(exporter := Exporter())) + + await runner.run_group(cfg.groups[0]) + + body = exporter.render().decode() + cost_lines = [ + line for line in body.splitlines() if line.startswith("forecast_threshold_action_cost{") + ] + assert len(cost_lines) == 2 + joined = "\n".join(cost_lines) + assert 'id="disk_used_pct"' in joined + assert 'group="g"' in joined + assert 'name="warn"' in joined + assert 'name="critical"' in joined + assert 'type="usd"' in joined + # ``info`` has no action_cost — it must not appear in this family. + assert 'name="info"' not in joined + # Sample values land as the cost (no per-model fan-out). + assert " 50.0" in joined + assert " 50000.0" in joined + + +@pytest.mark.asyncio +async def test_runner_action_cost_drops_when_threshold_removed( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A reconfigured group that drops a threshold drops its action_cost row. + + The family is rebuilt from current config every run, so /metrics + reflects the live YAML rather than the historical union. + """ + from promforecast import models as model_registry # noqa: PLC0415 + + monkeypatch.setattr(model_registry, "build", lambda *a, **kw: _RampForecastModel()) + + cfg = _config_with_query( + QueryConfig( + id="m", + promql="up", + thresholds=[ + ThresholdConfig( + name="warn", + value=60.0, + action_cost=ActionCostConfig(type="usd", value=50.0), + ), + ], + ) + ) + source = _FakeSource({"up": [_frame(20)]}) + runner = Runner(config=cfg, source=source, exporter=(exporter := Exporter())) + + await runner.run_group(cfg.groups[0]) + initial = exporter.render().decode() + assert "forecast_threshold_action_cost" in initial + + # Re-run with the threshold's action_cost removed; the row must + # disappear from /metrics on the next refresh. + cfg.groups[0].queries[0].thresholds = [ + ThresholdConfig(name="warn", value=60.0), + ] + await runner.run_group(cfg.groups[0]) + after = exporter.render().decode() + assert "forecast_threshold_action_cost" not in after diff --git a/forecaster/tests/test_runner_inheritance_registry.py b/forecaster/tests/test_runner_inheritance_registry.py index 8bfff4d..4f9a336 100644 --- a/forecaster/tests/test_runner_inheritance_registry.py +++ b/forecaster/tests/test_runner_inheritance_registry.py @@ -71,6 +71,15 @@ def _registry_ctx_attrs() -> set[str]: "BandCoveragePoint rows are rebuilt from the persistent " "BandCoverageTracker on every tick, not inherited." ), + # Config-derived: rebuilt every run from the current group config + # (via ``_build_threshold_action_cost_points``). Inheriting the prior + # snapshot would mean a threshold removed from YAML keeps emitting + # its cost row forever — the opposite of "tied to the threshold's + # lifecycle". + "threshold_action_cost_points": ( + "ThresholdActionCostPoint rows are derived from live config " + "on every run; inheritance would defeat the lifecycle semantics." + ), } diff --git a/forecaster/tests/test_runner_integration.py b/forecaster/tests/test_runner_integration.py index bfe67d4..59cb5b3 100644 --- a/forecaster/tests/test_runner_integration.py +++ b/forecaster/tests/test_runner_integration.py @@ -566,3 +566,348 @@ async def test_run_group_global_cap_truncates_across_queries() -> None: assert 'forecast_query_series_count{group="g",query="a"} 1.0' in body assert 'forecast_series_count{group="g"} 1.0' in body assert 'reason="total_cap"' in body + + +@pytest.mark.asyncio +async def test_run_group_records_per_series_staleness( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A successful fit must populate ``forecast_data_staleness_seconds``. + + Guards the runner wiring: ``Exporter.record_series_fit_success`` is + called from ``_run_one_series`` after a successful per-series fit, + once per emitted model, with the source labels + id + group + model. + Without this hook the operational ``ForecastSeriesStale`` alert + would never fire even on a healthy install. + """ + from promforecast import models as model_registry # noqa: PLC0415 + + cfg = Config( + datasource=DatasourceConfig(url="http://x"), + server=ServerConfig(), + safety=SafetyConfig(), + defaults=DefaultsConfig( + min_points=10, + models=["AutoARIMA"], + confidence_levels=[80], + accuracy=AccuracyConfig( + evaluate=False, + auto_select=False, + horizons=[timedelta(minutes=10)], + ), + emission=EmissionConfig(deviation=False, quality=False, contribution=False), + ), + groups=[GroupConfig(name="g", queries=[QueryConfig(id="m", promql="up")])], + ) + monkeypatch.setattr( + model_registry, + "build", + lambda name, season_length, params=None: _FakeModel(name, value=1.0), + ) + source = _FakeSource({"up": [_ramp_frame(60)]}) + exporter = Exporter() + runner = Runner(config=cfg, source=source, exporter=exporter) + + await runner.run_group(cfg.groups[0]) + + body = exporter.render().decode() + assert "# TYPE forecast_data_staleness_seconds gauge" in body + assert 'id="m"' in body + assert 'model="AutoARIMA"' in body + assert 'group="g"' in body + # The recorded timestamp is "just now" so elapsed must be small. + line = next(ln for ln in body.splitlines() if ln.startswith("forecast_data_staleness_seconds{")) + elapsed = float(line.rsplit(" ", 1)[-1]) + assert 0.0 <= elapsed < 30.0 + + +@pytest.mark.asyncio +async def test_apply_config_prunes_staleness_for_removed_groups( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A config reload that drops a group must clear its staleness rows. + + Without retention the dropped group's series would linger on + /metrics with a forever-growing staleness value because no fit + will ever reset their clock. + """ + from promforecast import models as model_registry # noqa: PLC0415 + + monkeypatch.setattr( + model_registry, + "build", + lambda name, season_length, params=None: _FakeModel(name, value=1.0), + ) + cfg = Config( + datasource=DatasourceConfig(url="http://x"), + server=ServerConfig(), + safety=SafetyConfig(), + defaults=DefaultsConfig( + min_points=10, + models=["AutoARIMA"], + confidence_levels=[80], + accuracy=AccuracyConfig( + evaluate=False, auto_select=False, horizons=[timedelta(minutes=10)] + ), + emission=EmissionConfig(deviation=False, quality=False, contribution=False), + ), + groups=[ + GroupConfig(name="keep", queries=[QueryConfig(id="m", promql="up")]), + GroupConfig(name="drop", queries=[QueryConfig(id="m", promql="up")]), + ], + ) + source = _FakeSource({"up": [_ramp_frame(60)]}) + exporter = Exporter() + runner = Runner(config=cfg, source=source, exporter=exporter) + await runner.run_group(cfg.groups[0]) + await runner.run_group(cfg.groups[1]) + + body = exporter.render().decode() + assert 'group="keep"' in body + assert 'group="drop"' in body + + new_cfg = Config( + datasource=cfg.datasource, + server=cfg.server, + safety=cfg.safety, + defaults=cfg.defaults, + groups=[cfg.groups[0]], # 'drop' removed + ) + await runner.apply_config(new_cfg) + body = exporter.render().decode() + assert 'group="keep"' in body + assert 'group="drop"' not in body + + +@pytest.mark.asyncio +async def test_apply_config_prunes_stale_staleness_after_model_removal( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A model dropped from defaults.models is cleaned up by the age sweep. + + The per-group / per-query retention helpers can't reach this case + because the (group, query) pair still exists — only the model + label changed. Without the time-based sweep the dropped model's + rows would refresh-never and ``ForecastSeriesStale`` would fire + on them forever. We simulate "model dropped" by recording a fit + under model X, ageing it past the threshold, then reloading the + config; apply_config must call ``prune_stale_staleness`` with a + threshold low enough to drop the aged row. + """ + from promforecast import models as model_registry # noqa: PLC0415 + + monkeypatch.setattr( + model_registry, + "build", + lambda name, season_length, params=None: _FakeModel(name, value=1.0), + ) + cfg = Config( + datasource=DatasourceConfig(url="http://x"), + server=ServerConfig(), + safety=SafetyConfig(), + defaults=DefaultsConfig( + min_points=10, + models=["AutoARIMA"], + confidence_levels=[80], + accuracy=AccuracyConfig( + evaluate=False, auto_select=False, horizons=[timedelta(minutes=10)] + ), + emission=EmissionConfig(deviation=False, quality=False, contribution=False), + ), + groups=[GroupConfig(name="g", queries=[QueryConfig(id="m", promql="up")])], + ) + source = _FakeSource({"up": [_ramp_frame(60)]}) + exporter = Exporter() + runner = Runner(config=cfg, source=source, exporter=exporter) + await runner.run_group(cfg.groups[0]) + + # Confirm the row is present. + assert any(key[2] == "AutoARIMA" for key in exporter._series_last_success), ( + "fit must have recorded the AutoARIMA row" + ) + + # Age every staleness entry past the day threshold. In production + # this happens naturally when a config change removes a model and + # subsequent reloads don't see the row refreshed. + for k, (ts, labels) in list(exporter._series_last_success.items()): + exporter._series_last_success[k] = (ts - 100_000.0, labels) + + # Reload identical config — the sweep is unconditional on + # apply_config, so even a no-op reload prunes aged rows. The + # actual call must use the production-shaped threshold; building + # a new Config object is enough to trigger the sweep. + new_cfg = Config( + datasource=cfg.datasource, + server=cfg.server, + safety=cfg.safety, + defaults=cfg.defaults, + groups=list(cfg.groups), + ) + await runner.apply_config(new_cfg) + assert not exporter._series_last_success, "aged rows must be pruned by apply_config sweep" + + +@pytest.mark.asyncio +async def test_run_group_emits_self_observability_and_stability_metrics( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """End-to-end binding test for the self-observability + stability metrics. + + Confirms the runner-level wiring lands the new families on /metrics + after a single group run with two configured models in explicit + mode + accuracy.evaluate=True (so per-horizon quality, revision, + and disagreement all have backtest results to bind against). Also + asserts the CPU/memory gauges are populated and the fit-success + ratio gauge is 1.0 (every fit succeeded). + """ + from promforecast import models as model_registry # noqa: PLC0415 + + cfg = Config( + datasource=DatasourceConfig(url="http://x"), + server=ServerConfig(), + safety=SafetyConfig(), + defaults=DefaultsConfig( + min_points=10, + models=["AutoARIMA", "SeasonalNaive"], + confidence_levels=[80], + accuracy=AccuracyConfig( + evaluate=True, + auto_select=False, + horizons=[timedelta(minutes=10), timedelta(minutes=30)], + ), + emission=EmissionConfig(deviation=True, quality=True, contribution=False), + ), + groups=[GroupConfig(name="g", queries=[QueryConfig(id="m", promql="up")])], + ) + + # Use two models that produce different yhat values so the + # disagreement gauge has a real spread to measure. + def _fake_build(name: str, season_length: int, params: dict | None = None) -> _FakeModel: + return _FakeModel(name, value=1.0 if name == "AutoARIMA" else 5.0) + + monkeypatch.setattr(model_registry, "build", _fake_build) + source = _FakeSource({"up": [_ramp_frame(60)]}) + exporter = Exporter() + runner = Runner(config=cfg, source=source, exporter=exporter) + + # First run seeds the revision cache; revision points are NaN + + # dropped at render. Per-horizon quality + disagreement land + # immediately because they don't depend on a prior fit. + await runner.run_group(cfg.groups[0]) + + body = exporter.render().decode() + # Fit success ratio: both models fit cleanly → ratio 1.0 in both windows. + # Each ratio row is asserted as a plain substring (the trailing `}` is + # not part of an f-string, so we build the expected line by hand). + arima_row = 'forecast_model_fit_success_ratio{group="g",model="AutoARIMA",window="1h"} 1.0' + snaive_row = 'forecast_model_fit_success_ratio{group="g",model="SeasonalNaive",window="1h"} 1.0' + assert arima_row in body + assert snaive_row in body + # Group-level CPU + memory present. + assert 'forecast_group_cpu_seconds_total{group="g"}' in body + assert 'forecast_group_memory_bytes{group="g"}' in body + # Per-horizon quality emission: aggregated + per-horizon rows. + quality_lines = [ + line for line in body.splitlines() if line.startswith("forecast_quality_score{") + ] + horizons_seen = { + line.split('horizon="', 1)[1].split('"', 1)[0] + for line in quality_lines + if 'horizon="' in line + } + # Two emitted models * (1 aggregated + 2 per-horizon) = at least 6 rows; + # the horizon label set covers the empty (aggregated) and both + # configured horizons. + assert "" in horizons_seen + assert "10m" in horizons_seen + assert "30m" in horizons_seen + # Disagreement: two emitted models with distinct yhat → finite CoV. + assert "forecast_model_disagreement" in body + disagreement_lines = [ + line for line in body.splitlines() if line.startswith("forecast_model_disagreement{") + ] + assert disagreement_lines, "disagreement rows must be emitted in explicit mode" + # No revision rows on the first fit (cache was empty → NaN → dropped). + assert "forecast_revision_magnitude{" not in body + + # Second run populates the revision cache from the first run, so + # revision magnitudes are now finite (in this test both fits + # produced the same yhat, so the magnitude is exactly 0.0). + await runner.run_group(cfg.groups[0]) + body2 = exporter.render().decode() + revision_lines = [ + line for line in body2.splitlines() if line.startswith("forecast_revision_magnitude{") + ] + assert revision_lines, "second fit must emit revision rows (cache now populated)" + + # CPU counter is monotonic + increasing across runs. + cpu_rows = [ + line for line in body2.splitlines() if line.startswith("forecast_group_cpu_seconds_total{") + ] + cpu_value = float(cpu_rows[0].rsplit(" ", 1)[1]) + assert cpu_value > 0 + + +@pytest.mark.asyncio +async def test_apply_config_prunes_fit_outcome_when_model_dropped( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Removing a model from defaults.models must clear its success-ratio rows. + + Regression guard for the self-observability cleanup: without + the model-removal sweep, dropping a model from the config would + leave its frozen-in-time ratio on /metrics until process restart, + misleading the ``ForecastModelFitFailing`` alert. + """ + from promforecast import models as model_registry # noqa: PLC0415 + + monkeypatch.setattr( + model_registry, + "build", + lambda name, season_length, params=None: _FakeModel(name, value=1.0), + ) + cfg = Config( + datasource=DatasourceConfig(url="http://x"), + server=ServerConfig(), + safety=SafetyConfig(), + defaults=DefaultsConfig( + min_points=10, + models=["AutoARIMA", "SeasonalNaive"], + confidence_levels=[80], + accuracy=AccuracyConfig( + evaluate=False, auto_select=False, horizons=[timedelta(minutes=10)] + ), + ), + groups=[GroupConfig(name="g", queries=[QueryConfig(id="m", promql="up")])], + ) + source = _FakeSource({"up": [_ramp_frame(60)]}) + exporter = Exporter() + runner = Runner(config=cfg, source=source, exporter=exporter) + await runner.run_group(cfg.groups[0]) + + body = exporter.render().decode() + assert "AutoARIMA" in body + assert "SeasonalNaive" in body + + # Reload with SeasonalNaive removed. The sweep must drop its rows. + new_cfg = Config( + datasource=cfg.datasource, + server=cfg.server, + safety=cfg.safety, + defaults=DefaultsConfig( + min_points=10, + models=["AutoARIMA"], + confidence_levels=[80], + accuracy=cfg.defaults.accuracy, + ), + groups=list(cfg.groups), + ) + await runner.apply_config(new_cfg) + + body2 = exporter.render().decode() + fit_ratio_lines = [ + line for line in body2.splitlines() if line.startswith("forecast_model_fit_success_ratio{") + ] + # SeasonalNaive must be gone; AutoARIMA must still be present. + assert any('model="AutoARIMA"' in line for line in fit_ratio_lines) + assert not any('model="SeasonalNaive"' in line for line in fit_ratio_lines) diff --git a/forecaster/tests/test_self_observability.py b/forecaster/tests/test_self_observability.py new file mode 100644 index 0000000..c6fbff9 --- /dev/null +++ b/forecaster/tests/test_self_observability.py @@ -0,0 +1,141 @@ +"""Unit tests for the self-observability trackers and rendering.""" + +from __future__ import annotations + +import math + +import pytest + +from promforecast.exporter import Exporter +from promforecast.self_observability import ( + FitOutcomeTracker, + GroupResourceTracker, + RegressorStabilityTracker, + compute_stability_score, +) + + +def test_fit_outcome_tracker_records_successes_and_failures() -> None: + tracker = FitOutcomeTracker(windows_seconds=(60, 3600)) + now = 1_000_000.0 + for _ in range(8): + tracker.record(group="g", model="AutoARIMA", success=True, now=now) + for _ in range(2): + tracker.record(group="g", model="AutoARIMA", success=False, now=now) + points = tracker.render(now=now) + by_window = {p.window_label: p.ratio for p in points} + assert by_window["1m"] == pytest.approx(0.8) + assert by_window["1h"] == pytest.approx(0.8) + + +def test_fit_outcome_tracker_prunes_outside_window() -> None: + tracker = FitOutcomeTracker(windows_seconds=(60,)) + now = 1_000_000.0 + # Old samples + tracker.record(group="g", model="m", success=False, now=now - 1000) + tracker.record(group="g", model="m", success=False, now=now - 800) + # Recent samples + tracker.record(group="g", model="m", success=True, now=now - 30) + points = tracker.render(now=now) + assert len(points) == 1 + assert points[0].ratio == pytest.approx(1.0) + + +def test_fit_outcome_tracker_empty_window_emits_no_point() -> None: + tracker = FitOutcomeTracker(windows_seconds=(60,)) + now = 1_000_000.0 + tracker.record(group="g", model="m", success=True, now=now - 1000) + # The sample is outside the 60s window; render should be empty. + assert tracker.render(now=now) == [] + + +def test_fit_outcome_tracker_retain_groups_drops_others() -> None: + tracker = FitOutcomeTracker() + tracker.record(group="keep", model="m", success=True) + tracker.record(group="drop", model="m", success=True) + tracker.retain_groups({"keep"}) + points = tracker.render() + assert {p.group for p in points} == {"keep"} + + +def test_compute_stability_score_no_drift() -> None: + # Identical lookback and recent -> 1.0 + values = [10.0] * 50 + score = compute_stability_score(values) + assert score == pytest.approx(1.0) + + +def test_compute_stability_score_moderate_drift() -> None: + # Baseline mean 100, recent mean 110 -> 10% shift -> score 0.9 + values = [100.0] * 40 + [110.0] * 10 + score = compute_stability_score(values, recent_fraction=0.2) + assert score == pytest.approx(0.9, abs=0.01) + + +def test_compute_stability_score_saturates_at_zero() -> None: + # Baseline 100, recent 250 -> 150% shift -> saturates at 0 + values = [100.0] * 40 + [250.0] * 10 + score = compute_stability_score(values, recent_fraction=0.2, saturation=1.0) + assert score == 0.0 + + +def test_compute_stability_score_nan_on_thin_input() -> None: + assert math.isnan(compute_stability_score([1.0, 2.0])) + + +def test_compute_stability_score_constant_zero_is_stable() -> None: + # A constant zero series is by definition stable — recent and + # baseline both have mean 0, and the implementation's denom-floor + # keeps the relative metric well-defined. + assert compute_stability_score([0.0] * 50) == pytest.approx(1.0) + + +def test_regressor_stability_tracker_records_and_renders() -> None: + tracker = RegressorStabilityTracker() + tracker.record( + group="g", + query="q", + regressor_id="deploys", + values=[100.0] * 50, + ) + points = tracker.render() + assert len(points) == 1 + assert points[0].group == "g" + assert points[0].score == pytest.approx(1.0) + + +def test_group_resource_tracker_accumulates_cpu() -> None: + tracker = GroupResourceTracker() + tracker.add_cpu("g", 1.5) + tracker.add_cpu("g", 2.5) + tracker.set_memory("g", 1024.0) + points = tracker.render() + assert len(points) == 1 + assert points[0].cpu_seconds_total == pytest.approx(4.0) + assert points[0].memory_bytes == pytest.approx(1024.0) + + +def test_exporter_records_self_observability_state() -> None: + exporter = Exporter() + exporter.record_fit_outcome(group="g", model="AutoARIMA", success=True) + exporter.record_fit_outcome(group="g", model="AutoARIMA", success=False) + exporter.record_regressor_stability( + group="g", + query="cpu", + regressor_id="deploys", + values=[1.0] * 50, + ) + exporter.add_group_cpu_seconds("g", 3.0) + exporter.set_group_memory_bytes("g", 4096.0) + body = exporter.render().decode() + assert "forecast_model_fit_success_ratio" in body + assert "forecast_regressor_stability_score" in body + assert "forecast_group_memory_bytes" in body + assert "forecast_group_cpu_seconds_total" in body + # 1/2 success -> ratio 0.5 + assert 'forecast_model_fit_success_ratio{group="g",model="AutoARIMA"' in body + # Cleanup hook drops the group's rows. + exporter.retain_self_observability_groups(set()) + body2 = exporter.render().decode() + assert "forecast_model_fit_success_ratio" not in body2 + assert "forecast_group_memory_bytes" not in body2 diff --git a/forecaster/tests/test_tuner.py b/forecaster/tests/test_tuner.py new file mode 100644 index 0000000..e0da4dd --- /dev/null +++ b/forecaster/tests/test_tuner.py @@ -0,0 +1,283 @@ +"""Unit tests for the hyperparameter tuner. + +Covers the pure helpers (grid generation, improvement-percentage +math, run-outcome classification, recommendation push payload) and +the CLI's early-exit paths (missing config, missing sink). The full +async grid walk is exercised against the docker-compose dev stack +manually (``make dev-up`` + ``promforecast tune --config docker/dev-config.yaml``); +no in-process integration test ships because the grid is genuinely +non-deterministic across model versions and would be a flaky-on-CI +regression target. +""" + +from __future__ import annotations + +import asyncio +import io +import textwrap +from pathlib import Path + +import pytest + +from promforecast import tuner as tuner_module +from promforecast.tuner import ( + RecommendedParam, + _improvement_pct, + _push_results, + _run_outcome, + _season_length_grid, + run_tuner, +) + +# ---------- pure helpers --------------------------------------------------- + + +def test_season_length_grid_returns_half_and_double() -> None: + grid = _season_length_grid(288) + assert grid == [144, 576] + + +def test_season_length_grid_caps_huge_doubles() -> None: + # ``current = 6000`` -> ``double = 12000`` is above the 10000 cap; + # only the halved candidate survives. + grid = _season_length_grid(6000) + assert grid == [3000] + + +def test_season_length_grid_floor_below_two() -> None: + # A configured season_length of 2 cannot be halved further; the + # grid still emits the doubled candidate. + grid = _season_length_grid(2) + assert grid == [4] + + +def test_improvement_pct_positive_when_candidate_better() -> None: + assert _improvement_pct(baseline=20.0, candidate=10.0) == pytest.approx(50.0) + + +def test_improvement_pct_negative_when_candidate_worse() -> None: + assert _improvement_pct(baseline=10.0, candidate=20.0) == pytest.approx(-100.0) + + +def test_improvement_pct_zero_when_baseline_zero() -> None: + # Avoid divide-by-zero when the configured baseline produces an + # impossible-to-beat 0 MAPE on a degenerate series. + assert _improvement_pct(baseline=0.0, candidate=5.0) == 0.0 + + +def test_run_outcome_ok_when_no_failures() -> None: + assert _run_outcome(failed_queries=0, group_count=3) == "ok" + + +def test_run_outcome_partial_with_some_failures() -> None: + assert _run_outcome(failed_queries=2, group_count=3) == "partial" + + +def test_run_outcome_error_when_no_groups() -> None: + assert _run_outcome(failed_queries=1, group_count=0) == "error" + + +# ---------- CLI entry-point early exits ------------------------------------ + + +_CONFIG_YAML = textwrap.dedent( + """\ + apiVersion: promforecast.io/v1 + datasource: + url: http://localhost:8428 + server: + listen: ":9091" + refresh_interval: 1h + safety: + max_series_per_query: 100 + defaults: + lookback: 14d + step: 5m + horizon: 24h + models: [AutoARIMA, SeasonalNaive] + season_length: 288 + min_points: 200 + confidence_levels: [80] + sink: + remote_write: + enabled: false + groups: + - name: node_capacity + queries: + - id: node_cpu_busy_pct + promql: 'avg by (instance)(rate(node_cpu_seconds_total{mode!="idle"}[5m])) * 100' + """ +) + + +def _write_config(tmp_path: Path) -> Path: + path = tmp_path / "config.yaml" + path.write_text(_CONFIG_YAML) + return path + + +def test_run_tuner_missing_config_returns_1(tmp_path: Path) -> None: + out = io.StringIO() + rc = run_tuner( + config_path=tmp_path / "does-not-exist.yaml", + group_selector=None, + sink_url_override=None, + out=out, + ) + assert rc == 1 + assert "config not found" in out.getvalue() + + +def test_run_tuner_missing_sink_returns_1(tmp_path: Path) -> None: + out = io.StringIO() + rc = run_tuner( + config_path=_write_config(tmp_path), + group_selector=None, + sink_url_override=None, # config also has no sink configured + out=out, + ) + assert rc == 1 + assert "no sink configured" in out.getvalue() + + +def test_run_tuner_unknown_group_returns_1(tmp_path: Path) -> None: + out = io.StringIO() + rc = run_tuner( + config_path=_write_config(tmp_path), + group_selector=["does-not-exist"], + sink_url_override="http://victoriametrics:8428/api/v1/import/prometheus", + out=out, + ) + assert rc == 1 + assert "no groups selected" in out.getvalue() + + +def test_recommended_param_is_hashable_dataclass() -> None: + """``frozen=True`` is intentional — recommendations are passed through a + chain of functions and should be safe to use as dict keys / set members. + """ + rec = RecommendedParam( + group="g", + query="q", + model="AutoARIMA", + name="season_length", + current_value="288", + recommended_value="576", + expected_mape_improvement_pct=12.3, + ) + assert hash(rec) # should not raise + assert rec.expected_mape_improvement_pct == 12.3 + + +def test_module_help_is_documented() -> None: + """The module docstring should mention the metric name a Grafana + panel author needs to grep for. Regression guard against silent + metric renames.""" + assert "forecast_recommended_param" in tuner_module.__doc__ or "" + + +# ---------- push payload shape --------------------------------------------- + + +class _RecordingSink: + """A drop-in ``ForecastSink`` that captures the written points. + + Lets us assert the metric names, labels, and values the tuner + actually pushes without standing up a real TSDB. Matches the + protocol surface (``write`` + ``aclose``) so it satisfies the + ``ForecastSink`` Protocol. + """ + + name = "recording" + + def __init__(self) -> None: + self.written: list = [] + self.closed = False + + async def write(self, points, *, on_retry=None) -> None: + self.written.extend(list(points)) + + async def aclose(self) -> None: + self.closed = True + + +def test_push_results_emits_gauge_timestamps_not_counter() -> None: + """Regression guard for the stateless-pusher semantics. + + Before the fix the tuner pushed ``forecast_tuner_runs_total`` as + a constant ``1.0`` per run — a no-op for ``rate()`` against the + flat series the TSDB ended up with. The fix swaps to + ``forecast_tuner_last_run_timestamp_seconds{group, outcome}`` as + a gauge of unix-seconds; this test pins both the metric name and + the gauge-of-timestamp semantics so a future revert is caught at + CI time. + """ + sink = _RecordingSink() + recs = [ + RecommendedParam( + group="g1", + query="q1", + model="AutoARIMA", + name="season_length", + current_value="288", + recommended_value="576", + expected_mape_improvement_pct=12.5, + ), + ] + asyncio.run( + _push_results( + sink=sink, + recommendations=recs, + outcome="ok", + elapsed_seconds=42.0, + groups=["g1", "g2"], + ) + ) + metrics = {p.metric for p in sink.written} + assert "forecast_recommended_param" in metrics + assert "forecast_tuner_last_run_timestamp_seconds" in metrics + assert "forecast_tuner_last_run_duration_seconds" in metrics + # The pre-fix counter must not reappear under either name shape. + assert "forecast_tuner_runs_total" not in metrics + assert "forecast_tuner_run_duration_seconds" not in metrics + + # Timestamp gauge value should be approximately now (unix seconds), + # not the constant 1.0 the pre-fix counter pushed. + timestamp_points = [ + p for p in sink.written if p.metric == "forecast_tuner_last_run_timestamp_seconds" + ] + assert len(timestamp_points) == 2 # one per group + for p in timestamp_points: + assert p.value > 1_700_000_000 # roughly post-2023 + assert p.labels["outcome"] == "ok" + + # The recommendation row carries the improvement-pct as the sample value. + rec_points = [p for p in sink.written if p.metric == "forecast_recommended_param"] + assert len(rec_points) == 1 + assert rec_points[0].value == 12.5 + assert rec_points[0].labels["current_value"] == "288" + assert rec_points[0].labels["recommended_value"] == "576" + + +def test_push_results_empty_recommendations_still_emits_telemetry() -> None: + """A no-recommendations run must still push the per-group + telemetry so the staleness alert + ``time() - max by (group) (forecast_tuner_last_run_timestamp_seconds) > 8d`` + fires on a missing run rather than on a successful "no + recommendations" run. + """ + sink = _RecordingSink() + asyncio.run( + _push_results( + sink=sink, + recommendations=[], + outcome="ok", + elapsed_seconds=10.0, + groups=["g1"], + ) + ) + metrics = {p.metric for p in sink.written} + assert metrics == { + "forecast_tuner_last_run_timestamp_seconds", + "forecast_tuner_last_run_duration_seconds", + } diff --git a/forecaster/tests/test_validate_cli.py b/forecaster/tests/test_validate_cli.py index 1c5ab41..54c63db 100644 --- a/forecaster/tests/test_validate_cli.py +++ b/forecaster/tests/test_validate_cli.py @@ -129,14 +129,100 @@ def test_cardinality_estimate_matches_doc_formula() -> None: assert est.accuracy_lines == 1 * 100 * 2 * 2 * 2 # 1 * 100 * 2 emitted * 2 levels * 2 (ratio + outside) = 800. assert est.deviation_lines == 1 * 100 * 2 * 2 * 2 - # 1 * 100 * 2 emitted = 200. - assert est.quality_lines == 1 * 100 * 2 + # 1 * 100 * 2 emitted * (1 aggregated + 2 horizons) = 600. + # Per-horizon quality segmentation; existing alerts gating on + # ``horizon=""`` keep matching the aggregated row. + assert est.quality_lines == 1 * 100 * 2 * (1 + 2) + # Revision: 1 * 100 * 2 emitted * 2 horizons = 400 (steady-state; + # first-fit NaN rows are dropped at render). + assert est.revision_lines == 1 * 100 * 2 * 2 + # Disagreement: 1 * 100 * 2 horizons = 200 (explicit mode, 2+ models). + assert est.disagreement_lines == 1 * 100 * 2 # Without thresholds / growth-rate emission, the capacity families # contribute zero to the estimate. assert est.threshold_eta_lines == 0 assert est.growth_rate_lines == 0 +def test_revision_and_disagreement_lines_dont_gate_on_evaluate() -> None: + """Revision + disagreement emit on configured horizons, regardless of evaluate. + + Reading the central forecast at each horizon needs no backtest; + the runtime emits these metrics even when ``accuracy.evaluate`` + is off. The estimator must match the runtime, or the + ``--strict-cardinality`` gate falsely lets oversized configs + through when the operator has explicitly disabled backtesting. + """ + from datetime import timedelta # noqa: PLC0415 + + from promforecast.config import ( # noqa: PLC0415 + AccuracyConfig, + Config, + DatasourceConfig, + DefaultsConfig, + GroupConfig, + QueryConfig, + SafetyConfig, + ServerConfig, + ) + + cfg = Config( + datasource=DatasourceConfig(url="http://x"), + server=ServerConfig(), + safety=SafetyConfig(max_series_per_query=10, max_total_series=0), + defaults=DefaultsConfig( + models=["AutoARIMA", "SeasonalNaive"], + confidence_levels=[80], + accuracy=AccuracyConfig( + evaluate=False, + horizons=[timedelta(hours=1), timedelta(hours=24)], + auto_select=False, + ), + ), + groups=[GroupConfig(name="g", queries=[QueryConfig(id="m", promql="up")])], + ) + est = _estimate_cardinality(cfg) + # 1 query * 10 series * 2 emitted models * 2 horizons = 40. + assert est.revision_lines == 40 + # Explicit mode + 2 models: 1 * 10 * 2 horizons = 20. + assert est.disagreement_lines == 20 + + +def test_disagreement_lines_suppressed_under_auto_select() -> None: + """auto_select (including ensemble) suppresses model_disagreement entirely.""" + from datetime import timedelta # noqa: PLC0415 + + from promforecast.config import ( # noqa: PLC0415 + AccuracyConfig, + Config, + DatasourceConfig, + DefaultsConfig, + GroupConfig, + QueryConfig, + SafetyConfig, + ServerConfig, + ) + + for selector in (True, "ensemble"): + cfg = Config( + datasource=DatasourceConfig(url="http://x"), + server=ServerConfig(), + safety=SafetyConfig(max_series_per_query=10, max_total_series=0), + defaults=DefaultsConfig( + models=["AutoARIMA", "SeasonalNaive"], + confidence_levels=[80], + accuracy=AccuracyConfig( + evaluate=True, + horizons=[timedelta(hours=1)], + auto_select=selector, + ), + ), + groups=[GroupConfig(name="g", queries=[QueryConfig(id="m", promql="up")])], + ) + est = _estimate_cardinality(cfg) + assert est.disagreement_lines == 0, f"auto_select={selector!r} must suppress disagreement" + + def test_cardinality_estimate_counts_threshold_and_growth_rate_lines() -> None: """Capacity ETA + growth-rate families must show up in the cardinality estimate.""" from datetime import timedelta # noqa: PLC0415 @@ -199,6 +285,163 @@ def test_cardinality_estimate_counts_threshold_and_growth_rate_lines() -> None: assert est.growth_rate_lines == 100 * 1 * 2 +def test_cardinality_estimate_counts_service_health_only_when_enabled() -> None: + """``governance.health_score`` configured => bounded estimate; absent => zero.""" + from datetime import timedelta # noqa: PLC0415 + + from promforecast.config import ( # noqa: PLC0415 + AccuracyConfig, + Config, + DatasourceConfig, + DefaultsConfig, + EmissionConfig, + GovernanceConfig, + GroupConfig, + HealthScoreConfig, + QueryConfig, + SafetyConfig, + ServerConfig, + ) + + def _build(governance: GovernanceConfig) -> Config: + return Config( + datasource=DatasourceConfig(url="http://x"), + server=ServerConfig(), + safety=SafetyConfig(max_series_per_query=100, max_total_series=0), + defaults=DefaultsConfig( + models=["AutoARIMA"], + confidence_levels=[80], + accuracy=AccuracyConfig( + evaluate=True, + auto_select=False, + horizons=[timedelta(hours=1)], + ), + emission=EmissionConfig(quality=True), + ), + groups=[GroupConfig(name="g", queries=[QueryConfig(id="m", promql="up")])], + governance=governance, + ) + + off = _estimate_cardinality(_build(GovernanceConfig())) + on = _estimate_cardinality( + _build(GovernanceConfig(health_score=HealthScoreConfig(aggregate_by=["service"]))) + ) + assert off.service_health_lines == 0 + # Upper bound matches the quality_lines total when the feature is on + # (one bucket per quality_point in the worst case). + assert on.service_health_lines == on.quality_lines + assert on.service_health_lines > 0 + + +def test_cardinality_estimate_counts_resource_sizing_rows() -> None: + """Right-sizing charges 1 row per workload, plus 2 more when ``current_promql`` is set.""" + from promforecast.config import ( # noqa: PLC0415 + Config, + DatasourceConfig, + DefaultsConfig, + GroupConfig, + QueryConfig, + RecommendationsConfig, + ResourceSizingConfig, + ResourceSizingWorkloadConfig, + SafetyConfig, + ServerConfig, + ) + + cfg = Config( + datasource=DatasourceConfig(url="http://x"), + server=ServerConfig(), + safety=SafetyConfig(max_series_per_query=10, max_total_series=0), + defaults=DefaultsConfig(models=["AutoARIMA"]), + groups=[GroupConfig(name="g", queries=[QueryConfig(id="svc", promql="up")])], + recommendations=RecommendationsConfig( + resource_sizing=ResourceSizingConfig( + enabled=True, + workloads=[ + # No current_promql -> 1 line (recommended only). + ResourceSizingWorkloadConfig(workload="api", kind="cpu", forecast_query="svc"), + # current_promql wired -> 3 lines (recommended + current + savings_pct). + ResourceSizingWorkloadConfig( + workload="api", + kind="memory", + forecast_query="svc", + current_promql="kube_pod_container_resource_requests", + ), + ], + ) + ), + ) + est = _estimate_cardinality(cfg) + # 1 (cpu, recommended only) + 3 (memory, recommended + current + savings_pct) = 4. + assert est.resource_sizing_lines == 4 + + +def test_cardinality_estimate_counts_action_cost_only_for_opted_in() -> None: + """``forecast_threshold_action_cost`` charges one line per opted-in threshold. + + Independent of model / series / level fan-out — the metric is + config-derived. Thresholds without ``action_cost`` configured must + not contribute to the cardinality budget. + """ + from datetime import timedelta # noqa: PLC0415 + + from promforecast.config import ( # noqa: PLC0415 + AccuracyConfig, + ActionCostConfig, + Config, + DatasourceConfig, + DefaultsConfig, + GroupConfig, + QueryConfig, + SafetyConfig, + ServerConfig, + ThresholdConfig, + ) + + cfg = Config( + datasource=DatasourceConfig(url="http://x"), + server=ServerConfig(), + safety=SafetyConfig(max_series_per_query=100, max_total_series=0), + defaults=DefaultsConfig( + models=["AutoARIMA"], + confidence_levels=[80, 95], + accuracy=AccuracyConfig( + evaluate=False, + auto_select=False, + horizons=[timedelta(hours=1)], + ), + ), + groups=[ + GroupConfig( + name="g", + queries=[ + QueryConfig( + id="m", + promql="up", + thresholds=[ + ThresholdConfig( + name="warn", + value=80, + action_cost=ActionCostConfig(type="usd", value=50.0), + ), + ThresholdConfig( + name="crit", + value=95, + action_cost=ActionCostConfig(type="usd", value=50_000.0), + ), + # No action_cost — must not contribute. + ThresholdConfig(name="info", value=50), + ], + ), + ], + ) + ], + ) + est = _estimate_cardinality(cfg) + # 2 thresholds opted in x 1 row each = 2 lines (no series fan-out). + assert est.action_cost_lines == 2 + + def test_cardinality_estimate_counts_conditional_calibration_when_conformal_on() -> None: """The dedicated conditional-band conformal gauge must show up in the estimate. diff --git a/forecaster/tests/test_warmup.py b/forecaster/tests/test_warmup.py new file mode 100644 index 0000000..8cea84c --- /dev/null +++ b/forecaster/tests/test_warmup.py @@ -0,0 +1,569 @@ +"""Tests for the warm-up status endpoint + CLI machinery. + +The warmup probe issues one cheap PromQL ``query_range`` per +configured query. These tests use a fake source so they cover the +status-classification logic (ready / warming / error, blocked_by +enum, ETA estimate) without hitting a real TSDB. +""" + +from __future__ import annotations + +from datetime import UTC, datetime, timedelta + +import pytest +from fastapi.testclient import TestClient + +from promforecast.config import ( + Config, + DatasourceConfig, + DefaultsConfig, + GroupConfig, + QueryConfig, + RegressorConfig, + SafetyConfig, + ServerConfig, +) +from promforecast.exporter import Exporter +from promforecast.main import _build_app +from promforecast.reload import ConfigReloader +from promforecast.runner import Runner +from promforecast.source import PromSource, SeriesFrame +from promforecast.warmup import ( + GroupWarmupStatus, + QueryWarmupStatus, + WarmupReport, + compute_warmup, + render_table, +) + + +class _FakeSource(PromSource): + """Returns canned series for each PromQL string; never opens a socket.""" + + def __init__(self, by_query: dict[str, list[SeriesFrame]]) -> None: + self._by_query = by_query + self.calls: list[str] = [] + + async def query_range( + self, + promql: str, + start: datetime, + end: datetime, + step_seconds: int, + ) -> list[SeriesFrame]: + self.calls.append(promql) + return self._by_query.get(promql, []) + + async def aclose(self) -> None: + return None + + +def _config( + queries: list[QueryConfig], + *, + min_points: int = 100, + lookback_minutes: int = 60, +) -> Config: + return Config( + datasource=DatasourceConfig(url="http://x"), + server=ServerConfig(), + safety=SafetyConfig(), + defaults=DefaultsConfig( + min_points=min_points, + models=["SeasonalNaive"], + lookback=timedelta(minutes=lookback_minutes), + step=timedelta(minutes=1), + ), + groups=[GroupConfig(name="g", queries=queries)], + ) + + +def _frame(n_points: int) -> SeriesFrame: + return SeriesFrame( + labels={"instance": "host-a"}, + timestamps=[datetime.now(tz=UTC)] * n_points, + values=[1.0] * n_points, + ) + + +@pytest.mark.asyncio +async def test_warmup_reports_ready_when_min_points_satisfied() -> None: + cfg = _config([QueryConfig(id="m", promql="up")], min_points=10) + source = _FakeSource({"up": [_frame(20)]}) + report = await compute_warmup(config=cfg, source=source) + assert len(report.groups) == 1 + group = report.groups[0] + assert group.name == "g" + assert len(group.queries) == 1 + status = group.queries[0] + assert status.status == "ready" + assert status.points_available == 20 + assert status.points_needed == 10 + assert status.blocked_by == "none" + assert status.eta_seconds == 0.0 + + +@pytest.mark.asyncio +async def test_warmup_reports_warming_with_eta() -> None: + """A series under min_points reports ETA = missing-points * step.""" + cfg = _config([QueryConfig(id="m", promql="up")], min_points=100) + source = _FakeSource({"up": [_frame(40)]}) + report = await compute_warmup(config=cfg, source=source) + status = report.groups[0].queries[0] + assert status.status == "warming" + assert status.points_available == 40 + assert status.points_needed == 100 + assert status.blocked_by == "min_points" + # Default step is 1m = 60s; missing 60 points * 60s = 3600s. + assert status.eta_seconds == 60 * 60 + + +@pytest.mark.asyncio +async def test_warmup_reports_lookback_when_no_samples() -> None: + """A query that returned zero samples blocks on lookback, not min_points.""" + cfg = _config([QueryConfig(id="m", promql="up")]) + source = _FakeSource({"up": []}) + report = await compute_warmup(config=cfg, source=source) + status = report.groups[0].queries[0] + assert status.status == "warming" + assert status.points_available == 0 + assert status.blocked_by == "lookback" + + +@pytest.mark.asyncio +async def test_warmup_reports_longest_series() -> None: + """A multi-series query reports the best-off (longest) series. + + The runner's per-series fit treats each frame independently, so + the longest series unblocks the query first. Reporting the mean + would silently smear over a single slow-to-warm peer. + """ + cfg = _config([QueryConfig(id="m", promql="up")], min_points=15) + source = _FakeSource({"up": [_frame(5), _frame(20), _frame(10)]}) + report = await compute_warmup(config=cfg, source=source) + status = report.groups[0].queries[0] + assert status.points_available == 20 + assert status.status == "ready" + + +@pytest.mark.asyncio +async def test_warmup_reports_error_on_probe_failure() -> None: + class _BadSource(_FakeSource): + async def query_range( + self, + promql: str, + start: datetime, + end: datetime, + step_seconds: int, + ) -> list[SeriesFrame]: + raise RuntimeError("boom") + + cfg = _config([QueryConfig(id="m", promql="up")]) + source = _BadSource({}) + report = await compute_warmup(config=cfg, source=source) + status = report.groups[0].queries[0] + assert status.status == "error" + assert status.blocked_by == "lookback" + + +@pytest.mark.asyncio +async def test_warmup_event_profile_short_circuits_probe() -> None: + """Event-profile queries don't have a min_points contract.""" + cfg = _config( + [QueryConfig(id="m", promql="pagerduty_incidents_total", data_profile="events")], + ) + source = _FakeSource({}) + report = await compute_warmup(config=cfg, source=source) + status = report.groups[0].queries[0] + assert status.status == "ready" + # No probe issued — the event path skips the cost entirely. + assert source.calls == [] + + +@pytest.mark.asyncio +async def test_warmup_required_regressor_visible_when_no_primary_samples() -> None: + """When the primary returns zero but a regressor is required, surface the dependency.""" + query = QueryConfig( + id="m", + promql="up", + regressors=[RegressorConfig(id="deploys", promql="changes(deploys[1h])", required=True)], + ) + cfg = _config([query]) + source = _FakeSource({"up": []}) + report = await compute_warmup(config=cfg, source=source) + status = report.groups[0].queries[0] + # Primary returned 0 points -> blocked on lookback first; required-regressor + # surface kicks in only at the deeper code path where min_points is hit + # *with* zero samples — the lookback wording dominates here, which matches + # the runner's failure order. + assert status.blocked_by == "lookback" + + +@pytest.mark.asyncio +async def test_warmup_to_dict_round_trip() -> None: + """The JSON envelope shape is stable and re-parseable.""" + cfg = _config([QueryConfig(id="m", promql="up")], min_points=5) + source = _FakeSource({"up": [_frame(10)]}) + report = await compute_warmup(config=cfg, source=source) + payload = report.to_dict() + assert "groups" in payload + assert payload["groups"][0]["name"] == "g" + q0 = payload["groups"][0]["queries"][0] + assert q0["id"] == "m" + assert q0["status"] == "ready" + assert q0["points_available"] == 10 + assert q0["points_needed"] == 5 + assert q0["blocked_by"] == "none" + assert q0["eta_seconds"] == 0.0 + + +def test_render_table_includes_header_and_rows() -> None: + report = WarmupReport( + groups=[ + GroupWarmupStatus( + name="g", + queries=[ + QueryWarmupStatus( + id="ready_one", + status="ready", + points_available=200, + points_needed=100, + blocked_by="none", + eta_seconds=0.0, + ), + QueryWarmupStatus( + id="warming_one", + status="warming", + points_available=40, + points_needed=100, + blocked_by="min_points", + eta_seconds=3600.0, + ), + ], + ) + ] + ) + text = render_table(report) + assert "group" in text + assert "ready_one" in text + assert "warming_one" in text + assert "1h" in text # the 3600s ETA renders as 1h + + +def test_render_table_handles_empty_report() -> None: + text = render_table(WarmupReport(groups=[])) + assert "no groups" in text + + +# ----- HTTP-level wiring tests ------------------------------------------- +# +# Cover the route registration in ``main._build_app``: the endpoint +# only mounts when ``expose_warmup_endpoint=True`` AND a ``source`` is +# wired, and the JSON envelope must match what the CLI's +# ``_report_from_payload`` re-parses. + + +def test_endpoint_off_by_default_returns_404() -> None: + cfg = _config([QueryConfig(id="m", promql="up")]) + exporter = Exporter() + runner = Runner(config=cfg, source=_FakeSource({"up": [_frame(20)]}), exporter=exporter) + reloader = ConfigReloader( + runner=runner, + exporter=exporter, + config_path=None, # type: ignore[arg-type] + ) + app = _build_app( + exporter, + runner, + reloader, + reload_enabled=False, + expose_warmup_endpoint=False, + ) + client = TestClient(app) + assert client.get("/forecast/warmup").status_code == 404 + + +def test_endpoint_returns_report_when_enabled() -> None: + cfg = _config([QueryConfig(id="m", promql="up")], min_points=10) + exporter = Exporter() + source = _FakeSource({"up": [_frame(20)]}) + runner = Runner(config=cfg, source=source, exporter=exporter) + reloader = ConfigReloader( + runner=runner, + exporter=exporter, + config_path=None, # type: ignore[arg-type] + ) + app = _build_app( + exporter, + runner, + reloader, + reload_enabled=False, + expose_warmup_endpoint=True, + source=source, + ) + client = TestClient(app) + response = client.get("/forecast/warmup") + assert response.status_code == 200 + payload = response.json() + assert payload["groups"][0]["name"] == "g" + q0 = payload["groups"][0]["queries"][0] + assert q0["id"] == "m" + assert q0["status"] == "ready" + assert q0["points_available"] == 20 + assert q0["blocked_by"] == "none" + + +def test_endpoint_disabled_when_source_missing() -> None: + """``expose_warmup_endpoint=True`` + no source logs a warning, no route.""" + cfg = _config([QueryConfig(id="m", promql="up")]) + exporter = Exporter() + runner = Runner(config=cfg, source=_FakeSource({}), exporter=exporter) + reloader = ConfigReloader( + runner=runner, + exporter=exporter, + config_path=None, # type: ignore[arg-type] + ) + app = _build_app( + exporter, + runner, + reloader, + reload_enabled=False, + expose_warmup_endpoint=True, + source=None, + ) + client = TestClient(app) + assert client.get("/forecast/warmup").status_code == 404 + + +# ----- Required-regressor probing ---------------------------------------- + + +@pytest.mark.asyncio +async def test_warmup_blocks_on_missing_required_regressor() -> None: + """Primary returns enough samples; required regressor returns zero -> block.""" + query = QueryConfig( + id="m", + promql="up", + regressors=[ + RegressorConfig(id="deploys", promql="deploys_total", required=True), + ], + ) + cfg = _config([query], min_points=5) + source = _FakeSource( + { + "up": [_frame(10)], + "deploys_total": [], # required regressor is empty + } + ) + report = await compute_warmup(config=cfg, source=source) + status = report.groups[0].queries[0] + assert status.status == "warming" + assert status.blocked_by == "regressor:deploys" + # The probe issued one extra query against the regressor PromQL. + assert "deploys_total" in source.calls + + +@pytest.mark.asyncio +async def test_warmup_ready_when_required_regressor_has_samples() -> None: + """A populated required regressor doesn't block.""" + query = QueryConfig( + id="m", + promql="up", + regressors=[ + RegressorConfig(id="deploys", promql="deploys_total", required=True), + ], + ) + cfg = _config([query], min_points=5) + source = _FakeSource( + { + "up": [_frame(10)], + "deploys_total": [_frame(10)], + } + ) + report = await compute_warmup(config=cfg, source=source) + status = report.groups[0].queries[0] + assert status.status == "ready" + assert status.blocked_by == "none" + + +@pytest.mark.asyncio +async def test_warmup_skips_non_required_regressors() -> None: + """Optional regressors are not probed, even when empty.""" + query = QueryConfig( + id="m", + promql="up", + regressors=[ + RegressorConfig(id="deploys", promql="deploys_total", required=False), + ], + ) + cfg = _config([query], min_points=5) + source = _FakeSource({"up": [_frame(10)]}) + report = await compute_warmup(config=cfg, source=source) + status = report.groups[0].queries[0] + assert status.status == "ready" + # The optional regressor's PromQL is never queried. + assert "deploys_total" not in source.calls + + +@pytest.mark.asyncio +async def test_warmup_blocks_when_required_regressor_probe_raises() -> None: + """A regressor whose probe raises is treated as blocking. + + Mirrors the runner's behaviour: a required regressor the TSDB + can't serve would land under ``forecast_regressor_failures_total`` + on the live runner, so the warmup endpoint surfaces the same + dependency upfront rather than letting the operator wait for the + fit attempt to fail. + """ + + class _PrimaryOkRegressorBoom(_FakeSource): + async def query_range( + self, + promql: str, + start: datetime, + end: datetime, + step_seconds: int, + ) -> list[SeriesFrame]: + self.calls.append(promql) + if promql == "deploys_total": + raise RuntimeError("regressor probe blew up") + return self._by_query.get(promql, []) + + query = QueryConfig( + id="m", + promql="up", + regressors=[ + RegressorConfig(id="deploys", promql="deploys_total", required=True), + ], + ) + cfg = _config([query], min_points=5) + source = _PrimaryOkRegressorBoom({"up": [_frame(10)]}) + report = await compute_warmup(config=cfg, source=source) + status = report.groups[0].queries[0] + assert status.status == "warming" + assert status.blocked_by == "regressor:deploys" + assert "deploys_total" in source.calls + + +@pytest.mark.asyncio +async def test_warmup_discover_query_blocks_with_discovery_token() -> None: + """Templated queries short-circuit to status=ready with blocked_by=discovery. + + The probe can't run against an unresolved template, but reporting + ``blocked_by=none`` would mislead the operator into thinking the + query is actively forecasting. ``discovery`` makes the dependency + explicit while keeping status=ready so the CLI's until-loop + terminates (templated queries are not blocking the install). + """ + from promforecast.config import DiscoveryVariable # noqa: PLC0415 + + query = QueryConfig( + id="m", + promql="up{instance=~'{{ instance }}'}", + discover=[ + DiscoveryVariable(name="instance", label="instance", promql="up"), + ], + ) + cfg = _config([query]) + source = _FakeSource({}) # never reached + report = await compute_warmup(config=cfg, source=source) + status = report.groups[0].queries[0] + assert status.status == "ready" + assert status.blocked_by == "discovery" + # The probe was never issued — templated queries don't touch the + # source at all. + assert source.calls == [] + + +@pytest.mark.asyncio +async def test_warmup_required_regressor_ordering_is_deterministic() -> None: + """Multiple blocked regressors resolve to config order, not race order. + + The runner-side ``forecast_regressor_failures_total`` counter is + keyed by regressor id; the warmup endpoint reports exactly one + ``blocked_by`` per query, so the first-by-config rule must be + stable so successive calls don't flap between "blocked on A" + and "blocked on B" depending on which probe response landed + first. + """ + query = QueryConfig( + id="m", + promql="up", + regressors=[ + RegressorConfig(id="deploys", promql="deploys_total", required=True), + RegressorConfig(id="rollouts", promql="rollouts_total", required=True), + ], + ) + cfg = _config([query], min_points=5) + source = _FakeSource( + { + "up": [_frame(10)], + "deploys_total": [], + "rollouts_total": [], + } + ) + report = await compute_warmup(config=cfg, source=source) + status = report.groups[0].queries[0] + assert status.blocked_by == "regressor:deploys" + + +# ----- CLI plumbing ------------------------------------------------------ + + +def test_run_warmup_cli_exits_zero_on_empty_config( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """``until promforecast warmup`` must terminate against an empty config. + + Returning ``1`` for "no groups configured" would spin a docs-recommended + ``until promforecast warmup`` loop forever against a fresh install whose + operator hasn't added any queries yet. The forecaster itself is healthy + — there's just nothing to wait on — so ``0`` is the right exit code. + """ + import httpx # noqa: PLC0415 + + from promforecast.main import _run_warmup_cli # noqa: PLC0415 + + class _FakeResponse: + status_code = 200 + + def raise_for_status(self) -> None: + return None + + def json(self) -> dict[str, list[dict[str, object]]]: + return {"groups": []} + + monkeypatch.setattr(httpx, "get", lambda url, timeout: _FakeResponse()) + rc = _run_warmup_cli(url="http://x", timeout_seconds=1.0, json_output=True) + assert rc == 0 + + +def test_run_warmup_cli_exits_one_when_a_query_is_warming( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A non-ready status returns exit ``1`` so the until-loop keeps polling.""" + import httpx # noqa: PLC0415 + + from promforecast.main import _run_warmup_cli # noqa: PLC0415 + + payload = { + "groups": [ + { + "name": "g", + "queries": [{"id": "q", "status": "warming", "blocked_by": "min_points"}], + } + ] + } + + class _FakeResponse: + status_code = 200 + + def raise_for_status(self) -> None: + return None + + def json(self) -> dict[str, object]: + return payload + + monkeypatch.setattr(httpx, "get", lambda url, timeout: _FakeResponse()) + rc = _run_warmup_cli(url="http://x", timeout_seconds=1.0, json_output=True) + assert rc == 1