A drift-detection sidecar that watches ML feature and prediction streams and raises statistically justified alarms using two-sample tests under a Benjamini-Hochberg alert budget, cutting false-alarm evaluation rounds on drift-free traffic from 35.9% to 4.1% (measured).
- Pain: models rot silently in production; teams usually find out from business metrics weeks later.
- Mechanism: a sidecar runs per-feature KS and chi-squared tests against a training-time reference, spends one corrected alert budget across all features, and latches alarms through a hysteresis state machine.
- Measured result: on simulated drift-free streams, 35.9% of naive evaluation rounds flagged something; with the corrected budget it is 4.1%, while all 75 injected drifts across 5 drift types were still detected (mean 5.5 to 12 batches after the change point).
A scoring model does not throw an exception when the world changes under it. An upstream team renames a category code, a marketing push shifts the applicant mix, a feature pipeline silently starts emitting a default value, and the model keeps returning confident numbers. If a mispriced decision costs even a few currency units and the service scores tens of thousands of rows a day, every silent week is a five-to-six figure hole, and the usual detection channel (a monthly business review) is weeks behind the change point.
drift-radar attacks the mechanism, not the symptom. At training time you register a reference snapshot per feature (numeric samples, categorical counts, and the prediction distribution). In production, the model service posts each scored batch to the sidecar's /log endpoint. The sidecar maintains sliding windows per stream, and every eval_interval rows it runs two-sample Kolmogorov-Smirnov tests on numeric streams and chi-squared tests on categorical streams, plus PSI as a magnitude measure. The nameable twist is the alerting layer: instead of nine independent "p < 0.05" tripwires, the per-round p-values go through Benjamini-Hochberg correction so the whole feature set spends one false-discovery budget, and a state machine with hysteresis and window-refresh confirmation turns rejections into OK, WARN, ALARM states that Prometheus scrapes and Grafana displays. scipy provides the tests themselves; the windowing, category alignment, correction layer, state machine, service and exposition are built in this repo.
The claims are measured, not asserted, and all inputs are simulated streams with a committed generator (benchmark/generator.py). On 30 drift-free credit-scoring streams of 15,000 rows (2 vCPU, 4GB shared container), naive per-feature alerting flagged 35.9% of evaluation rounds and eventually paged on 28 of 30 runs; the corrected pipeline flagged 4.1% of rounds and paged on 4 of 30. Against 75 injected drifts (mean shift, variance change, category mix shift, prediction shift, gradual creep, 15 runs each) it detected 75 of 75, with mean detection delay between 5.5 and 5.9 batches of 250 rows for abrupt drifts and 12.0 for gradual creep. The ingest path sustains 11,340 rows/sec at concurrency 4 with p95 of 49 ms on the same container.
flowchart LR
subgraph model_boundary["model service failure domain"]
MS[model service] -- "scored batches POST /log" --> SC
end
subgraph sidecar["drift-radar sidecar (its crash never blocks scoring)"]
SC[FastAPI ingest] --> WIN["sliding windows\nper feature stream"]
REF["reference snapshot\n(training distribution)"] --> DET
WIN --> DET["detectors: KS, chi-squared, PSI"]
DET --> COR["BH / Bonferroni\nalert budget"]
COR --> SM["alert state machine\nOK to WARN to ALARM"]
SM --> EXP["/metrics exposition"]
WIN -. "periodic snapshot\n(atomic write)" .-> DISK[(snapshot file)]
end
subgraph observability["observability failure domain (loss = blind, not broken)"]
EXP -- scrape 15s --> PROM[Prometheus]
PROM --> GRAF[Grafana dashboard]
PROM --> AM["alert rules\n(page on state == 2)"]
end
Failure boundaries: the sidecar is fire-and-forget from the model service's point of view, a radar crash costs monitoring, never scoring. Prometheus or Grafana loss costs visibility but the radar keeps evaluating and latching state. Snapshot loss costs at most the last snapshot interval of window state (see Failure Modes).
| Technology | Role | Why chosen here |
|---|---|---|
| FastAPI + pydantic | HTTP sidecar, request validation | Batch payloads validated at the edge; async handlers keep ingest cheap |
| scipy.stats | ks_2samp, chi2_contingency | Reference implementations of the two-sample tests; not worth rewriting |
| numpy | window math, PSI, BH procedure | Vectorized quantile bins and step-up procedure |
| prometheus-client | /metrics exposition | Real registry with counters, gauges, histogram; native Grafana pairing |
| uvicorn | ASGI server | Single-worker profile matches the sidecar deployment |
| pytest + pytest-cov | test suite, 95% line coverage | Detector math and state transitions are exactly the code that must not rot |
| Prometheus + Grafana (compose) | scraping, dashboard, paging rules | Committed under deploy/, validated YAML; see honesty note in Quickstart |
Author-built vs library-provided: scipy provides the two test statistics. The sliding windows, category alignment, PSI, the correction layer, the hysteresis and confirmation state machine, the sidecar API, the metrics exposition and the benchmark harness are this repo's code.
Requires Python 3.10+.
git clone https://github.com/panchalvedant13/drift-radar.git
cd drift-radar
python3 -m venv venv
./venv/bin/pip install -e ".[dev]"
./venv/bin/pytest -q # 63 tests, all greenStart the sidecar, then stream demo data and watch the alarm latch:
./venv/bin/uvicorn --factory driftradar.service:create_app --port 8000 &
./venv/bin/python examples/demo_stream.pyThe demo registers a 5,000-row reference, streams 10 stable batches (overall state stays OK), then injects an income mean shift and polls /drift/status until the income stream reaches ALARM (6 drifted batches in the committed run). Then look at the raw metrics:
curl -s localhost:8000/metrics | grep driftradar_feature_state
# driftradar_feature_state{feature="income"} 2.0 <- ALARMConfiguration is env-first (DRIFTRADAR_ALPHA, DRIFTRADAR_CORRECTION, DRIFTRADAR_WINDOW_SIZE, DRIFTRADAR_SNAPSHOT_PATH, ...) or a YAML file via DRIFTRADAR_CONFIG; see driftradar/config.py for every knob and default.
The Prometheus + Grafana stack is committed under deploy/ (compose file, scrape config, paging rules, dashboard JSON, all validated YAML/JSON). Honesty note: the build sandbox has no docker daemon, so the compose stack is provided but was not executed there; the service itself was benchmarked bare with uvicorn.
cd deploy && docker compose up --build # sidecar :8000, Prometheus :9090, Grafana :3000Methodology: asyncio load generator (benchmark/load_test.py, httpx) fires POST /log batches of 100 rows (8 features plus prediction per row) against uvicorn (single worker) for 15 s per concurrency step after a 20-request warmup. Client and server share the same 2 vCPU, 4GB container, so client overhead is included in the latencies. Raw outputs: benchmark/results/load_test.json.
xychart-beta
title "POST /log latency vs concurrency (ms, 100-row batches, 2 vCPU shared)"
x-axis "concurrent clients" [4, 16, 64]
y-axis "latency (ms)" 0 --> 1700
line [36.04, 183.62, 835.29]
line [49.36, 269.06, 1386.52]
line [64.83, 301.89, 1669.88]
Lines bottom to top: p50, p95, p99 (mermaid xychart has no legend yet; the raw table below carries the same numbers).
| Concurrency | Requests | Errors | p50 (ms) | p95 (ms) | p99 (ms) | Rows/sec |
|---|---|---|---|---|---|---|
| 4 | 1,706 | 0 | 36.04 | 49.36 | 64.83 | 11,340 |
| 16 | 1,258 | 0 | 183.62 | 269.06 | 301.89 | 8,323 |
| 64 | 1,062 | 0 | 835.29 | 1,386.52 | 1,669.88 | 6,782 |
Honest degradation: throughput peaks at low concurrency and falls 40% by concurrency 64 while p99 grows 26x, because a single-process Python service on 2 shared vCPUs queues rather than parallelizes; past roughly 4 in-flight requests added concurrency only buys queueing delay.
Full setup and raw numbers: benchmark/detection_experiment.py and benchmark/results/detection_quality.json. Streams simulate a credit-scoring service (9 monitored streams: 5 numeric, 3 categorical, 1 prediction). Window 2000, batch 250, alpha 0.05, hysteresis 2/3, confirmation half-window.
False positives on drift-free streams (30 runs x 60 evaluation rounds):
| Correction | Eval rounds with any rejection | Runs that ever paged |
|---|---|---|
| none (naive p < 0.05) | 35.9% | 28 / 30 |
| Bonferroni | 4.0% | 4 / 30 |
| Benjamini-Hochberg (default) | 4.1% | 4 / 30 |
Detection of injected drifts (BH, 15 runs each, delay in 250-row batches after the change point):
| Drift type | Detected | Mean delay | Min | Max |
|---|---|---|---|---|
| Mean shift (income) | 15/15 | 5.87 | 5 | 6 |
| Variance change (utilization) | 15/15 | 5.80 | 3 | 6 |
| Category mix shift (employment) | 15/15 | 5.87 | 5 | 7 |
| Prediction shift (score) | 15/15 | 5.53 | 5 | 6 |
| Gradual creep (age) | 15/15 | 12.00 | 8 | 14 |
Residual honesty: even corrected, 4 of 30 clean 15,000-row runs eventually paged once. That is the price of testing every 250 rows against a finite reference sample; alpha and the confirmation fraction are the knobs to trade this against detection delay.
- ADR-001: Statistical tests with FDR correction over ML-based drift classifiers
- ADR-002: In-memory sliding windows with periodic snapshot over Kafka plus a stream processor
- Concept-drift feedback (comparing predictions to realized outcomes): out until ground-truth labels arrive within an SLA the radar can rely on; covariate and prediction drift need no labels at all.
- Multivariate drift detection (correlation changes that preserve marginals): out until per-feature alarms are demonstrably insufficient in practice; ADR-001 records the trigger.
- Automated retraining or rollback: the radar raises alarms with evidence; acting on them stays a human or CD-pipeline decision.
- Cross-replica window aggregation: one radar per model replica by design; ADR-002 records the trigger to move to a stream processor.
- Text, image or embedding drift: numeric and categorical tabular streams only.
- No secrets in the repo or images: Grafana credentials come from the environment (GRAFANA_ADMIN_PASSWORD), and the radar itself holds no credentials.
- Raw feature values are never logged: structured JSON logs carry request ids, feature names and aggregate statistics (p-values, PSI, counts) only. Exception payloads contain feature names, not values.
- PII note: the radar necessarily holds recent raw numeric values in memory (that is what a KS test needs) and in the optional snapshot file. Treat the snapshot path as sensitive storage, restrict it to the pod volume, and prefer max_reference_sample tuning over dumping full training sets into /reference. Categorical windows store category labels and counts only.
- Input validation at the edge: pydantic schemas, a 10,000-row batch cap, unknown features rejected with 422, feature and category count caps to bound memory and metric cardinality.
| Failure | Detection | Behavior | Recovery |
|---|---|---|---|
| Reference window missing | /log returns 409 with "no reference registered"; driftradar_ingest_errors_total{reason="no_reference"} climbs | Ingest refuses rather than silently buffering unverifiable rows | POST /reference; the client keeps scoring, only monitoring is deferred |
| Window starvation (low traffic) | driftradar_window_rows below min_window; WindowStarvation Prometheus rule after 30m | Streams report INSUFFICIENT_DATA, tests do not run, no false verdicts from thin data | Traffic resumes or min_window is lowered deliberately |
| Metric cardinality explosion | max_features (100) and max_categories (50) enforced at registration with 422 | Bounded by construction; label sets cannot grow past the caps | Raise caps consciously in config, or split streams across radars |
| Sidecar restart (state loss) | has_reference false in /health; gap in scrape | With DRIFTRADAR_SNAPSHOT_PATH set, boot restores reference, windows and alert latches from the last atomic snapshot (at most snapshot_every_evals evaluations stale). Without it, radar starts empty | Re-register reference (or rely on restore); windows refill within min_window rows per stream |
The first full detection run produced a number I did not want to commit: with Benjamini-Hochberg correction working exactly as designed (per-round false-positive rate 4.1%, right at alpha), 11 of 30 drift-free runs still ended up paging. The correction was fine; the state machine was lying to itself. With 250-row batches sliding over a 2000-row window, consecutive evaluations share 87.5% of their data, so my "two consecutive rejections before ALARM" hysteresis was not demanding two pieces of evidence, it was counting the same unlucky sample twice. The fix (commit cf57d72) makes escalation from WARN to ALARM additionally require that at least half the window has refreshed with fresh rows since WARN was entered, so the confirming test runs on mostly independent data; a severe PSI still fast-tracks large real shifts. Rerunning the full experiment: clean-run paging fell from 11/30 to 4/30 while all 75 injected drifts remained detected, at the honest cost of mean detection delay rising from 2.9 to 5.8 batches for abrupt drifts. Both runs are committed (before: d052fe0, after: c677446) so the trade-off is inspectable, not narrated.
- Adaptive eval_interval: evaluate more often when any stream is in WARN, less when everything is quiet, to buy back detection delay without raising the false-alarm budget.
- Per-feature alpha weighting so business-critical features get a larger share of the BH budget.
- Anderson-Darling as an alternative numeric test for tail-sensitive features (KS is weakest in the tails).
- A /drift/explain endpoint returning reference-vs-window histograms per alarming feature for one-click incident context.
- Optional label-feedback module for concept drift once ground-truth labels arrive within SLA (see out of scope).
MIT, see LICENSE.