Live App: https://driftmonitor.streamlit.app/ Data Source: https://archive.ics.uci.edu/dataset/350/default+of+credit+card+clients
Production monitoring for a deployed credit-default model — the half of the ML lifecycle that starts after the model ships.
DriftWatch trains a real credit-default classifier on the UCI Default of Credit Card Clients dataset, then watches it in production: it detects when the input data drifts, when the model's accuracy decays, and when the service gets slow — and fires an alert, with false-alarm discipline, when any of the three crosses a threshold you control.
The headline result (reproducible from a clean run): under a concept-drift scenario — a regime change like a recession or a relief program — the model's F2 falls from a 0.640 baseline to 0.565 and recall collapses from 0.88 to 0.62, while data drift stays at just 4%. The inputs look normal; the relationship between borrower and default has inverted, and the model is silently missing roughly 4 in 10 of the defaults it used to catch. A monitor watching only data drift would report "all clear." This is the failure mode pure drift monitoring never catches — the real 2008 / COVID risk-model failure — and it fires a critical decay alert.
A shipped model is not a finished model. Two things go wrong after deployment, on two different clocks. Data drift — the input distribution moving — you can see immediately, on every prediction. Performance decay — the model actually getting worse — you can only see once ground-truth labels arrive, which for loan defaults is a full billing cycle later. DriftWatch keeps them separate, because they mean different things (drift is an early warning; decay is confirmed loss) and demand different responses (watch vs. retrain). Add latency against an SLO and alerting with hysteresis, and you have the operational layer a real deployment needs.
A gradient-boosted classifier trained in model.py on the real UCI dataset — 30,000 cardholders, 23 features plus a derived utilization feature — predicting next-month default. It reports honest held-out metrics: AUC ≈ 0.78, which is realistic for a genuinely hard problem, not an inflated number from synthetic data. Default is imbalanced (~22%), so the model is tuned and judged on F2 (recall-weighted: a missed default costs more than a false flag), not accuracy.
- Instruments every prediction (features, output, latency) via FastAPI middleware that never blocks or breaks the prediction path.
- Detects drift per-feature with explicit, testable statistics — KS for numeric features, PSI for categorical — plus an Evidently report per run for deep-dives.
- Detects decay by reconciling predictions against late-arriving labels, reporting F2 against the deployment baseline. Honestly reports "awaiting labels" instead of faking a number from a tiny sample.
- Monitors latency at p50/p95/p99 — tail latency, because that's what a user actually feels.
- Alerts with hysteresis (a breach must persist N consecutive runs before firing) and auto-recovery, to console / GitHub issue / Slack.
- Visualizes everything on a live Streamlit dashboard.
- Demonstrates itself with an honest drift simulator — six named, credit-framed scenarios that deliberately inject regressions so the monitor has something real to catch.
pip install -r requirements.txt
# clean run that reproduces the headline critical decay alert
python simulate.py concept_drift --n 500 --label-frac 0.7 --intensity 1.5
python engine/run.py
python simulate.py concept_drift --n 500 --label-frac 0.7 --intensity 1.5
python engine/run.py # the second run fires the critical decay alert
# view it
streamlit run dashboard/app.pyThe alert prints to the terminal:
[DECAY/critical] F2 0.565 is 0.075 below baseline 0.640 (recall 0.619) — model decay.
run f5a612f3 | drift 4% (1 feats) | p95 11ms | F2 0.565
Note: start each scenario from a fresh
observatory.db(or let the 24-hour window roll) so the monitoring window isn't mixing scenarios — otherwise prior predictions accumulate in the window. In production the time-scoped window handles this automatically.
| Scenario | Demonstrates |
|---|---|
baseline |
Stable economy — no alerts (drift 0%, F2 ~0.73) |
economic_downturn |
Utilization rises, limits cut -> drift alert |
payment_shock |
Repayment status worsens -> drift |
new_product_segment |
Younger, thinner-file applicants -> localized drift |
concept_drift |
Decay with near-zero drift — the headline |
latency_spike |
Latency SLO breach only |
UCI Default of Credit Card Clients — 30,000 records, 23 features, ~22% default rate.
Source: https://archive.ics.uci.edu/dataset/350/default+of+credit+card+clients
The .xls lives in data/. If it's ever absent, model.py falls back to a faithful synthetic dataset on the same schema (clearly logged as a fallback).
pytest -q # 15 tests: schema, store, drift math, decay, hysteresis
ruff check .SPEC.md— architecture and design decisionsRUNBOOK.md— what each alert means and what an on-call analyst does
The model and its ~0.78 AUC are real, trained on the real dataset. The drift is deliberately injected and labelled as such — the point isn't a performance claim, it's a faithful demonstration of production monitoring methodology. Alert thresholds are calibrated to this model's real operating envelope and documented in the runbook. The Postgres store path is provided but untested against a live database; SQLite is the tested default.
Python · scikit-learn · scipy · Evidently · FastAPI · Streamlit · Plotly · pytest · GitHub Actions