A threshold tuned for F1 treats a missed 900 dollar account takeover and a missed 4 dollar card test as the same mistake. This pipeline picks its operating threshold in dollars instead: false positives cost a fixed analyst review, false negatives cost the transaction amount, and the optimizer minimizes the measured total. On the held-out test window that cuts operating cost by 30.5 percent (9,871 dollars over nine days) against the default threshold and reduces false positives from 189 to 16, while a from-scratch PSI monitor with a flag-rate canary catches the injected drift window that follows.
Built end to end: seeded synthetic data with three fraud archetypes, leakage-safe features verified against a PySpark implementation, XGBoost and LightGBM against a calibrated linear baseline, an Airflow DAG with a failing drift gate, warehouse DDL with mart queries, and Power BI ready star-schema exports. Every number below was measured by code in this repository.
flowchart LR
G[generator<br/>252,818 txns, 1.39% fraud] --> V[velocity features<br/>strictly past windows]
V --> S[temporal split<br/>train / valid / test / drift]
S --> T[train<br/>logreg, XGBoost, LightGBM]
T --> O[threshold optimizer<br/>dollars, not F1]
O --> SC[batch scoring<br/>327k rows/s]
SC --> M[drift monitor<br/>PSI + flag rate canary]
M --> E[exports<br/>Power BI star schema + SQL marts]
SP[Spark feature job] -. parity test .- V
AF[Airflow DAG] --> SC
M -. action status fails the DAG .-> AF
The cost of operating at threshold t is measured, not assumed:
cost(t) = review_cost x n_flagged(t) + sum(amount of each missed fraud at t)
The false negative term uses each transaction's own amount. A 900 dollar takeover pulls the optimum harder than a 4 dollar card test, which is exactly how the fraud ledger sees it.
Measured on the test window (19,112 transactions, 246 fraud, 9 days, review cost 75 dollars):
| Threshold policy | t | Total cost | Reviews | Missed fraud | FP | FN | Precision | Recall |
|---|---|---|---|---|---|---|---|---|
| Default 0.5 | 0.500 | $32,381 | $32,025 | $356 | 189 | 8 | 0.557 | 0.968 |
| F1 optimal | 0.955 | $23,194 | $17,175 | $6,019 | 14 | 31 | 0.939 | 0.874 |
| Cost optimal | 0.950 | $22,510 | $17,475 | $5,035 | 16 | 29 | 0.931 | 0.882 |
Savings: 9,871 dollars (30.5 percent) against the default and 685 dollars against F1, with 91.5 percent fewer false positives than the default. The optimizer accepts missing 29 mostly small frauds because reviewing everything the default flags costs six times more than the fraud it prevents.
The part F1 cannot do at all: when the review cost changes, the operating point follows the economics.
| Review cost | Cost-optimal threshold | F1-optimal threshold |
|---|---|---|
| $25 | 0.800 | 0.955 |
| $75 | 0.950 | 0.955 |
| $150 | 0.975 | 0.955 |
Cost curve on the validation window (dollars vs threshold, minimum at 0.95):
xychart-beta
x-axis "threshold" [0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 0.98]
y-axis "total cost (USD)" 40000 --> 180000
line [174974, 133302, 104055, 87460, 77988, 72233, 66256, 61618, 55875, 55916, 55483, 58621]
Temporal split by day, never random: train days 1 to 70, validation 71 to 90 (early stopping and threshold selection), test 91 to 99, drift window 100 to 120. Test window metrics:
| Model | PR-AUC | ROC-AUC |
|---|---|---|
| Logistic regression (scaled, class weighted) | 0.794 | 0.973 |
| LightGBM | 0.955 | 0.999 |
| XGBoost (champion) | 0.957 | 0.999 |
The 0.16 PR-AUC gap between the linear baseline and the tree models is the value of interactions: card testing is only visible when small amounts, card-not-present, and single-card velocity fire together. ROC-AUC barely separates the models at 1.3 percent positives, which is why selection uses PR-AUC.
Population PSI catches covariate drift, shifts in all traffic. It is structurally blind to concept drift confined to the fraud minority: 1.4 percent of rows cannot move a population histogram. The flag rate, the share of traffic crossing the operating threshold, is the label-free canary for exactly that blind spot. Reference for both is the validation window, not training data; the model's optimistic scores on its own training rows biased the reference flag rate high and made every serving window look drifted.
Measured results, same monitor on both windows:
| Window | Status | Top PSI | Flag rate | Post-hoc recall |
|---|---|---|---|---|
| Test (days 91 to 99) | stable | all below 0.002 | -4.2% | 0.882 |
| Drift (days 100 to 120) | warning | cat_risk 0.125, card_present 0.040 | -7.4% | 0.707 |
The drift window simulates both sides of a real Q4: legit traffic shifts toward online holiday categories (covariate drift, caught by cat_risk PSI) while fraudsters adapt with smaller amounts and card-present takeovers (concept drift, visible as the flag rate softening). The two effects partially cancel in the aggregate flag rate, holiday false positives rise while adapted fraud slips through, which is why the monitor needs both detectors. Post-hoc labels confirm what the label-free monitor suspected: recall fell from 0.882 to 0.707.
The model scored a perfect ROC-AUC, and that was the bug. The first full training run returned ROC-AUC 1.0000 and PR-AUC 0.9987 for both boosting models. A perfect score on a fraud task means the data is broken, not that the model is good: the fraud archetypes had giveaway signatures (night-only card testing in one category, takeovers always far away on new devices, probe amounts in a tight band under round limits). The fix added overlap in both directions: legit travelers with large geo distances and device changes, night-shift spending, cheap digital subscriptions, and fraud camouflage. Champion PR-AUC landed at a defensible 0.957 with a real gap over the baseline. The fix is its own commit with the failure in the message.
Spark and pandas disagreed on exactly 1 row in 15,915. The parity test caught a 0.006 percent mismatch: two card-testing transactions sharing the same second. The pandas build counted the equal-timestamp predecessor as past; Spark's rangeBetween(-(w-1), -1) excluded it. Equal timestamps are concurrent events, and whether one is "before" the other depends on sort order that a distributed engine does not guarantee. Canonical semantics chosen: strictly past means ts_j < ts_i. The pandas side now uses the tie-group boundary via searchsorted, and the parity test pins both implementations together.
A ten point shift scored a PSI of exactly 0.0. Card-present share moves from 62 to 52 percent in the drift window, and the first monitor version scored it 0.0: quantile bin edges collapse on a binary feature and all mass lands in one bin. Low-cardinality features are now compared as exact-value categories, and a regression test locks the behavior. Building PSI by hand surfaced a failure a library default would have hidden.
pip install -e ".[dev]"
python -m frauddet.pipeline # full run, under a minute: data to exports
pytest -q # 26 tests, 92 percent coverage
python benchmark/run_benchmark.py # throughput numbers on your machineArtifacts land in artifacts/: eval_report.json (every number above), cost_curve.csv, drift reports, model.pkl, and powerbi/ exports.
Spark parity (optional, needs Java 17 or 21):
pip install -e ".[dev,spark]"
pytest tests/test_spark_parity.py -qAirflow (optional): pip install -e ".[dev,orchestration]", point AIRFLOW_HOME at airflow_home/, and the fraud_detection_pipeline DAG appears with a drift gate that fails the run on action status.
| Operation | Measured |
|---|---|
| XGBoost batch scoring | 326,987 rows/s (252,818 rows in 0.77s) |
| Feature build, pandas | 6.7s for 252,818 rows |
| Feature build, Spark local | 12.8s for the same rows |
Local Spark loses to pandas at this scale because it pays JVM and shuffle overhead; it exists for the scale where pandas cannot hold the data, and the parity test guarantees both produce identical features.
sql/ddl/create_tables.sql defines the star schema; the pipeline writes matching CSVs to artifacts/powerbi/ (fact_scores, fact_daily_summary, dim_category, dim_threshold, drift_history). sql/marts/ holds the queries behind the dashboard pages: daily fraud summary with catch rates and dollars, an analyst queue ranked by expected loss (score times amount, so a 900 dollar takeover outranks a 4 dollar card test at equal score), and a threshold what-if grid. Load the CSVs into Power BI Desktop directly or run the DDL and marts in a warehouse.
| Failure | Behavior |
|---|---|
| Drift status reaches action | Airflow drift gate raises, run goes red, scoring stops until retrain |
| Binary or low-cardinality feature drifts | Exact-value categorical PSI, regression tested |
| Unseen category at scoring time | cat_risk falls back to the training prior |
| Same-second transaction bursts | Tie semantics pinned by the Spark parity test |
| Future data leaking into features | Leakage guard test appends future rows and asserts past features are byte identical |
| Single-class evaluation window | Metrics module raises instead of returning NaN |
- Synthetic data. The generator is adversarial to the models (overlapping segments, camouflage, adaptation) but real fraud is stranger; absolute metrics would be lower in production. Relative comparisons, model gaps, threshold economics, and the drift mechanics are what transfer.
- Dollar costs are simplified: review cost is constant, fraud loss equals transaction amount, and chargeback fees, recovery rates, and customer attrition from false declines are out of scope. The optimizer's structure accepts any refined cost model.
- A single global threshold. The cost asymmetry by amount argues for amount-conditional thresholds or expected-loss ranking; the analyst queue mart is the first step in that direction.
- The drift response is detection and a red DAG run, not automated retraining. Retraining on a drifted window without labeled feedback would be guessing.
src/frauddet/
data/generator.py seeded transactions, three fraud archetypes, drift window
features/build.py strictly past velocity, category risk, temporal split
spark/features_job.py same features in PySpark, parity tested
models/train.py logreg, XGBoost, LightGBM, PR-AUC early stopping
models/threshold.py the dollar cost optimizer and review-cost sweep
monitoring/drift.py PSI from scratch plus the flag rate canary
reporting/export.py Power BI star schema CSVs
pipeline.py end to end run, writes eval_report.json
airflow_home/dags/ the DAG with the failing drift gate
sql/ddl, sql/marts warehouse schema and dashboard queries
tests/ 26 tests including leakage guard and Spark parity
benchmark/ measured throughput, results committed
docs/adr/ 0001 dollars over F1, 0002 PSI from scratch