feat(customer-analytics): case 02 — churn without leakage

[ronaldmego]

Refs ronaldmego/site-ronaldmego#64 · builds on data-intelligence-lab#1 (the data model)

The second deliverable of the Customer Intelligence Lab: the first of the five cases, on the data model merged in #1.

What it does

Trains at one observation cutoff and scores six months later — an out-of-time backtest, the way a deployed model is judged. Then it deliberately reruns the same model two dishonest ways.

Reading	AUC	What it measures
Out of time (fit 2025-06, scored 2025-12)	0.690 ± 0.014	Whether it generalises forward
In-time random split	0.709	Whether it generalises to unseen customers, in the period it was fitted on
With one post-outcome feature	1.000	Nothing

The third row is the point: one field derived from the label, nothing errors, no test fails, and the model is worthless. Leakage does not announce itself — it shows up as a result good enough that nobody questions it.

The first two rows are the point in a quieter way, and this is the bit I want reviewed. The gap is +0.020 against a standard error of ±0.014 — inside two standard errors. The report says plainly that this dataset does not establish the random split is optimistic, and argues the case for out-of-time evaluation on methodological grounds instead. It would have been easy to bank the number; it isn't real at this sample size.

What it finds

• The ranking survives the time gap; the calibration does not (slope 0.77). Recalibrating on the training period cannot fix a base rate that moved after the calibrator was fitted. The operational answer is not a better model — it is recalibration against the most recent closed window plus a drift alarm on predicted-vs-actual base rate.
• Re-sorting the same probabilities by expected value instead of risk changes 44% of the contact list and +66% of the profit at the same budget, and peaks at fewer calls. Accuracy is not reported anywhere in the case: at an 11.8% base rate, "nobody churns" scores 88.2%.
• Of 9 designed drivers, 5 recovered with the correct sign, 0 contradicted, 4 too weak to read individually — reported as inconclusive rather than as contradictions, because their marginal effect is smaller than its own standard error.
• retention_offer_taken is inert marginally (−0.045) and strongly protective conditionally (−0.290) — the targeting confound the data model was built to contain. Flagged as a warning, explicitly not read as campaign ROI. That is case 05.

Two commits

1 · data model — a second labelled cutoff. Case 02 needed something the model could not give it. churn_labels_prior emits the same 90-day label 6 months earlier, with the risk proxy re-derived from only the events that had happened by then. Emitted after every other table so the random stream is untouched — the twelve existing tables are byte-for-byte identical, verified against sha256 of a pre-change seed-42 run.

2 · the case itself. Pipeline, tests, report, charts.

How leakage is prevented, not just avoided

• One function drops every fact dated after the cutoff. The test poisons the input tables with facts dated 2099-01-01 and asserts no feature moves.
• The standardiser, collinearity filter and calibrator are bundled with the model, because re-deriving any of them from scoring data is leakage nothing would flag.
• The scoring population excludes customers who churned in the earlier window. They left. No metric catches that error.
• Three near-duplicate features are pruned before the coefficients are read (arpu_last3 ~ monthly_fee, r=0.997). Not for accuracy — unpruned, the model reports that an unresolved escalation reduces churn, the opposite of how the data was built.

Two real defects the new tests caught

Both were in code I had just written, and both produced better-looking numbers:

• ks_statistic stepped through tied scores row by row, so a constant model could score KS 1.0 purely because the input arrived sorted by outcome.
• Equal-count bins split tied predictions across boundaries, so a perfectly calibrated constant model could report ECE 0.36.

Checks

• 40 tests, uvx pytest with nothing installed · uvx ruff check . clean
• Report and charts reproducible byte-for-byte across runs (no timestamps, no RNG)
• outputs/ committed, so the result is visible on GitHub without running anything

Scope note

Standard library only — no pandas/scikit-learn. The estimator is a ridge-penalised logistic regression fitted by IRLS (~60 lines): the right default for a decision that has to be defended to a retention team, it keeps every number auditable, and it means no library version can silently change the result in three years. The hard part of this case was never the estimator.

Distribution (post/video) is Hermes' layer — ronaldmego/site-ronaldmego#65 — with this as the base material.