shiftweight

Covariate-shift correction by adversarial validation. Train a classifier to tell your TRAINING rows from the TARGET (test / held-out) rows using features only — never y — then weight each training row by how much it looks like the target, so the fit concentrates on the region it will actually be scored on. The weighting is validated on a held-out split against a magnitude-matched null, so a reported gain is real, not memorized. numpy-only.

from shiftweight import held_out
from shiftweight.synth import make_shift_misspecified
X, y, _ = make_shift_misspecified()
r = held_out(X, y)
print(r["uniform"], r["learned"], r["beats_null"])   # 0.916 0.939 True

$ shiftweight demo
## 1. Misspecification + covariate shift (a target-like subpopulation the
##    linear fit under-weights — importance weighting should genuinely help)
  best weight curve : ratio
  held-out transfer : 0.9391   (uniform 0.9156, lift +0.0235)
  null p95 / p      : 0.9249 / 0.000
  VERDICT           : REAL — beats held-out null
  -> mean weight on the most target-like train rows 3.59 vs the rest 0.24

## 2. No-shift control (well-specified linear truth, same distribution —
##    importance weighting must NOT beat the null)
  best weight curve : ratio
  held-out transfer : 0.9902   (uniform 0.9902, lift +0.0000)
  null p95 / p      : 0.9902 / 0.130
  VERDICT           : no gain over null (the honest answer)

The idea

This is Kaggle's "adversarial validation" turned into sample weights. Stack your train and test rows, label them 0/1, and fit a classifier on the features only. If it can separate them, your train and test distributions differ (covariate shift), and a training row's predicted P(target | x) is exactly how much that row resembles the test set. Weight training rows by that propensity and the model stops wasting capacity on regions it will never be scored on.

The propensity is mapped to a mean-1 weight by one of four curves — selection tries all four and keeps whichever transfers best:

Curve	Map from propensity p	Note
ratio	p / (1 − p), clipped	the density-ratio importance weight (principled)
prob	p	a gentler tilt toward target-like rows
rank_high	linear in the rank of p	scale-free; robust to a miscalibrated classifier
sigmoid	soft top-k gate on the rank of p	concentrates weight on the most target-like rows

Weights are renormalized to mean 1 and clamped to [1/clip, clip], so a weighting only moves attention around — it never changes the effective sample size or regularization versus a uniform fit, and no single near-certain row can dominate (the standard fix for the variance that makes naive importance weighting overfit). The fit itself is a sample-weighted ridge solved via the Gram trick, so every refit and every null permutation is cheap.

Honesty is the whole point

Importance-weighting toward the target is not free lunch:

• It helps under model misspecification + shift — when the model can't fit the whole feature space at once, tilting it toward the target region trades error you don't care about for error you do.
• It ties the null when the model is well-specified — a correct model is already right everywhere, so reweighting can only add variance. The honest verdict there is "no gain", and the demo's no-shift control shows exactly that.
• Naive versions overfit: upweighting a thin tail of rows collapses the effective sample size and the fit gets worse. The held-out + null check is what tells you which case you're actually in.

So nothing counts unless it beats a held-out null. held_out:

1. splits the rows by order into low (first 60%, the training rows we have labels for) and high (the target we'll be scored on);
2. fits adversarial-validation propensity of low against high features only (the target's y is never touched) and builds a candidate weight vector for each (mode, curve);
3. selects the best by a low-only 2-fold importance-weighted transfer — train on half of low, score the other half with a propensity-weighted correlation, so the winner is the curve that does best on the target-like sub-population (the only label-free signal of shift). The high rows are never seen during selection;
4. refits on all of low with the chosen weights and scores the untouched high rows with a plain (unweighted, true-label) Pearson — the reported learned;
5. compares against uniform weighting and a magnitude-matched null: the chosen weight vector permuted across the low rows (same weights, structure destroyed), refit, scored on high, n_null times.

beats_null is True only when learned beats both the uniform baseline (learned − uniform > 0.005) and the null's 95th percentile. A null result is reported as "no gain over null (the honest answer)" — not dressed up as a discovery.

API

from shiftweight import propensity, importance_weights, adversarial_auc, held_out

p = propensity(X_train, X_test)               # P(target | x) per training row — features only
w = importance_weights(p, mode="ratio")       # mean-1, clipped sample weights
adversarial_auc(X_train, X_test)              # ~0.5 = no shift, ->1 = strong shift

r = held_out(X, y, low_frac=0.6, n_null=200)  # learn + validate vs a held-out null
r["learned"], r["uniform"], r["lift"], r["beats_null"]
r["best_mode"], r["propensity"], r["weights"] # all inspectable

CLI

shiftweight demo                                    # misspecified+shift niche + a no-shift control
shiftweight fit train.csv --target y                # weight by held-out-tail propensity + null verdict
shiftweight fit train.csv --target y --test test.csv  # weight toward a real (unlabeled) test set

With --test, it reports the adversarial AUC (is there even a shift?), the importance weights, and the most test-like training rows — plus an internal held-out null check on your data, since the real test set has no labels to score.

Install

pip install numpy   # then use the package directly, or `pip install .`

MIT. Depends only on numpy.