Skip to content

data.md

Latest commit

 

History

History
258 lines (190 loc) · 11.3 KB

data.md

File metadata and controls

258 lines (190 loc) · 11.3 KB
Raw

Data Sources & Schema

Back to README · See also: Feature Catalog · Training Guide

Data Sources

Source Volume Fraud Rate Format Use
Real org transactions Varies 0.05–0.5% Custom CSV → canonical Primary training signal
Synthetic generator Configurable (1M+) 0.3% (tunable per rail) Parquet Cold start + augmentation
IEEE-CIS (Kaggle) 590K 3.5% CSV (opaque columns) GBM training supplement
UCI San Diego 284K 0.172% CSV (PCA-anonymised) Autoencoder pretraining

IEEE-CIS Harmonisation Rule

IEEE-CIS columns (V1–V339, C1–C14, D1–D15, card1–card6) are opaque — never map to canonical fields. Keep in source namespace (ieee_cis.V1 etc).

Only harmonise: TransactionAmt → amount_usd, TransactionDT → timestamp_utc, isFraud → is_fraud, DeviceType, ProductCD.

Why: Column semantics are unknown (PCA-transformed by Vesta). Mapping V23 → user_account_age_days introduces false semantics and training-serving skew.

Code: src/data/loader.py:load_ieee_cis

Canonical Transaction Schema

All data normalised to TransactionRecord (Pydantic v2). Multi-rail with rail-specific required fields.

class TransactionRecord(BaseModel):
    # ── Identity ──────────────────────────────────────────────────────────────
    transaction_id: str                      # UUIDv4 recommended
    timestamp_utc: datetime                  # always UTC; validator enforces
    data_source: Literal["ieee_cis", "uc_san_diego", "synthetic", "real_org"]
    schema_version: str = "2.0.0"

    # ── Rail ─────────────────────────────────────────────────────────────────
    rail: Literal["upi", "aeps", "dmt", "cards", "loans", "utility"]
    channel: str                             # see channel list in API Reference

    # ── Amount ───────────────────────────────────────────────────────────────
    amount_inr: Decimal                      # in INR, >= 0
    currency_code: str = "INR"              # ISO 4217

    # ── Parties ──────────────────────────────────────────────────────────────
    customer_id_hash: str                   # SHA-256, 64 hex chars
    merchant_id: str
    merchant_country: str = "IN"           # ISO 3166-1 alpha-2

    # ── Card (PCI-DSS — no raw PAN) ──────────────────────────────────────────
    card_token: Optional[str]               # tokenized; never raw PAN
    bin_8: Optional[str]                    # first 8 digits
    pan_last4: Optional[str]               # last 4 only
    card_type: Optional[Literal["debit", "credit", "prepaid", "corporate", "rupay"]]
    cvv_result: Optional[Literal["match", "no_match", "not_provided"]]
    three_ds_result: Optional[Literal["authenticated", "not_authenticated", "not_attempted"]]
    is_prepaid: bool = False

    # ── UPI-specific ─────────────────────────────────────────────────────────
    vpa_sender_hash: Optional[str]          # required for rail=upi
    vpa_receiver_hash: Optional[str]
    upi_app: Optional[str]
    consent_timestamp_utc: Optional[datetime]

    # ── AEPS-specific (Aadhaar Act §29 compliant) ─────────────────────────────
    aadhaar_last4: Optional[str]            # last 4 only; no full Aadhaar
    agent_id: Optional[str]                 # required for rail=aeps
    biometric_quality_score: Optional[float]    # UIDAI quality 0–100; NOT raw biometric
    biometric_attempt_count: Optional[int]
    session_duration_seconds: Optional[float]

    # ── DMT-specific ─────────────────────────────────────────────────────────
    beneficiary_account_hash: Optional[str]
    beneficiary_bank_ifsc: Optional[str]
    beneficiary_is_new: Optional[bool]

    # ── Device / Network ─────────────────────────────────────────────────────
    device_id_hash: Optional[str]           # SHA-256 of device fingerprint
    ip_address_hash: Optional[str]          # SHA-256+salt; never raw IP
    device_firmware_version: Optional[str]
    txn_lat: Optional[float]
    txn_lon: Optional[float]

    # ── Label ────────────────────────────────────────────────────────────────
    is_fraud: Optional[int]                 # 0 or 1; absent for inference
    label_source: Optional[str]             # "confirmed_chargeback" (gold standard)
    label_confirmed_at_utc: Optional[datetime]

Validators

Validator Rule
validate_hash_format customer_id_hash, device_id_hash, ip_address_hash must be 64 or 128 hex chars
validate_rail_fields UPI requires vpa_sender_hash; AEPS requires agent_id; DMT requires beneficiary_account_hash
no_full_aadhaar aadhaar_last4 must be exactly 4 digits; rejects 12-digit input
timestamp_utc Converts naive datetime to UTC; rejects future timestamps > 5 minutes

Code: src/data/schema.py

Synthetic Data Generator

Generates realistic per-rail transaction data with configurable fraud injection.

Usage

python -m fraud_detection.src.data.generator \
  --n_transactions 1_000_000 \
  --output fraud_detection/data/synthetic/txns.parquet \
  --seed 42 \
  --n_customers 50_000

Rail Configuration

Rail Fraud Rate Amount Distribution Fraud Patterns
UPI 0.3% Log-normal μ=7.5, σ=1.2 (INR) Phishing, fake QR, SIM swap
AEPS 0.15% Bimodal (₹100–₹2000 withdrawals) Ghost txn, rogue agent
DMT 0.5% Log-normal, spike at ₹97K–₹99K Structuring, mule
Cards 0.8% Log-normal μ=8.0, σ=1.5 CNP, BIN attack

Temporal Realism

  • Poisson arrivals with IST peak hours (10am–1pm and 5pm–8pm higher rate)
  • Month-end spike (salary day: 1st–3rd and 28th–31st)
  • Weekend suppression for corporate/DMT channels
  • Random seed → fully reproducible

Fraud Injectors

_inject_aeps_ghost()     # session < 10s, quality=100, attempts=1
_inject_dmt_structuring()  # amount in ₹97K–₹99.5K band, same-day repeat
_inject_card_cnp()       # ecommerce, no 3DS, new merchant, foreign BIN

Code: src/data/generator.py

Data Cleaning Pipeline

Applied to all sources before training. Idempotent — safe to run multiple times.

Steps (in order)

Step Function Description
1 dedup Drop exact-duplicate transaction_id rows; keep first
2 normalise_timestamps Convert all timestamps to UTC; drop rows with null timestamp
3 filter_test_transactions Drop transaction_id containing test, mock, sandbox
4 filter_amount_outliers Drop amount = 0; cap at 99.9th percentile per rail
5 filter_label_maturity Drop rows where label_confirmed_at_utc > cutoff - 14d (labels not yet mature)
6 coerce_schema Cast all columns to canonical types; fill missing with rail-appropriate defaults
7 clean Master pipeline: runs steps 1–6 in sequence; returns cleaned Polars DataFrame

Label Maturity Filter

# Exclude last 14 days from training set
cutoff = df["timestamp_utc"].max() - timedelta(days=14)
df = df.filter(pl.col("timestamp_utc") <= cutoff)

Chargebacks arrive up to 45 days after transaction. Labels in the last 14 days have high false-negative rate (fraud not yet confirmed). Including them trains the model to think recent fraud is legitimate.

Code: src/data/cleaner.py

Multi-Source Loader

from fraud_detection.src.data.loader import load_all, combine

dfs = load_all(
    real_org_path="data/raw/org_txns.csv",
    ieee_cis_path="data/external/ieee_cis/",
    uci_path="data/external/uc_san_diego/creditcard.csv",
    synthetic_path="data/synthetic/txns.parquet",
)
combined = combine(dfs)  # unified canonical DataFrame

Source Adapters

Function Reads Normalises
load_synthetic Parquet Already canonical
load_uci_san_diego CSV V1–V28, Amount, Class → canonical amount + label
load_ieee_cis CSV directory TransactionAmt, isFraud → canonical; V*/C*/D* kept as-is
load_real_org CSV Custom columns → canonical via configurable mapping
load_all All sources Returns dict keyed by source name
combine dict Concatenates, adds data_source column

Code: src/data/loader.py

Data Directory Layout

fraud_detection/data/
├── raw/                   # Real org transactions — anonymised before commit
│   └── .gitkeep
├── processed/             # Cleaned, point-in-time feature snapshots
│   └── .gitkeep
├── synthetic/             # Generator output — can be regenerated; not committed
│   └── .gitkeep
└── external/              # Kaggle datasets — download separately
    ├── .gitkeep
    ├── ieee_cis/          # Place Kaggle IEEE-CIS files here
    │   ├── train_transaction.csv
    │   └── train_identity.csv
    └── uc_san_diego/      # Place UCI credit card dataset here
        └── creditcard.csv

All data directories in .gitignore. Data managed outside git (S3 / DVC for large files).

Downloading External Datasets

# IEEE-CIS Fraud Detection (Kaggle)
kaggle competitions download -c ieee-fraud-detection -p fraud_detection/data/external/ieee_cis/

# UCI Credit Card Fraud (Kaggle)
kaggle datasets download -d mlg-ulb/creditcardfraud -p fraud_detection/data/external/uc_san_diego/

Requires kaggle CLI + ~/.kaggle/kaggle.json credentials.

Point-in-Time Feature Snapshots

For training, velocity features are pre-materialised as-of each transaction's timestamp to prevent training-serving skew:

processed/
├── features_train.parquet     # features computed as-of txn timestamp
├── features_val.parquet
├── features_test.parquet
└── feature_schema_v2.json     # feature names + types for version tracking

Why not compute live from raw logs? Computing 14-window velocity for 1M training examples from raw logs takes hours. Pre-materialise once, reuse for all HPO trials.

Next: Feature Catalog · Training Guide