CERT-to-LANL Zero-Shot Transfer

Evaluating the Zero-Shot Transferability of Capacity-Constrained Models for Insider-Threat Detection in Authentication Logs

This repository contains the code used to reproduce a strict zero-shot transfer experiment from synthetic CERT r5.2 authentication logs to the LANL Cyber-1 authentication dataset (human-only accounts). Two capacity-constrained model classes are evaluated:

• Logistic Regression (LR)
• Gradient-Boosted Decision Trees (GBT, shallow)

The repo includes:

• Reproducible feature extraction (user-day aggregation)
• Training on CERT only
• Zero-shot inference on LANL (no adaptation)
• Central drift + failure analysis (PSI, score behavior, loudness bias, top-k overlap)
• Dataset “data understanding” utilities and plotting

Datasets are not included. See DATASETS.md for acquisition and expected folder layout.

Repository layout

repo/
README.md
DATASETS.md
requirements.txt
python_scripts/
  features_scripts/
    build_cert_features.py
    verify_cert_features.py
    verify_lanl_features.py
  dataset_scripts/
    data_understanding_stats.py
  analysis_scripts/
    analyze_zero_shot_results.py
  model_scripts/
    train_cert_gbt.py
    train_cert_lr.py
    zero_shot_lanl_gbt.py
    zero_shot_lanl_lr.py
featuresdata/      # generated, normally not committed
modelData/         # generated, normally not committed
analysisData/      # generated, normally not committed

Reproduction workflow

1. Build CERT features
   • python_scripts/features_scripts/build_cert_features.py --dataset CERT
2. Verify CERT features
   • python_scripts/features_scripts/verify_cert_features.py
3. Build LANL features
   • python_scripts/features_scripts/build_cert_features.py --dataset LANL
4. Verify LANL features
   • python_scripts/features_scripts/verify_lanl_features.py
5. Train LR on CERT
   • python_scripts/model_scripts/train_cert_lr.py
6. Train shallow GBT on CERT
   • python_scripts/model_scripts/train_cert_gbt.py
7. LR zero-shot on LANL
   • python_scripts/model_scripts/zero_shot_lanl_lr.py
8. GBT zero-shot on LANL
   • python_scripts/model_scripts/zero_shot_lanl_gbt.py
9. Central analysis/report/figures
   • python_scripts/analysis_scripts/analyze_zero_shot_results.py

Smoke tests (safe, partial run)

Before full-scale runs, validate the environment with tiny samples:

source .venv/bin/activate

# CERT partial feature build (safe)
python python_scripts/features_scripts/build_cert_features.py \
  --dataset CERT \
  --data-path datasets/r5.2/logon.csv \
  --labels-path datasets/r5.2/answers/insiders.csv \
  --out-dir featuresdata/smoke_cert \
  --nrows 10000 \
  --n-jobs 1

python python_scripts/features_scripts/verify_cert_features.py \
  --file featuresdata/smoke_cert/features_CERT_ZeroShot.csv

# LANL partial feature build (safe upper bound)
python python_scripts/features_scripts/build_cert_features.py \
  --dataset LANL \
  --data-path datasets/lanl/auth.txt \
  --labels-path datasets/lanl/redteam.txt \
  --out-dir featuresdata/smoke_lanl \
  --nrows 100000 \
  --n-jobs 1

# LANL full run requires explicit confirmation (guarded in CLI)
python python_scripts/features_scripts/build_cert_features.py \
  --dataset LANL \
  --data-path datasets/lanl/auth.txt \
  --labels-path datasets/lanl/redteam.txt \
  --out-dir featuresdata/lanl \
  --allow-full-lanl \
  --n-jobs 1

python python_scripts/features_scripts/verify_lanl_features.py \
  --file featuresdata/smoke_lanl/features_LANL_ZeroShot.csv

Warnings:

• datasets/lanl/auth.txt is ~69G. Do not run full LANL feature extraction on a RAM-limited machine.
• Run only --nrows-bounded smoke tests first, then controlled scale-up (--nrows increase, or chunked/restricted workflow).
• Keep --n-jobs 1 or 2 until stability is verified.
• For LANL full runs, pass --allow-full-lanl explicitly. Without --nrows, the script is designed to stop to prevent accidental full reads.
• No full model training/zero-shot/analysis should be run in this phase.
• For RAM-limited environments, a chunked/partitioned LANL feature-builder is recommended for future scale.

Generated outputs (expected naming)

These files are created by the scripts and are not committed:

• featuresdata/r5.2/features_CERT_ZeroShot.csv
• featuresdata/lanl/features_LANL_ZeroShot.csv
• modelData/model_lr_final.pkl
• modelData/model_gbt_final.pkl
• analysisData/results_zeroshot/
• analysisData/analysis_report/

Notes

• Existing scripts currently use robust defaults for the paths above.
• The repository focuses on reproducibility and hygiene; generated artifacts are intentionally kept out of version control.