Adaptive Contextual Bandit Recommender (Online Adaptation)

Purpose

This project builds a recommender system that adapts online to changing user preferences and catalog dynamics. The primary goal is to demonstrate that adaptive policies outperform static baselines under drift, using reproducible offline evaluation and a Streamlit demo that communicates the results clearly.

If you want a portfolio-grade example of adaptive AI systems, this repo is designed to show:

• How online learning is implemented (context -> action -> reward -> update).
• Why static models break under distribution shift.
• How to evaluate adaptivity with credible offline metrics (OPE + diagnostics).

---

Project Summary (TL;DR)

• Data: MovieLens 100K
• Approach: Contextual bandits with online updates
• Adaptive policies: LinUCB, Discounted LinUCB, LinTS, Reset LinUCB
• Static baselines: Random, Popularity
• Key result: Adaptive policies recover fast after drift, while static popularity degrades.
• Outputs: Plots, metrics CSVs, and an interactive Streamlit demo.

---

Methodology (Detailed)

1) Data + Event Stream

• MovieLens ratings are converted into time-ordered events.
• Each event includes:
  • user context
  • candidate items
  • logged action
  • binary reward (rating >= 4 => 1)
• Candidates = 1 logged item + sampled negatives (popular + random).

2) Context Features

Lightweight, online-friendly features:

• age_norm
• gender_bin
• occupation_id
• hour
• dow

3) Adaptive Policies

• LinUCB: upper-confidence bound exploration
• Discounted LinUCB: exponential forgetting for drift
• LinTS: Thompson sampling for exploration
• Reset LinUCB: change-point detection (Page-Hinkley) + reset

4) Static Baselines

• Random: uniform over candidates
• Popularity: always pick most globally popular

5) Simulation Environments

• Logged replay: conservative offline replay (reward known only if matches logged item).
• Synthetic environment: reward model generates feedback for any action.
• Drifted environment: reward model changes at step 60k, and popular items are penalized post-drift to stress-test static strategies.

6) Evaluation

• Online metrics: average reward, cumulative reward, expected regret
• Drift metrics: post-drift reward + recovery time
• OPE: IPS / SNIPS / DR / SWITCH-DR with bootstrap CIs
• OPE diagnostics: ESS + importance weight tail statistics

---

Technology and Packages

• Python 3
• numpy, pandas, scikit-learn
• matplotlib, seaborn
• pyarrow
• streamlit
• tqdm, pyyaml, joblib

---

Software Architecture (Visual)

flowchart LR
    A[MovieLens Raw] --> B[Event Builder]
    B --> C[events.parquet]

    C --> D[Replay Simulator]
    D --> D1[replay_metrics.csv]
    D1 --> D2[replay plots]

    C --> E[Synthetic Logger]
    E --> F[logged_synth.parquet]
    F --> G[OPE Evaluator]
    G --> G1[ope_results.csv]
    G --> G2[ope_diagnostics.csv]
    G1 --> G3[OPE plots]

    C --> H[Simulated Env]
    H --> H1[sim_policy_metrics.csv]
    H --> H2[sim_policy_drift_metrics.csv]
    H1 --> H3[sim plots]
    H2 --> H4[drift plots]

    D2 --> S[Streamlit Demo]
    G3 --> S
    H3 --> S
    H4 --> S

Loading

---

Experiments + Results (Visual, In Depth)

1) Logged Replay (Offline, Conservative)

Goal: Compare adaptive vs static in the same logged environment.

Artifacts:

• data/processed/replay_metrics.csv
• data/processed/baseline_metrics.csv

Plots:

• Reward curves: data/processed/baseline_vs_linucb_reward.png
• Match rate curves: data/processed/baseline_vs_linucb_match.png

Interpretation:

• Popularity often looks strongest in replay because the logs are biased toward popular items.
• This is why we add synthetic simulation + OPE to evaluate true adaptivity.

---

2) Stationary Simulation (Ground-Truth Reward Model)

Goal: Compare adaptive strategies when full reward feedback is available.

Artifacts:

• data/processed/sim_policy_metrics.csv
• data/processed/sim_policy_reward.png
• data/processed/sim_policy_regret.png

Summary Table (stationary environment):

Policy	Avg Reward (All)	Avg Expected Regret (All)
popularity	0.672	0.148
linucb	0.622	0.199
d_linucb	0.563	0.256
lin_ts	0.540	0.279
random	0.509	0.309

Interpretation:

• In a stationary environment, popularity can still look strong.
• Adaptive policies reduce regret over time (especially LinUCB).

---

3) Drift Simulation (Adaptation Under Distribution Shift)

Goal: Show adaptive recovery when the environment changes at step 60k.

Artifacts:

• data/processed/sim_policy_drift_metrics.csv
• data/processed/sim_policy_drift_summary.csv

Plots:

• data/processed/sim_policy_drift_reward.png
• data/processed/sim_policy_drift_recovery.png

Pre-Drift vs Post-Drift Average Reward (Recent Window)

Policy	Pre-Drift	Post-Drift
linucb	0.645	0.694
lin_ts	0.553	0.565
d_linucb	0.573	0.546
reset_linucb	0.542	0.504
random	0.513	0.481
popularity	0.677	0.480

Interpretation:

• Popularity drops hard after drift (static strategy breaks when the environment shifts).
• Adaptive methods maintain or recover stronger performance, with LinUCB leading.

Recovery Time (steps to regain 80% of pre-drift performance)

Policy	Recovery Steps
linucb	6,000
d_linucb	6,000
lin_ts	6,000
reset_linucb	6,000
random	6,000
popularity	26,000

Interpretation:

• Static popularity takes much longer to recover.
• Adaptive methods recover quickly, supporting the adaptation thesis.

---

4) Off-Policy Evaluation (OPE)

Goal: Provide credible offline evaluation with known propensities.

Artifacts:

• data/processed/ope_results.csv
• data/processed/ope_diagnostics.csv

Plots:

• data/processed/ope_results.png
• data/processed/ope_diagnostics.png

DR Estimator (with 95% CI, Oracle in parentheses)

Policy	DR Value	CI Low	CI High	Oracle
random	0.513	0.500	0.524	0.503
popularity	0.803	0.796	0.812	0.806
model	0.664	0.642	0.680	0.668

Interpretation:

• DR aligns closely with oracle values, increasing confidence in evaluation.
• Diagnostics show weight tails and ESS for stability checks.

---

5) Dashboard (Portfolio Summary)

Goal: One visual that communicates the full story.

---

How to Run (Full Pipeline)

Build events

python -m scripts.00_download_movielens python -m scripts.01_build_bandit_event

Logged replay

python -m scripts.02_run_online_replay python -m scripts.03_train_static_baseline python -m scripts.05_plot_replay_metrics python -m scripts.06_plot_baseline_vs_linucb

Drift replay (logged data)

python -m scripts.07_run_drift_replay python -m scripts.08_plot_drift python -m scripts.09_plot_drift_post_only python -m scripts.10_run_drift_baselines python -m scripts.11_plot_drift_vs_baselines

Synthetic logging + OPE

python -m scripts.12_build_synthetic_logged_data python -m scripts.14_evaluate_ope python -m scripts.15_plot_ope_results python -m scripts.19_plot_ope_diagnostics

Simulated adaptive policies

python -m scripts.13_run_simulated_policies python -m scripts.16_plot_simulated_policies

Drifted simulation + dashboard

python -m scripts.17_run_simulated_policies_drift python -m scripts.18_plot_simulated_drift python -m scripts.20_plot_dashboard

---

Streamlit Demo

Launch after generating plots/CSVs:

streamlit run app/streamlit_app.py

Tabs:

• Dashboard (summary)
• Online Adaptation (Simulated)
• Offline Replay (Logged)
• OPE & Diagnostics

---

Conclusion

This project demonstrates a full adaptive recommender workflow: online learning, drift resilience, and credible offline evaluation. The experiments show that adaptive contextual bandits outperform static baselines after distribution shifts, with faster recovery and higher post-drift reward. The Streamlit demo packages the entire narrative into a clear, portfolio-ready story of adaptable AI systems.