IMDB Sentiment - LOO Data Attribution

Which individual training examples actually help (or hurt) a sentiment classifier? This project answers that with exact Leave-One-Out (LOO) data attribution: retraining a logistic regression model once per training point, removed, to measure its true causal effect on test performance.

Overview

Most feature-importance tools tell you which words mattered. This project asks a different question - which training examples mattered. For each of 800 training reviews, the model is retrained without that review, and the resulting shift in test log loss becomes that review's influence score. Positive influence means the review helped the model; negative influence means the model does better without it - often a sign of a mislabeled or ambiguous example.

Method

1. Data -> IMDB Dataset of 50K Movie Reviews (Kaggle), deduplicated from 50,000 → 49,582 rows.
2. Sampling -> a stratified random sample of 1,000 reviews, since exact LOO requires one full retrain per training point and doesn't scale to the full dataset.
3. Cleaning -> light-touch only: strip HTML artifacts (<br />) and URLs, collapse whitespace. Punctuation, casing, and contractions are deliberately preserved, since the embedding model below uses them as genuine semantic signal.
4. Featurization -> sentence embeddings via all-MiniLM-L6-v2 (384-dim), chosen over TF-IDF because it captures context and negation, and its low dimensionality keeps hundreds of logistic regression refits fast.
5. Model -> logistic regression, 80/20 train/test split (800 / 200), stratified.
6. Attribution -> exact LOO: for each of the 800 training points, remove it, refit logistic regression from scratch, and record the change in test log loss vs. the full-training baseline.

Results

• Baseline test accuracy: 0.8000
• Baseline test log loss: 0.4594
• LOO retrains: 800 (one per training point)

Influence scores cluster tightly around zero (roughly ±0.001), which is expected - removing 1 of 800 points rarely shifts a linear decision boundary by much. The extremes are still informative: the most harmful points (whose removal improved test loss) tend to be reviews with mixed, sarcastic, or ambiguous sentiment relative to their assigned label, while the most helpful points tend to be clear-cut, strongly worded reviews that reinforce the decision boundary.

Full ranked results are saved to loo_results.csv when the notebook is run.

Repo Structure

.
├── imdb_sentiment_loo_data_attribution.ipynb   # full pipeline: EDA → cleaning → embeddings → LOO
├── requirements.txt
├── README.md

The raw dataset and all generated artifacts (sample_imdb_datatset.csv, embeddings.npy, sentiments.csv, loo_results.csv) are not committed - they're either downloaded or regenerated by running the notebook.

How to Run

1. Download IMDB Dataset.csv from Kaggle and place it in the repo root.
2. Install dependencies:

   pip install -r requirements.txt

3. Run the notebook top to bottom (Restart Kernel & Run All):

   jupyter notebook imdb_sentiment_loo_data_attribution.ipynb