This project implements a complete pipeline for sentiment analysis on movie review data using Python, combining text preprocessing, feature reduction, machine learning modeling, and model interpretability.
The workflow begins by cleaning raw review text to remove HTML tags and standardizing all text to lowercase. A working vocabulary is constructed using CountVectorizer with stop words drawn from both the SpaCy and NLTK libraries. LASSO regression is then applied to the vectorized training data to select only the most predictive features. The reduced feature set is used to construct the final CountVectorizer for model training.
An XGBoost classifier (XGBClassifier) is trained on this reduced feature set across five data splits to ensure robust evaluation. Model predictions are evaluated using ROC-AUC scores, consistently achieving approximately 0.989 on each test split. The trained model is saved as a JSON file (598_PSL_Proj_xgboost.json), and the reduced vocabulary is stored in total_features.txt for future use.
The project also includes explainability features using LIME. Positive and negative reviews are sampled from the test data, and LIME highlights the words most responsible for each model prediction, providing insight into the decision-making process of the XGBoost classifier.
Import_XGBoost_Model.ipynb demonstrates how to import the pre-trained XGBoost model directly from GitHub, load the total feature set, and evaluate performance on test splits. Train_XGBoost_with_Feature_Reduction.ipynb rebuilds the model from scratch by performing text vectorization, feature reduction with LASSO, and model training. Both notebooks include preprocessing, stop word handling, and code to save the final model and features.
Clone this repository and ensure all dependencies are installed, including pandas, numpy, scikit-learn, xgboost, nltk, spacy, regex, and LIME. Open the notebooks in Jupyter and run the cells sequentially. After training or loading the model, the final_vectorizer and xgmodel objects can be used to predict sentiment on new review data.
To explain individual predictions, use the LIME text explainer included in the notebooks. This allows you to visualize the top words contributing to positive or negative predictions and better understand model behavior.
The stop word list consists of over 375 words combined from SpaCy and NLTK. Feature reduction via LASSO prevents overfitting and reduces dimensionality. XGBoost training can be time-intensive, taking up to approximately one minute per fold on standard hardware. LIME explainability is applied only to sampled reviews for interpretability purposes.