A scalable Big Data Analytics project that performs large-scale text classification and sentiment analysis on Amazon customer reviews using Apache Spark, Spark MLlib, Spark SQL, and Apache Hive. The project demonstrates distributed data processing, machine learning, and real-time sentiment prediction on a multi-gigabyte dataset containing millions of customer reviews.
With the rapid growth of e-commerce platforms, millions of customer reviews are generated every day. Processing such massive textual data using traditional machine learning techniques becomes computationally expensive and inefficient.
This project leverages Apache Spark's distributed computing capabilities to efficiently process, analyze, and classify customer reviews into Positive and Negative sentiments. The system incorporates text preprocessing, TF-IDF feature extraction, Logistic Regression classification, Spark SQL analytics, Hive integration, and a real-time prediction module.
- Multi-GB Amazon Review Dataset
- Distributed Data Processing using Apache Spark
- Text Preprocessing
- TF-IDF Feature Extraction
- Logistic Regression Classification
- Spark SQL Analytics
- Apache Hive Integration
- Real-Time Sentiment Prediction
- Performance Evaluation using Accuracy, Precision, Recall, and F1-Score
- Scalable Big Data Architecture
- Python
- Apache Spark
- Spark MLlib
- Spark SQL
- Apache Hive
- Google Colab
- Pandas
- Matplotlib
The project utilizes multiple Amazon Review datasets merged into a single large-scale dataset.
- Electronics
- Books
- Movies & TV
- Sports & Outdoors
- Home & Kitchen
| Category | Records |
|---|---|
| Electronics | 1,689,188 |
| Books | 8,898,041 |
| Movies & TV | 1,697,533 |
| Sports & Outdoors | 1,296,785 |
| Home & Kitchen | 5,518,682 |
| Total | 19,100,229+ Reviews |
Amazon Review Dataset
│
▼
Data Ingestion (Apache Spark)
│
▼
Text Preprocessing
(Tokenization & Stopword Removal)
│
▼
TF-IDF Feature Extraction
│
▼
Logistic Regression Model
│
▼
Sentiment Classification
│
▼
Spark SQL Analytics
│
▼
Hive Storage
│
▼
Real-Time Prediction
- Load multi-GB Amazon review datasets.
- Merge multiple product categories.
- Store data using Spark DataFrames.
- Remove null values.
- Tokenize review text.
- Remove stop words.
- Apply TF-IDF.
- Convert text into numerical feature vectors.
- Train Logistic Regression using Spark MLlib.
- Perform binary sentiment classification.
- Execute distributed SQL queries using Spark SQL.
- Analyze review statistics and prediction distributions.
- Accept new review text.
- Predict sentiment instantly.
The following analytical queries were performed:
- Rating Distribution
- Sentiment Distribution
- Average Rating
- Prediction Count
Example:
SELECT prediction, COUNT(*)
FROM reviews
GROUP BY prediction;The model was evaluated using:
- Accuracy
- Precision
- Recall
- F1-Score
Compared to traditional machine learning techniques, Apache Spark provides:
- Distributed Processing
- In-Memory Computation
- Faster Execution
- Scalability
- Fault Tolerance
- Efficient Processing of Millions of Records
- Integrated Machine Learning Library
- Built-in SQL Analytics
Large-Text-Classification/
│
├── Dataset/
├── notebooks/
├── models/
├── outputs/
├── README.md
└── requirements.txt
- Transformer-based models (BERT, RoBERTa)
- Apache Kafka Streaming
- Spark Structured Streaming
- Multi-class Sentiment Classification
- Multilingual Text Classification
- Cloud Deployment (AWS/Azure/GCP)
- Interactive Dashboards (Power BI/Tableau)
- Apache Spark Documentation
- Spark MLlib Documentation
- Spark SQL Documentation
- Amazon Review Dataset
- Spark NLP Documentation
Developed as a Big Data Analytics project demonstrating large-scale text classification using Apache Spark and distributed machine learning.