-
Notifications
You must be signed in to change notification settings - Fork 0
05. Modelling
Develop customer segmentation models based on purchasing behavior.
- Machine learning models
- Dimension reduction, comparing performance between
- Principal Component Analysis (PCA) (Scikit-learn developers, n.d.-b) and
- T-distributed Stochastic Neighbor Embedding (t-SNE) (Scikit-learn developers, n.d.-a) followed by
- Clustering algorithms, comparing performance between
- Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN) (Scikit-learn developers, n.d.-d) and
- K-means Clustering (Scikit-learn developers, n.d.-c)
- Dimension reduction, comparing performance between
- Human knowledge-based model
- Behavioral Segmentation of customers based on their actions and spending habits (Tarver, 2024)
For the Machine Learning Model code, you may refer to Customer Clustering Code
For the Human knowledge-based model code, you may refer to Final Customer Segmentation Model
In selecting a model for customer segmentation based on purchasing behavior, interpretability was a key consideration. The goal was to ensure that the customer segments were not only data-driven but also intuitive and actionable within a business context.
Despite the advanced clustering achieved with machine learning models, the optimal clustering parameter 𝑘 = 64 suggested a large number of segments. Such a high number of clusters posed challenges for business interpretability, as it would be difficult to discern and apply insights from so many distinct customer groups effectively.
Thus, I opted for the Human Knowledge-Based Model. This model allowed for more meaningful segmentation that aligned with real-world business practices, making the segments easier to interpret and more directly applicable to strategic decision-making.
To segment customers based on their actions and spending habits, I calculated various features from our database, summarizing purchasing behavior at an individual level. These features provided insights into each customer's frequency, spending patterns, and product preferences, forming the basis for intuitive, human knowledge-based customer segments.
The following columns were calculated for each customer to capture key aspects of their behavior:
-
ordered: Indicates order history:
-
0: No orders made. -
1: Orders made on only one day. -
2: Orders made on multiple days.
-
-
total_order: Total number of orders a customer has ever made.
-
ave_monthly_orders: Average number of orders per month, calculated between the earliest and latest orders for customers with multiple order days. Set to
0for customers with fewer than two order days. -
days_last_order: Days since the customer’s last order to the final date in the dataset. Marked as
NAif no orders were made. -
only_promo_order: Binary indicator:
-
1: Customer only ordered during promotional periods. -
0: Otherwise.
-
-
total spend: Total amount a customer has spent across all orders.
-
ave_monthly_spending: Average monthly spending between the earliest and latest order dates for customers with multiple order days, otherwise
0. -
most_ordered_cat: The product category that the customer ordered the most.
-
moc_ratio = (number of orders a customer made for their most ordered category) / (total orders the customer made)
Based on the calculated features, I labeled each customer into one of six segments. These segments were derived using percentile-based thresholds (with the top 25% defined as "high" and bottom 25% as "low") established during exploratory data analysis (EDA). The feature distributions used in this labelling are documented in the Appendices.
-
Discount Seekers:
- Identified by having
only_promo_order = 1or high values in CouponUsed (≥ 2) or Cashback Amount (≥ 196.06).
- Identified by having
-
Loyal High-Spenders:
- Characterized by high total spend (≥ 61.78), high ave_monthly_orders (≥ 0.45), and long tenure (≥ 13 months).
-
Occasional Shoppers:
- Defined by low ave_monthly_orders (≤ 0.15) and low HourSpendOnApp (≤ 2).
-
Tech-Savvy Users:
- Spend a high number of hours on the app (HourSpendOnApp ≥ 4) and have multiple registered devices (NumberOfDeviceRegistered ≥ 4).
-
Single-Category Shoppers:
- Show a strong preference for a single product category, indicated by a high moc_ratio (≥ 1).
-
Long-Tenured Non-Buyers:
- Customers who have not made any orders (ordered = 0) and have a long tenure (≥ 13 months).
Following the segmentation, I calculated the number of customers in each of the six segments to understand the distribution across these identified customer types.
Since this is a human knowledge-based model, traditional performance metrics (such as accuracy or clustering scores) do not apply. Instead, the model’s effectiveness is evaluated based on its interpretability and utility for strategic business insights. The segmentation was designed to align closely with real-world customer behaviors, allowing for intuitive and actionable insights.
Key interpretative metrics for this model includes:
- Segment Size: By examining the number of customers within each segment, stakeholders gain a clear understanding of customer distribution, enabling them to focus efforts on larger segments.
- Strategic Value: The different behaviours of each segment allows the business to identify specific customer needs and tailor engagement efforts accordingly.
- Logistic Regression
- Decision Tree
- Random Forest
Since objective is to predict and identify customers likely to churn (binary classification) and churn values were imbalanced, best model was selected based on its F1 score, ensuring that both false positives and false negatives are minimized.
Random Forest model had the highest F1 score among the 3 models.
After running train_test_split on Customer Dataset, RandomizedSearchCV was then ran with Random Forest model on training set to find the best hyperparameters, which are:
-
n_estimators: 300 -
min_samples_split: 2 -
min_samples_leaf: 1 -
max_depth: 10 -
bootstrap: True
-
Test Accuracy: 86.83%. While accuracy provides an overall measure of the model's performance, it may not fully represent its effectiveness in predicting the minority class (churned customers) due to the class imbalance in our dataset.
-
Classification Report:
-
Precision for class
1(churn) is 0.73, indicating that 73% of the customers the model predicted as churned were indeed correct. -
Recall for class
1is 0.63, meaning the model successfully identified 63% of actual churned customers. -
F1-Score for class
1is 0.68, balancing both precision and recall, making it a useful metric given the importance of accurately identifying churned customers.
-
Precision for class
-
Feature importance analysis:
- Highlights the key drivers of churn: features with the highest importance are
Tenure,Complain,CashbackAmount, andNumberOfDeviceRegistered, which aligns well with trends observed in the correlation matrix. - Unexpectedly,
CouponUsedandHourSpendOnAppalso emerged as significant predictors, whileSatisfactionScore, contrary to initial assumptions, had relatively low importance.
- Highlights the key drivers of churn: features with the highest importance are
-
Further Analysis with SHAP values and Partial Dependence Plots (PDP):
- Confirm observed patterns that customers with tenure under 2 years, who make complaints, and who receive limited cashback are more likely to churn.
- Additional insights revealed that churn-prone customers often exhibit:
- A high number of registered devices.
- Moderate app engagement (approximately 2 hours/day).
- Average satisfaction scores around 3, suggesting neither strong satisfaction nor dissatisfaction.
- Seasonal ARIMA
- Linear programming
- Stock level tracking
To manage inventory efficiently, we aimed to balance cost minimization with consistent product availability. Our approach used a combination of time series forecasting and linear programming to predict demand and set optimal stock levels, while also implementing stock level tracking for ongoing monitoring.
Our SARIMA model was optimized with grid search, resulting in the following parameters:
- order (p, d, q): (1, 1, 1), capturing the autoregressive, differencing, and moving average components of demand,
- seasonal_order (P, D, Q, s): (1, 1, 1, 12), reflecting a 12-month seasonal period.
Using demand predictions from SARIMA, we optimized inventory levels through linear programming, minimizing total inventory costs through considering holding costs and backorder costs. Constraints included that inventory levels should not fall below a certain threshold (safety stock), and should not exceed storage capacity limits.
SARIMA-based demand forecasting forms a basis for demand prediction and guide inventory levels.
Linear programming reduced holding and backorder costs, providing an optimized balance that minimizes total cost while meeting demand.
Monitoring automatically flags restocking needs when inventory falls below a 20% threshold, ensuring dynamic reorder points (ROP).
These inventory optimization strategies are demand-responsive, minimizing stockouts and overstock.
In order to maximize revenue, we developed a dynamic pricing model which takes into account demand indicators (e.g., sales trends), competitor's prices and inventory levels.
- XGBRegressor
- RandomForestRegressor
- GradientBoostingRegressor
- LinearRegression
- SupportVectorRegressor
- We chose our evaluation metric to be Root Mean Squared Error (RMSE). This is because RMSE provides a straightforward interpretation in the same scale as our target variable (Price). Moreover, RMSE penalises large errors more heavily, allowing the model to focus on minimising large deviations between our price predictions and the actual prices.
- The model must be efficient on large datasets for timely computation of price predictions.
- SupportVectorRegressor was too computationally expensive on large datasets.
- We chose GradientBoostingRegressor because it had the best RMSE score out of all the other regression models considered.
- It is also reasonably efficient on large datasets after fine-tuning.
- Tuned hyperparameters: ('learning_rate', 0.1), ('max_depth', 5), ('n_estimators', 300), ('subsample', 0.8)
- RMSE (GradientBoostingregressor): approximately 11.4
- An RMSE of 11.4 indicates that on average, our model's predicted prices is $11.40 off from the actual prices. For lower prices items (e.g., clothing), this could be a substantial error if the prices are less than $40. However, for higher prices items (e.g., Electronic, Home Appliances) which usually costs a few hundred dollars, our model provides a reasonably accurate price prediction.
- Random Forest Classifier
- Logistic Regression
- Gradient Boosting
To enhance supply chain efficiency by identifying bottlenecks and predicting delays, we required a model that provided robust interpretability and worked well with imbalanced data. Interpretability was especially important, as the insights needed to be actionable and understandable to stakeholders.
Given the class imbalance between delayed and on-time orders, we evaluated models primarily on their F1 score, aiming to balance precision and recall. This focus ensured that the model would effectively identify delays without excessive false positives, which could lead to unnecessary interventions.
We ultimately selected Logistic Regression due to its simplicity, interpretability, and reliable performance with binary classification. The model’s coefficients allowed us to clearly identify influential factors, such as scheduled shipping days and delivery risk, providing a straightforward foundation for targeted interventions in the order fulfillment process. This choice aligns with business requirements by offering clear insights into specific factors driving delays.
Our chosen model, Logistic Regression, underwent hyperparameter tuning using grid search, resulting in an optimal configuration with the following parameters:
- C: 0.01 (indicating stronger regularization),
- max_iter: 1000,
- penalty: 'l2', and
- solver: 'lbfgs'.
This configuration achieved a cross-validation accuracy of 97.56%, indicating strong performance across various subsets of the data.
Using this optimized logistic regression model on the test set, we achieved an accuracy of 97.51%, demonstrating the model’s reliability in classifying orders as delayed or on time. Additional performance metrics include:
- Precision for On-Time Orders (Class 0): 1.00, indicating that nearly all orders predicted as on-time were indeed delivered on time.
- Recall for Delayed Orders (Class 1): 1.00, reflecting the model’s ability to identify nearly all actual delays, essential for minimizing overlooked bottlenecks.
- F1-Score for Delayed Orders (Class 1): 0.98, providing a balanced measure of precision and recall, crucial in accurately identifying delays without excessive false positives.
The high precision and recall values across both classes make this logistic regression model well-suited for actionable insights. The model accurately flags high-risk, delayed orders, guiding interventions such as prioritizing certain orders, adjusting schedules, and improving supplier coordination. Additionally, the interpretability of logistic regression allows us to understand the specific features driving delay predictions, aligning with our goal of optimizing the supply chain by targeting key bottlenecks.
This optimized model balances predictive power with interpretability, ensuring that results are both accurate and easily communicated to stakeholders, ultimately supporting strategic improvements in supply chain efficiency.
1. What is the potential of using natural language processing (NLP) to analyze customer reviews and feedback?
Below are pre-trained models we considered:
- Latent Dirichlet Allocation (LDA)
- Valence Aware Dictionary and Sentiment Reasoner (VADER)
- Named Entity Recognition (NER) using SpaCy
- BERT-based Sentiment Analysis
For this question, there is no restriction on which NLP algorithm we can use to analyze customer review data. Ultimately, we can pick any algorithm to simply implement sentiment analysis on these reviews and extract common issues and suggestions. The explanations below state our rationale for choosing one model over another, just because we felt that the model meets our criteria for carrying out the task as quickly and accurately as possible. Note that other models can also be used, and our selection of these models is subjective.
- Initially, we considered using LDA to identify common issues, but switched to NER with SpaCy, because the latter is a more popular use case for extracting entities in customer reviews/feedback compared to LDA. Furthermore, SpaCy’s NER model is a high-quality pre-trained model which is easier to use and can quickly generate almost the same set of results, if not better than LDA, with less code written.
- For sentiment analysis of customer reviews, we initially went with the BERT transformer model, which is robust in many language tasks. However, VADER is a common rule-based sentiment analysis tool that is used frequently in customer review sentiment scoring. Like the previous explanation, VADER can perform the same task as BERT and is easier to use, generating close to a similar set of results.
Comparing the outputs using different models, it is certain their accuracies are somewhat close to each other. Difficulty in interpreting code demonstrating model implementation is another factor we used for model selection. A model that uses more simplified code and easier to interpret is preferred to one that uses more complex code for implementation.
Hence, we went with the VADER model for implementing sentiment analysis on customer reviews and SpaCy’s NER model for identifying common issues and suggestions.
It is impossible to find a test data set from the customer reviews table that fully accurately classifies each review as “negative”, “neutral”, or “positive”. Traditional performance metrics such as accuracy cannot be used here to test which model is most accurate to answer our business question. We can test the accuracies of our models on test data sets outside this project, but it is out of the scope of this project. A model’s performance can be put to the test by comparing its output to another model’s, or by subjectively comparing with the data set (that is, you assign sentiment labels to each review and compare with the model’s output).
2. What are the key insights about product features, quality, and customer satisfaction from customer review analysis, using an LLM?
Below are some pre-trained models we considered:
- RoBERTa, a variant of BERT transformer model
- Another variant of BERT transformer model -- "yangheng/deberta-v3-large-absa-v1.1"
Our objective is to implement aspect-based sentiment analysis. We initially went with the model "yangheng/deberta-v3-large-absa-v1.1”.
We state our rationale for choosing to implement our LLM system in this way.
-
Selection of review column: First, we decided between two review columns: the first
review_title, a short review summary of customers’ reviews and the secondreview_content, the whole customer review.review_contentcontains entries that are up to a paragraph long or more, and considering the time constraint we have on our demo video, it overshot the time limit to achieve an output had we run our LLM system onreview_contententries. Furthermore, somereview_contententries may contain irrelevant information not needed for answering our business questions. That is why we chosereview_title, as a summary of the review is sufficient. -
Selection of model: It is because of the nature of the customer reviews in
review_title– many adjectives were used repeatedly in the same review, and many reviews were incoherent with little emphasis on key aspects, which was what our business objectives were focusing on. Note thatreview_contentwas not any better. After reducing noise within entries inreview_title, we decided a surefire way to implement ABSA was to use the transformer model RoBERTa and use regex searching to discard reviews that did not mention about aspects and label their sentiment as “N/A” instead. We decided that customer satisfaction is captured within the overall sentiment of the review, and there was no need to implement regex searching for that aspect.
We can only test these pre-trained models on trained data sets outside the scope of this project and compare with the test data sets. We will not cover traditional performance metrics here, as there was no building of a model nor was there testing of a model.