An AI-powered product recommendation engine that combines keyword-based (BM25) and semantic (transformer-based) search to improve product discoverability for e-commerce platforms.
Client: ShopEasy - an emerging e-commerce platform
Challenge: Traditional keyword-based search methods fail to capture user intent and context, leading to:
- Irrelevant product listings
- Poor click-through rates
- Lower conversion rates
- Suboptimal customer experience
Example Problem:
- User searches: "comfortable footwear for jogging"
- Keyword search finds: nothing (no exact word matches)
- Semantic search finds: "Nike Running Shoes", "Athletic Sneakers" (understands intent)
This project implements a Hybrid Search System that combines two complementary approaches:
- Matches exact words between query and product descriptions
- Strengths: Product names, model numbers, brand names, SKUs
- Formula: Uses term frequency, inverse document frequency, and document length normalization
- Converts text to numerical vectors capturing meaning
- Uses
all-mpnet-base-v2transformer model (768-dimensional embeddings) - Strengths: Understands synonyms, intent, and related concepts
hybrid_score = (alpha * BM25_score) + (beta * Semantic_score)
- Default weights: BM25=0.2, Semantic=0.8 (optimized through experimentation)
- Gets benefits of both approaches
| Metric | BM25 Only | Semantic Only | Hybrid (Optimized) |
|---|---|---|---|
| Hit Rate @10 | 30.0% | 48.0% | 54.0% |
| MRR Score | 0.162 | 0.270 | 0.316 |
Interpretation:
- Hit Rate: 54% of user queries found the correct product in top-10 results (vs 30% with keywords alone)
- MRR (Mean Reciprocal Rank): Correct products appear higher in results, meaning users find what they want faster
hybrid-search-recommendation/
│
├── README.md # This file
├── requirements.txt # Python dependencies
├── .gitignore # Git ignore rules
│
├── notebooks/
│ ├── Hybrid_Search_Part1_DataPrep.ipynb # Data loading & preprocessing
│ ├── Hybrid_Search_Part2_SearchIndexes.ipynb # Building BM25 & semantic indexes
│ └── Hybrid_Search_Part3_Evaluation.ipynb # Evaluation & visualization
│
├── data/
│ ├── Shopeasy_product_dataset.csv # 20,000 products with descriptions
│ └── Final_purchased_products.csv # 50 query-purchase pairs (ground truth)
│
├── results/
│ ├── hybrid_search_comparison.png # Performance comparison chart
│ └── weight_optimization.png # Weight tuning results
│
└── docs/
└── Business-Problem-overview.txt # Detailed business context
┌─────────────────────────────────────────────────────────────────┐
│ User Query │
│ "comfortable running shoes" │
└─────────────────────────┬───────────────────────────────────────┘
│
┌───────────────┴───────────────┐
│ │
▼ ▼
┌─────────────────────┐ ┌─────────────────────────┐
│ BM25 Pipeline │ │ Semantic Pipeline │
│ │ │ │
│ 1. Tokenize query │ │ 1. Encode with │
│ 2. Lemmatize │ │ SentenceTransformer │
│ 3. Match keywords │ │ 2. Cosine similarity │
│ 4. Score by TF-IDF │ │ 3. Rank by meaning │
└─────────┬───────────┘ └───────────┬─────────────┘
│ │
│ Normalized Scores │
└───────────────┬───────────────┘
│
▼
┌───────────────────────┐
│ Hybrid Fusion │
│ │
│ score = 0.2×BM25 + │
│ 0.8×Semantic │
└───────────┬───────────┘
│
▼
┌───────────────────────┐
│ Top-K Results │
│ (Ranked Products) │
└───────────────────────┘
| Category | Technology | Purpose |
|---|---|---|
| NLP/ML | Sentence Transformers | Text embeddings (all-mpnet-base-v2) |
| Search | rank-bm25 | BM25 probabilistic ranking |
| Text Processing | NLTK | Tokenization, lemmatization, stopwords |
| Similarity | scikit-learn | Cosine similarity computation |
| Data | pandas, numpy | Data manipulation and numerical operations |
| Visualization | matplotlib, seaborn | Performance charts |
- Clone the repository:
git clone https://github.com/virtualryder/hybrid-search-recommendation.git
cd hybrid-search-recommendation- Create virtual environment (recommended):
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate- Install dependencies:
pip install -r requirements.txt- Download NLTK data (automatic in notebooks, or run manually):
import nltk
nltk.download('punkt')
nltk.download('stopwords')
nltk.download('wordnet')
nltk.download('punkt_tab')-
Part 1 - Data Preparation (
Hybrid_Search_Part1_DataPrep.ipynb)- Loads and cleans the product dataset
- Creates combined text fields for searching
- Builds text preprocessing pipeline
-
Part 2 - Search Indexes (
Hybrid_Search_Part2_SearchIndexes.ipynb)- Builds BM25 inverted index (~24K unique terms)
- Generates semantic embeddings for all products
- Implements HybridSearch class
-
Part 3 - Evaluation (
Hybrid_Search_Part3_Evaluation.ipynb)- Evaluates all search methods on ground truth data
- Performs weight optimization experiments
- Generates visualizations and comparisons
from hybrid_search import HybridSearch
# Initialize (after running notebooks to generate indexes)
search = HybridSearch(bm25_index, product_embeddings, semantic_model,
preprocessor, products_df, bm25_weight=0.2, semantic_weight=0.8)
# Search
results = search.search_hybrid("wireless earbuds for gym", top_k=10)
print(results[['product_name', 'hybrid_score']])- Definition: Percentage of queries where the correct product appeared in top-K results
- Our Result: 54% at K=10 (optimized hybrid)
- Definition: Average of (1/rank) where rank is the position of the correct product
- Our Result: 0.316 (meaning correct products appear around rank 3 on average)
Experimented with 11 different BM25/Semantic weight combinations:
| BM25 Weight | Semantic Weight | Hit Rate | MRR |
|---|---|---|---|
| 0.0 | 1.0 | 50.0% | 0.289 |
| 0.1 | 0.9 | 48.0% | 0.305 |
| 0.2 | 0.8 | 54.0% | 0.316 |
| 0.3 | 0.7 | 46.0% | 0.263 |
| 0.5 | 0.5 | 36.0% | 0.227 |
| 1.0 | 0.0 | 30.0% | 0.208 |
Finding: Optimal weights are BM25=0.2, Semantic=0.8
-
Semantic search outperforms pure keyword search for e-commerce queries where users describe intent rather than exact product names.
-
Hybrid approach provides robustness - when one method fails, the other often succeeds.
-
Weight tuning matters - the optimal 0.2/0.8 split improved hit rate by 8 percentage points over equal weights.
-
BM25 still valuable for exact product name searches (e.g., "Asics Gel-Kayano 22").
- FAISS Integration - Scale to millions of products with approximate nearest neighbor search
- Cross-Encoder Re-ranking - Use more accurate but slower model for top-20 re-ranking
- Query Classification - Dynamic weights based on query type detection
- Fine-tuned Embeddings - Train on e-commerce domain data for better representations
- Real-time Learning - Update based on click-through and purchase signals
- Records: 20,000 products
- Fields: product_name, product_category_tree, description, brand, product_specifications
- Records: 50 query-purchase pairs
- Fields: query (what user searched), final_purchased_product (what they bought)
- Sentence Transformers - Pre-trained embedding models
- BM25 Algorithm - Probabilistic ranking function
- all-mpnet-base-v2 - Model card
D. Ryder Machine Learning Case Study - Week 6: Transformers
This project is for educational purposes as part of a machine learning curriculum.
Built with Python, Sentence Transformers, and a passion for improving search experiences.