This repository explores the use of Genetic Programming (GP) to detect duplicate questions by evolving interpretable decision functions over similarity features. Instead of relying on a fixed similarity threshold or a single learned model, GP is used to automatically construct and optimize expressions that combine linguistic, semantic, and structural features derived from question pairs. The approach is evaluated on the Quora Duplicate Questions Dataset.
- Uses genetic programming to evolve feature-combining expressions and adaptive similarity thresholds.
- Integrates linguistic, syntactic, semantic, and statistical similarity measures in a unified framework.
- Treats duplicate detection as a multi-objective optimization problem, balancing predictive performance and model complexity.
- Provides visual and quantitative analysis of evolved solutions and their trade-offs.
- Parts-of-Speech (POS) tag similarity
- Dependency parsing similarity
- Sentiment difference (VADER)
- Synonym overlap
- N-gram similarity
- Question length comparison
- Unique word overlap
- Evolution of symbolic decision expressions over similarity features
- Multi-objective optimization (e.g., accuracy vs. expression complexity)
- K-fold cross-validation during fitness evaluation
- Transparent, interpretable evolved models
- Fitness evolution across generations
- Pareto fronts of optimal solutions
- Threshold and expression statistics
- Confusion matrices and standard performance metrics
Performance comparison against:
- Decision Trees
- Random Forests
- Support Vector Machines
- Linear Regression
- Data Preparation
- Question preprocessing and normalization
- Feature Extraction
- Computation of custom similarity features for question pairs
- Genetic Programming Optimization
- Evolution of decision expressions and thresholds
- Cross-validated fitness evaluation
- Baseline Model Training
- Training and evaluation of traditional ML classifiers
- Analysis & Visualization
- Comparative performance analysis and interpretability inspection
os,math,random,operator
pandas,numpy,scipy,tqdm
nltk,spacy,vaderSentiment,TextBlob
scikit-learn,sentence-transformers
matplotlib,seaborn,networkx
deap