This project demonstrates how to use NLP, Transformers, and LLMs to automatically detect and improve low-quality Q&A posts. It combines multi-view feature engineering (embeddings, NER, subjective indicators, SoftNER) with Flan-T5 text generation for summarization and rephrasing.
- Multi-view pipeline with BERT embeddings, NER features, subjective indicators, and SoftNER
- Classification with XGBoost achieving 92% accuracy and 96% precision on LQ EDIT detection
- Explainability with SHAP to interpret model predictions
- LLM-powered generation:
- Summarization: Flan-T5 produces concise two-word labels
- Rephrasing: Fine-tuned Flan-T5 rewrites low-quality posts, upgrading 68% to high-quality
- All workflows implemented in a single Jupyter Notebook for reproducibility
- Data Preprocessing – Clean and normalize text (tokenization, stopword removal, etc.)
- Feature Engineering – Generate BERT [CLS] embeddings, extract NER & SoftNER counts, compute subjective indicators
- Classification – Train an XGBoost classifier and validate with accuracy, precision, recall, F1
- Explainability – Apply SHAP for model interpretation
- LLM Tasks –
- Summarization → Flan-T5 generates compact technical summaries
- Rephrasing → Flan-T5 fine-tuned to rewrite LQ posts into HQ versions
- Evaluation –
- Classification: Accuracy = 92%, Precision = 96%
- Rephrasing: BLEU = 0.78, BERTScore = 0.87
- Summarization: 90K+ posts tagged, reducing manual labeling effort by 60%
- Languages: Python
- Libraries: pandas, numpy, scikit-learn, XGBoost
- Transformers: HuggingFace (BERT, Flan-T5)
- NER Tools: NER, SoftNER
- Explainability: SHAP
- Evaluation Metrics: Accuracy, Precision, Recall, BLEU, BERTScore
- Visualization: matplotlib, seaborn
- Achieved 92% accuracy / 96% precision in classification
- Rephrased 11.2K posts, upgrading 68% to high-quality
- Summarized 90K+ entries into two-word labels, cutting manual tagging time by 60%
- Validated outputs with BLEU (0.78) and BERTScore (0.87)