PaRAGraph is a Retrieval-Augmented Generation (RAG) system designed to answer technical questions using IBM's Tech-QA dataset. The system uses different models suited for embedding, query rewriting, context reranking, answerability prediction, answer generation and answer evaluation.
The RAG pipeline consists of the following components, each one is a notebook contained in the techqa_report folder:
- Data Preprocessing - Document parsing and section extraction
- Vector Store Creation - Embedding generation and indexing
- Query Rewriting - Query optimization for better retrieval
- Query Embedding - Queries are embedded in a separate notebook to speed up the subsequent retrieval phase
- Contexts Retrieval - Semantic search and document reconstruction
- Contexts Reranking - Retrieved contexts are re-ranked based on relevance
- Contexts Evaluation - Evaluation of retrieved contexts using metrics such as hits@k, ndcg@k, and mrr
- Answerability Prediction - Determining if a question can be answered (binary classification)
- Answerability Evaluation - Computing classification metrics such as true positives, false positives, precision, recall, and F1-score
- Answer Generation - LLM-based response generation
- Evaluation - Performance evaluation using multiple metrics such as BLEU, ROUGE-L, METEOR, cosine similarity, Longest Match Ratio and Answer Relevancy (computed with a LLM-as-a-Judge)
The main RAG system implementation consists of:
- Document Retrieval: Semantic search with score thresholding
- Answer Generation: Context-aware response generation
Streamlit-based GUI providing:
- Interactive chat interface
- Real-time document visualization
- Conversation history management
- Reference document display with official links
MongoDB-based system for chat history persistence
- ibm-granite/granite-embedding-278m-multilingual: used for embedding creation and management.
- Qwen/Qwen3-Embedding-0.6B: used for embedding creation and management.
- infly/inf-retriever-v1-1.5b: used for embedding creation and management.
- catyung/t5l-turbo-hotpot-0331: used for rewriting queries before retrieving documents from the Vector Store Index.
- Qwen/Qwen3-Reranker-0.6B: used for re-ranking retrieved context documents.
- ibm-granite/granite-3.2-8b-instruct: used for answer generation.
- ibm-granite/granite-3.2-8b-lora-rag-answerability-prediction: LoRA adapter of the previous model, used for predicting the answerability of a question.
- openai/gpt-oss-20b: used only for demonstration purposes to simulate a production environment, since the model was too large to run on mid-range hardware.
- deepseek-ai/DeepSeek-R1-Distill-Qwen-7B: used for validating the answers generated by the RAG system.
- Vector Database: Elasticsearch 9.0.3
- Conversations Database: MongoDB
- GPU Support: DirectML (for AMD GPUs) / CUDA
- Framework: LlamaIndex for RAG orchestration
- UI: Streamlit for web interface
beautifulsoup4- Text extraction from HTML pagesdeepeval- Evaluation of generated answersevaluate- Evaluation of generated answersgroq- Answer generation (used only in the demo)llama-index- Document retrieval frameworknltk- Evaluation of generated answerspeft- LoRA adapter application during answer generationranx- Evaluation of retrieved context documentsscikit-learn- Evaluation of generated answerstextdistance- Evaluation of generated answerstransformers- Hugging Face model integrationvllm- Efficient LLM inference
- Prerequisites: Install Elasticsearch and
requirements.txt - Environment: Set up Python environment with required packages (3.10 if using torch_directml)
- Launch Elasticsearch DB: for storing the embeddings
- Data Preparation: Run notebooks 1-2 to process data and create indices
- Application Launch: Execute
streamlit run start_gui.pyto start the GUI