Epiphany is a dual-domain RAG system built for students. It routes queries between a medical route grounded in PubMed abstracts and a technical route grounded in PyTorch, Pandas, Sklearn, Matplotlib and Streamlit documentation. Queries are embedded with domain-specific bi-encoders, retrieved from Pinecone vector indexes, reranked with a cross-encoder, and answered by Gemini — strictly from retrieved context, no hallucination.
- Landing Page:
- Medical Route Answer:
- Technical Route Answer:
-
This is a dual domain RAG system, meaning there are 2 different domains at play: Medical and Technical.
-
When a user lands at the landing page, they're prompted to choose one of these 2 routes.
-
Based on the user selected route, the appropriate embedding model is used to embed the user's query.
-
Then the embedded query vector is used to search through the respective vector DB index in a process called Approximate Nearest Neighbors (ANN).
-
The ANN returns the top-k records which had the highest cosine similarity score when compared with the query's embedding.
-
These top-k records are then re-evaluated to determine the 'best of the best' in a process called 'reranking' or 'cross-encoding'. This cross-encoding is carried out by a separate embedding model whose only duty is reranking/cross-encoding the top-k retrievals.
-
The top-n best matches (top-n out of the top-k) are separated along with their cross-encoding score and respective text.
-
The text fields of the best matches are then fed to an LLM, which generates an answer. This answer is then displayed to the user.
-
Data Ingestion:
- Usage:
- Since this is a dual domain system, it needed 2 different data sources for respective domains.
- It also needed 2 different, domain specific embedding models which could understand the terms and phrases related to the domains.
- The data source for medical domain are PubMed publications, downloaded through the 'biopython' library, I downloaded about 100 publications for each of the 10 topics, totaling to almost 1k files. [The 10 topics can be found in Data Aggregation/download_medical_docs.py]
- For technical data, I extracted the docstrings of the popular and commonly used Python libraries: PyTorch, Pandas, Sklearn, Matplotlib and Streamlit
- After all the data was aggregated, the chunkers.py file was created to break the data in chunks of a set size and set overlap.
- Medical => Chunk Size = 400 Tokens | Overlap = 50 Tokens
- Technical => Chunk Size = 1000 Tokens | Overlap = 150 Tokens
- The data sources then flow through the data ingestion pipeline (/data_ingestion) so that they get chunked, embedded and stored in the vector DB.
- The usage pipeline collects the user's query and chosen route (at generator/user_gemini.py) and these fields flow through the retrieval pipeline (/retrieval) through a series of function calls.
- The entire usage/retrieval operation is orchestrated by a simple FastAPI server (main.py) and the frontend is split into the main landing page and chat page (/frontend).
- ML: HuggingFace Sentence Transformer, PyTorch, Gemini (Google GenAI SDK), LangChain Text Splitter
- Backend: FastAPI
- Frontend: Streamlit
- Vector Database: Pinecone
- Medical Embedding Model: abhinand/MedEmbed-small-v0.1
- Technical Embedding Model: jinaai/jina-embeddings-v2-base-code
- Reranking Model: cross-encoder/ms-marco-MiniLM-L6-v2
Requires Python >=3.11 and poetry
pip install poetry
git clone https://github.com/prajwalnayaka/Epiphany.git
poetry install
python -m fastapi run main.py
streamlit run frontend/app.py
The RAG system isn't perfect when it comes to retrieval quality, plenty of times it returns the "Didn't find anything matching the query" message. My goal envisioned a tool to which someone could ask questions along the lines of "How can I do X given Y?" or "What was that Y which had X properties?" and they'd be satisfied with a valid and accurate answer. However, that is not the case, and it has everything to do with the source data, especially for the technical domain. Docstrings aren't very great at explaining the capabilities and use cases of a function or method. But a user's natural language query is very much structured around the description of the tool/topic they're pondering about. This fundamental gap causes a lapse in aptness of the retrieved information or no information being retrieved at all.
Future Scope lies in creating and using better data sources which cover topics both vertically and horizontally in scale while incorporating natural language explanations.
@software{balachandran2024medembed,
author = {Balachandran, Abhinand},
title = {MedEmbed: Medical-Focused Embedding Models},
year = {2024},
url = {https://github.com/abhinand5/MedEmbed}
}