Feature/semantic search rag

.cursor/rules/general-rule.mdc

@@ -5,7 +5,6 @@ alwaysApply: true
 ---
 ## Rules to Follow
 
-- You must always commit your changes whenever you update code. 
-- You must always try and write code that is well documented. (self or commented is fine)
-- You must only work on a single feature at a time.
-- You must explain your decisions thouroughly to the user.
+You always prefer to use branch development. Before writing any code - you create a feature branch to hold those changes. 
+
+After you are done - provide instructions in a "MERGE.md" file that explains how to merge the changes back to main with both a GitHub PR route and a GitHub CLI route.

.gitignore

@@ -3,6 +3,9 @@ uv.lock
 
 # Byte-compiled / optimized / DLL files
 __pycache__/
+*.pyc
+*.pyo
+*.pyd
 *.py[cod]
 *$py.class
 
@@ -163,3 +166,11 @@ cython_debug/
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 #.idea/
 .vercel
+
+# Node.js
+node_modules
+.next
+
+# PDF filess
+*.pdf
+

.vercelignore

@@ -0,0 +1,17 @@
+# Ignore development files
+*.log
+*.tmp
+.DS_Store
+.vscode/
+.idea/
+
+# Ignore large files that aren't needed for deployment
+*.ipynb
+*.md
+.git/
+.gitignore
+
+# Keep only essential files for deployment
+!api/
+!frontend/
+!vercel.json

MERGE.md

@@ -0,0 +1,111 @@
+# Merge Instructions for Semantic Search RAG
+
+This document provides instructions for merging the semantic search RAG implementation back to the main branch.
+
+## Changes Made
+
+### 🔍 Semantic Search Implementation
+- **Enhanced RAG**: Replaced keyword-based search with semantic search using existing VectorDatabase
+- **Leverages aimakerspace**: Uses proven VectorDatabase and EmbeddingModel from aimakerspace modules
+- **AI-Powered Relevance**: Uses OpenAI embeddings to find semantically similar content
+- **Fallback System**: Graceful degradation to keyword search if embeddings fail
+- **Vercel-Compatible**: Stateless implementation that works with serverless functions
+
+### 📁 Files Modified
+1. `api/app.py` - Refactored to use existing VectorDatabase from aimakerspace modules
+2. `api/requirements.txt` - Added numpy dependency for vector operations
+
+### 🎯 Key Features
+- **Semantic Understanding**: Finds relevant content based on meaning, not just keywords
+- **Better Relevance**: Query "car" finds "automobile", "vehicle", "transportation" content
+- **Robust Fallback**: Falls back to keyword search if embedding API fails
+- **Vercel-Optimized**: No persistent storage, works with serverless constraints
+- **Cost-Efficient**: Generates embeddings on-demand per request
+- **Reuses Existing Code**: Leverages proven aimakerspace VectorDatabase implementation
+- **Better Architecture**: Uses established patterns instead of custom implementations
+
+## Merge Instructions
+
+### Option 1: GitHub Pull Request (Recommended)
+
+1. **Push the feature branch:**
+   ```bash
+   git push origin feature/semantic-search-rag
+   ```
+
+2. **Create Pull Request:**
+   - Go to your GitHub repository
+   - Click "Compare & pull request" for the `feature/semantic-search-rag` branch
+   - Title: "🔍 Implement Semantic Search RAG"
+   - Description: "Enhances RAG system with semantic search using OpenAI embeddings for better content relevance"
+   - Review the changes and create the PR
+   - Merge when ready
+
+### Option 2: GitHub CLI
+
+1. **Push the feature branch:**
+   ```bash
+   git push origin feature/semantic-search-rag
+   ```
+
+2. **Create and merge PR:**
+   ```bash
+   # Create pull request
+   gh pr create --title "🔍 Implement Semantic Search RAG" --body "Enhances RAG system with semantic search using OpenAI embeddings for better content relevance"
+
+   # Review the PR (optional)
+   gh pr view
+
+   # Merge the PR
+   gh pr merge --merge --delete-branch
+   ```
+
+### Option 3: Direct Merge (Not Recommended)
+
+```bash
+# Switch to main branch
+git checkout main
+
+# Merge the feature branch
+git merge feature/semantic-search-rag
+
+# Push to remote
+git push origin main
+
+# Clean up feature branch
+git branch -d feature/semantic-search-rag
+git push origin --delete feature/semantic-search-rag
+```
+
+## Testing
+
+After merging, test the following:
+
+1. **Semantic Search**: Upload a PDF and test queries that should find semantically similar content
+2. **Fallback System**: Test with invalid API keys to ensure keyword search fallback works
+3. **Performance**: Monitor embedding generation time and API costs
+4. **Vercel Deployment**: Ensure the stateless implementation works on Vercel
+5. **Error Handling**: Test edge cases like empty PDFs or network failures
+
+## Rollback Instructions
+
+If issues arise, you can rollback by reverting the merge:
+
+```bash
+# Find the merge commit
+git log --oneline
+
+# Revert the merge (replace COMMIT_HASH with actual hash)
+git revert -m 1 COMMIT_HASH
+
+# Push the revert
+git push origin main
+```
+
+## Notes
+
+- **Cost Consideration**: Semantic search generates more API calls (embeddings per request)
+- **Performance**: Slightly slower than keyword search due to embedding generation
+- **Reliability**: Robust fallback ensures system continues working even if embeddings fail
+- **Vercel Compatibility**: Stateless design works perfectly with serverless functions
+- **Better Results**: Semantic search provides much more relevant content retrieval

aimakerspace/openai_utils/chatmodel.py

@@ -0,0 +1,63 @@
+import os
+from typing import Any, AsyncIterator, Iterable, List, MutableMapping
+
+from openai import AsyncOpenAI, OpenAI
+
+ChatMessage = MutableMapping[str, Any]
+
+
+class ChatOpenAI:
+    """Thin wrapper around the OpenAI chat completion APIs."""
+
+    def __init__(self, model_name: str = "gpt-4o-mini"):
+        self.model_name = model_name
+        self.openai_api_key = os.getenv("OPENAI_API_KEY")
+        if self.openai_api_key is None:
+            raise ValueError("OPENAI_API_KEY is not set")
+
+        self._client = OpenAI()
+        self._async_client = AsyncOpenAI()
+
+    def run(
+        self,
+        messages: Iterable[ChatMessage],
+        text_only: bool = True,
+        **kwargs: Any,
+    ) -> Any:
+        """Execute a chat completion request.
+
+        ``messages`` must be an iterable of ``{"role": ..., "content": ...}``
+        dictionaries. When ``text_only`` is ``True`` (the default) only the
+        completion text is returned; otherwise the full response object is
+        provided.
+        """
+
+        message_list = self._coerce_messages(messages)
+        response = self._client.chat.completions.create(
+            model=self.model_name, messages=message_list, **kwargs
+        )
+
+        if text_only:
+            return response.choices[0].message.content
+
+        return response
+
+    async def astream(
+        self, messages: Iterable[ChatMessage], **kwargs: Any
+    ) -> AsyncIterator[str]:
+        """Yield streaming completion chunks as they arrive from the API."""
+
+        message_list = self._coerce_messages(messages)
+        stream = await self._async_client.chat.completions.create(
+            model=self.model_name, messages=message_list, stream=True, **kwargs
+        )
+
+        async for chunk in stream:
+            content = chunk.choices[0].delta.content
+            if content is not None:
+                yield content
+
+    def _coerce_messages(self, messages: Iterable[ChatMessage]) -> List[ChatMessage]:
+        if isinstance(messages, list):
+            return messages
+        return list(messages)

aimakerspace/openai_utils/embedding.py

@@ -0,0 +1,67 @@
+import asyncio
+import os
+from typing import Iterable, List
+
+from openai import AsyncOpenAI, OpenAI
+
+
+class EmbeddingModel:
+    """Helper for generating embeddings via the OpenAI API."""
+
+    def __init__(self, embeddings_model_name: str = "text-embedding-3-small"):
+        self.openai_api_key = os.getenv("OPENAI_API_KEY")
+        if self.openai_api_key is None:
+            raise ValueError(
+                "OPENAI_API_KEY environment variable is not set. "
+                "Please configure it with your OpenAI API key."
+            )
+
+        self.embeddings_model_name = embeddings_model_name
+        self.async_client = AsyncOpenAI()
+        self.client = OpenAI()
+
+    async def async_get_embeddings(self, list_of_text: Iterable[str]) -> List[List[float]]:
+        """Return embeddings for ``list_of_text`` using the async client."""
+
+        embedding_response = await self.async_client.embeddings.create(
+            input=list(list_of_text), model=self.embeddings_model_name
+        )
+
+        return [item.embedding for item in embedding_response.data]
+
+    async def async_get_embedding(self, text: str) -> List[float]:
+        """Return an embedding for a single text using the async client."""
+
+        embedding = await self.async_client.embeddings.create(
+            input=text, model=self.embeddings_model_name
+        )
+
+        return embedding.data[0].embedding
+
+    def get_embeddings(self, list_of_text: Iterable[str]) -> List[List[float]]:
+        """Return embeddings for ``list_of_text`` using the sync client."""
+
+        embedding_response = self.client.embeddings.create(
+            input=list(list_of_text), model=self.embeddings_model_name
+        )
+
+        return [item.embedding for item in embedding_response.data]
+
+    def get_embedding(self, text: str) -> List[float]:
+        """Return an embedding for a single text using the sync client."""
+
+        embedding = self.client.embeddings.create(
+            input=text, model=self.embeddings_model_name
+        )
+
+        return embedding.data[0].embedding
+
+
+if __name__ == "__main__":
+    embedding_model = EmbeddingModel()
+    print(asyncio.run(embedding_model.async_get_embedding("Hello, world!")))
+    print(
+        asyncio.run(
+            embedding_model.async_get_embeddings(["Hello, world!", "Goodbye, world!"])
+        )
+    )

aimakerspace/openai_utils/prompts.py

@@ -0,0 +1,60 @@
+import re
+from typing import Any, Dict, List
+
+
+class BasePrompt:
+    """Simple string template helper used to format prompt text."""
+
+    def __init__(self, prompt: str):
+        self.prompt = prompt
+        self._pattern = re.compile(r"\{([^}]+)\}")
+
+    def format_prompt(self, **kwargs: Any) -> str:
+        """Return the prompt with ``kwargs`` substituted for placeholders."""
+
+        matches = self._pattern.findall(self.prompt)
+        replacements = {match: kwargs.get(match, "") for match in matches}
+        return self.prompt.format(**replacements)
+
+    def get_input_variables(self) -> List[str]:
+        """Return the placeholder names used by this prompt."""
+
+        return self._pattern.findall(self.prompt)
+
+
+class RolePrompt(BasePrompt):
+    """Prompt template that also captures an accompanying chat role."""
+
+    def __init__(self, prompt: str, role: str):
+        super().__init__(prompt)
+        self.role = role
+
+    def create_message(self, apply_format: bool = True, **kwargs: Any) -> Dict[str, str]:
+        """Build an OpenAI chat message dictionary for this prompt."""
+
+        content = self.format_prompt(**kwargs) if apply_format else self.prompt
+        return {"role": self.role, "content": content}
+
+
+class SystemRolePrompt(RolePrompt):
+    def __init__(self, prompt: str):
+        super().__init__(prompt, "system")
+
+
+class UserRolePrompt(RolePrompt):
+    def __init__(self, prompt: str):
+        super().__init__(prompt, "user")
+
+
+class AssistantRolePrompt(RolePrompt):
+    def __init__(self, prompt: str):
+        super().__init__(prompt, "assistant")
+
+
+if __name__ == "__main__":
+    prompt = BasePrompt("Hello {name}, you are {age} years old")
+    print(prompt.format_prompt(name="John", age=30))
+
+    prompt = SystemRolePrompt("Hello {name}, you are {age} years old")
+    print(prompt.create_message(name="John", age=30))
+    print(prompt.get_input_variables())