Skip to content

feat: add Tavily search provider option to Agentic-RAG-Pipeline #35

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tavily-integrations wants to merge 1 commit into
base: master
Choose a base branch
from
Open

feat: add Tavily search provider option to Agentic-RAG-Pipeline #35

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions Agentic-RAG-Pipeline/.env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,11 @@ GOOGLE_API_KEY=YOUR_GEMINI_API_KEY
CSE_API_KEY=YOUR_GOOGLE_CSE_API_KEY
CSE_ENGINE_ID=YOUR_GOOGLE_CSE_ENGINE_ID

# Optional: choose search provider — "google_cse" (default) or "tavily"
# SEARCH_PROVIDER=google_cse

# Optional (enable web retrieval via Tavily Search)
# TAVILY_API_KEY=YOUR_TAVILY_API_KEY

# Optional (local corpus directory)
CORPUS_DIR=corpus
12 changes: 9 additions & 3 deletions Agentic-RAG-Pipeline/app.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

from core.vector import FAISSIndex, ingest_corpus
from core.memory import SessionMemory
from core.tools import WebSearch
from core.tools import WebSearch, TavilyWebSearch
from graph.orchestrator import Orchestrator

def ensure_env(var):
Expand Down Expand Up @@ -32,12 +32,18 @@ def main():
print("[ingest] No corpus/ directory found. Running with empty vector store.")

# --- Web search (optional) ---
search_provider = os.getenv("SEARCH_PROVIDER", "google_cse").lower()
tavily_key = os.getenv("TAVILY_API_KEY")

web = None
if cse_key and cse_engine:
if search_provider == "tavily" and tavily_key:
web = TavilyWebSearch(api_key=tavily_key)
print("[web] Tavily Search enabled.")
elif cse_key and cse_engine:
web = WebSearch(api_key=cse_key, engine_id=cse_engine)
print("[web] Google Programmable Search enabled.")
else:
print("[web] Web search disabled (set CSE_API_KEY & CSE_ENGINE_ID to enable).")
print("[web] Web search disabled (set CSE_API_KEY & CSE_ENGINE_ID, or TAVILY_API_KEY to enable).")
Comment on lines +39 to +46

@gemini-code-assist gemini-code-assist Bot May 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

When SEARCH_PROVIDER is explicitly set to 'tavily' but TAVILY_API_KEY is missing, the application currently falls back to Google CSE (if configured) or disables search entirely without a specific warning. It would be more helpful to the user to explicitly report that the requested provider is misconfigured.

Suggested change
if search_provider == "tavily" and tavily_key:
web = TavilyWebSearch(api_key=tavily_key)
print("[web] Tavily Search enabled.")
elif cse_key and cse_engine:
web = WebSearch(api_key=cse_key, engine_id=cse_engine)
print("[web] Google Programmable Search enabled.")
else:
print("[web] Web search disabled (set CSE_API_KEY & CSE_ENGINE_ID to enable).")
print("[web] Web search disabled (set CSE_API_KEY & CSE_ENGINE_ID, or TAVILY_API_KEY to enable).")
if search_provider == "tavily":
if tavily_key:
web = TavilyWebSearch(api_key=tavily_key)
print("[web] Tavily Search enabled.")
else:
print("[web] Tavily Search requested but TAVILY_API_KEY is missing.")
elif cse_key and cse_engine:
web = WebSearch(api_key=cse_key, engine_id=cse_engine)
print("[web] Google Programmable Search enabled.")
else:
print("[web] Web search disabled (set CSE_API_KEY & CSE_ENGINE_ID, or TAVILY_API_KEY to enable). ")


# --- Memory ---
memory = SessionMemory() # in-process JSONL-style memory
Expand Down
20 changes: 20 additions & 0 deletions Agentic-RAG-Pipeline/core/tools.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
import httpx
import requests
from bs4 import BeautifulSoup
from tavily import TavilyClient

class WebSearch:
"""
Expand All @@ -29,6 +30,25 @@ def search(self, q: str, num: int = 5) -> List[Dict]:
})
return out

class TavilyWebSearch:
"""
Tavily Search API wrapper, matching the WebSearch interface.
"""
def __init__(self, api_key: str):
self.client = TavilyClient(api_key=api_key)

def search(self, q: str, num: int = 5) -> List[Dict]:
response = self.client.search(query=q, max_results=num)
out = []
for it in response.get("results", []):
out.append({
"title": it.get("title"),
"url": it.get("url"),
"snippet": it.get("content", "")
})
return out


def fetch_page_text(url: str, timeout: int = 20) -> Optional[str]:
"""
Fetch a URL and extract visible text via BeautifulSoup.
Expand Down
1 change: 1 addition & 0 deletions Agentic-RAG-Pipeline/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,4 @@ beautifulsoup4>=4.12.2,<5.0.0
pydantic>=2.7.0,<3.0.0
python-dotenv>=1.0.1,<2.0.0
numpy>=1.26.4,<3.0.0
tavily-python>=0.3.0

@gemini-code-assist gemini-code-assist Bot May 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The tavily-python dependency is missing an upper bound, which is inconsistent with the other dependencies in this file. Adding an upper bound (e.g., <1.0.0) helps ensure stability by preventing automatic upgrades to potentially breaking major versions.

tavily-python>=0.3.0,<1.0.0