feat: add Tavily search provider option to Agentic-RAG-Pipeline#35
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces Tavily Search as an optional web search provider alongside Google CSE. The changes include a new TavilyWebSearch wrapper class in core/tools.py, updated environment variable examples, and logic in app.py to toggle between search providers. Feedback suggests improving error reporting when a specific search provider is requested but its API key is missing, and adding an upper version bound to the tavily-python dependency in requirements.txt to ensure consistency and long-term stability.
| if search_provider == "tavily" and tavily_key: | ||
| web = TavilyWebSearch(api_key=tavily_key) | ||
| print("[web] Tavily Search enabled.") | ||
| elif cse_key and cse_engine: | ||
| web = WebSearch(api_key=cse_key, engine_id=cse_engine) | ||
| print("[web] Google Programmable Search enabled.") | ||
| else: | ||
| print("[web] Web search disabled (set CSE_API_KEY & CSE_ENGINE_ID to enable).") | ||
| print("[web] Web search disabled (set CSE_API_KEY & CSE_ENGINE_ID, or TAVILY_API_KEY to enable).") |
There was a problem hiding this comment.
When SEARCH_PROVIDER is explicitly set to 'tavily' but TAVILY_API_KEY is missing, the application currently falls back to Google CSE (if configured) or disables search entirely without a specific warning. It would be more helpful to the user to explicitly report that the requested provider is misconfigured.
| if search_provider == "tavily" and tavily_key: | |
| web = TavilyWebSearch(api_key=tavily_key) | |
| print("[web] Tavily Search enabled.") | |
| elif cse_key and cse_engine: | |
| web = WebSearch(api_key=cse_key, engine_id=cse_engine) | |
| print("[web] Google Programmable Search enabled.") | |
| else: | |
| print("[web] Web search disabled (set CSE_API_KEY & CSE_ENGINE_ID to enable).") | |
| print("[web] Web search disabled (set CSE_API_KEY & CSE_ENGINE_ID, or TAVILY_API_KEY to enable).") | |
| if search_provider == "tavily": | |
| if tavily_key: | |
| web = TavilyWebSearch(api_key=tavily_key) | |
| print("[web] Tavily Search enabled.") | |
| else: | |
| print("[web] Tavily Search requested but TAVILY_API_KEY is missing.") | |
| elif cse_key and cse_engine: | |
| web = WebSearch(api_key=cse_key, engine_id=cse_engine) | |
| print("[web] Google Programmable Search enabled.") | |
| else: | |
| print("[web] Web search disabled (set CSE_API_KEY & CSE_ENGINE_ID, or TAVILY_API_KEY to enable). ") |
| pydantic>=2.7.0,<3.0.0 | ||
| python-dotenv>=1.0.1,<2.0.0 | ||
| numpy>=1.26.4,<3.0.0 | ||
| tavily-python>=0.3.0 |
There was a problem hiding this comment.
Summary
Adds Tavily as a configurable web search provider alongside the existing Google Custom Search Engine (CSE) in the Agentic-RAG-Pipeline sub-app. This is an additive change — Google CSE remains the default and is fully intact.
What changed
Agentic-RAG-Pipeline/core/tools.py: AddedTavilyWebSearchclass usingtavily-pythonSDK, exposing the samesearch(q, num)interface and{title, url, snippet}return format as the existingWebSearchclass.Agentic-RAG-Pipeline/app.py: Extended provider selection logic to checkSEARCH_PROVIDERenv var. When set to"tavily"andTAVILY_API_KEYis present, usesTavilyWebSearch; otherwise falls through to Google CSE as before.Agentic-RAG-Pipeline/.env.example: AddedSEARCH_PROVIDERandTAVILY_API_KEYentries alongside existing CSE vars.Agentic-RAG-Pipeline/requirements.txt: Addedtavily-python>=0.3.0.How to use
Set these environment variables to switch to Tavily:
To keep using Google CSE (the default), no changes are needed.
Notes for reviewers
CSE_API_KEY,CSE_ENGINE_ID), and dependencies are untouched.TavilyWebSearchmaps Tavily'scontentfield tosnippetfor interface compatibility.Automated Review
TavilyWebSearchproperly mirrors theWebSearchinterface, uses the right Tavily SDK call (TavilyClient.search(query=..., max_results=...)) with correct response field mapping (content→snippet), and the synchronous client is appropriate for this non-async app.app.pycorrectly implements theSEARCH_PROVIDERenv-var gate while preserving the Google CSE fallback path. All four planned files are updated. Two minor issues: thetavily-pythonpin lacks an upper bound (inconsistent with all other deps), and there is no warning log whenSEARCH_PROVIDER=tavilyis set butTAVILY_API_KEYis absent (silent fallthrough to CSE/disabled).