FEATURE REQUEST: Migrate from Tavily Search to DuckDuckGo Search

[smokeyScraper]

Is your feature request related to a problem?

• Yes, it is related to a problem

Describe the feature you'd like

🌟 Feature Description

Migrate the web search functionality from Tavily Search to DuckDuckGo Search.

🔍 Problem Statement

The current implementation of the web search feature relies on the Tavily Search API. This feature is utilized in the handle_web_search_node to perform web searches. The TavilySearchTool class in backend/app/agents/devrel/tools/search_tool.py directly interacts with the Tavily API. The dependency on Tavily is also specified in the project's dependencies. This migration is intended to replace Tavily Search with DuckDuckGo Search as the primary web search provider.

🎯 Expected Outcome

Upon completion, the project will use DuckDuckGo Search for all web search queries. The langchain-tavily and tavily-python dependencies will be removed from the project, and a new dependency for a DuckDuckGo search library will be added. The handle_web_search_node and the search tool will be updated to use the new DuckDuckGo search functionality.

📷 Screenshots and Design Ideas

N/A (This is a backend implementation change with no direct UI impact).

📋 Additional Context

The following files will be impacted by this change:

• backend/app/agents/devrel/tools/search_tool.py: The TavilySearchTool class will need to be replaced with a new class for DuckDuckGo search.
• backend/app/agents/devrel/nodes/handle_web_search_node.py: The handle_web_search_node will need to be updated to use the new DuckDuckGo search tool.
• pyproject.toml: The langchain-tavily and tavily-python dependencies will be removed and a new dependency for DuckDuckGo search will be added.
• backend/app/core/config.py: The tavily_api_key setting will be removed and a new setting for the DuckDuckGo API key may need to be added.

Record

• I agree to follow this project's Code of Conduct
• I want to work on implementing this feature

[Rudra775]

hello @smokeyScraper , I'd love to work on this issue. Can you please assign it to me?

[smokeyScraper]

sure, give it a go.
The major focus is to remove the "credit dependency" of the Tavilly API. Fulfill .env.example properly if you want to run the bot; rest a PR structuring up DuckDuckGo as per the current structure separately tested would work, I'll test from my side.

[Rudra775]

hey @smokeyScraper, I've implemented the DuckDuckGo search tool as a replacement for Tavily, preserving the project structure and not adding any new dependencies.
I tested it locally, the tool returns valid search results via command-line, as expected.
just wanted to check: would you prefer I test it further through the Discord bot, or is this CLI verification enough for a PR? just a little new to this testing stuff

[smokeyScraper]

cool, awesome bruv.

just wanted to check: would you prefer I test it further through the Discord bot, or is this CLI verification enough for a PR? just a little new to this testing stuff

I mean, yeah, it works, and surely you could test it, though, it would be a lot nicer. The only thing I was concerned about was that testing it as a bot will require a lot of API keys to be initialized, so yeah, didn't want that to cause a problem for you while contributing.

[Rudra775]

i'll try to get it done, can i get 1-2 days ? have a few work related to college.

[smokeyScraper]

sure sure, works, take your time...

[Rudra775]

ok so, i think i've tried whatever was possible, but I couldnt resolve this issue related langchain. i was able to setup the bot succesfully. but it kept giving me this error "Graph not properly initialized for DevRelAgent: Unknown field for Part: thought".

again i've successfully made changes in the files i was required to. and was getting result of my temporary testing :
"Result: ['Introduction to LangChain - GeeksforGeeks', 'Introduction | ️ LangChain', 'What is LangChain? - LangChain Explained - AWS', 'What Is LangChain? - IBM', 'What Is LangChain: Components, Benefits & How to Get Started - lakeFS', 'Getting Started with LangChain: A Beginner's Guide to Building LLM ...', 'What Is LangChain? - Built In', 'Ultimate Guide to Getting Started with LangChain | by Zilliz - Medium', 'LangChain tutorial: An intro to building LLM-powered apps', 'What Is LangChain? - DataStax']"

if this sounds good enough to you then i'll make a pr ? @smokeyScraper

[smokeyScraper]

@Rudra775
can't say much on the topic without seeing any code, but I think there must be some versioning issues.
Have you tried this? langchain-ai/langchain-google#956

I get it, got the problem. we are probably using a very old version of langchain[google-genai], dk why poetry had a cap on the versioning; maybe a few dependencies. (check poetry)
Refer -> https://ai.google.dev/gemini-api/docs/libraries https://python.langchain.com/api_reference/google_genai/

Thanks a lot for helping me point this out. I'll be updating the dependencies soon. Can you please have a quick check on the search tool after this fix on your local?

[Rudra775]

ok sure will do

I think its fine now, do check if this is the expected output

[smokeyScraper]

LGTM!!!
take your time, refine it around at its best. Can it provide citations similar to Tavilly API?

[Rudra775]

cool, i'll work on it on Sunday, will have to spend like 10 hours in college tomorrow :(

[Rudra775]

File "C:\Users\Rudra\OneDrive\Desktop\JS\opensource\Devr.AI\backend\app\database\weaviate\operations.py", line 6, in
from app.database.weaviate.client import get_weaviate_client
File "C:\Users\Rudra\OneDrive\Desktop\JS\opensource\Devr.AI\backend\app\database\weaviate\client.py", line 1, in
import weaviate
from .v4 import ConnectionV4
File "C:\Users\Rudra\AppData\Local\Programs\Python\Python312\Lib\site-packages\weaviate\connect\v4.py", line 30, in
from grpc_health.v1 import health_pb2 # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Rudra\AppData\Local\Programs\Python\Python312\Lib\site-packages\grpc_health\v1\health_pb2.py", line 12, in
_runtime_version.ValidateProtobufRuntimeVersion(
File "C:\Users\Rudra\AppData\Local\Programs\Python\Python312\Lib\site-packages\google\protobuf\runtime_version.py", line 106, in ValidateProtobufRuntimeVersion
_ReportVersionError(
File "C:\Users\Rudra\AppData\Local\Programs\Python\Python312\Lib\site-packages\google\protobuf\runtime_version.py", line 50, in _ReportVersionError
raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected mismatched Protobuf Gencode/Runtime major versions when loading grpc_health/v1/health.proto: gencode 6.31.0 runtime 5.29.5. Same major version is required. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

i've been getting this error, didnt encounter this previously. tried going to that docs reference and searched the web too, still no luck. could you help me ?
@smokeyScraper
attaching the error for your refernce