Hi @dzhng,
First off, deep-research is a fantastic piece of engineering. The iterative search-> scrape -> synthesize loop is exactly the workflow I was looking for, and the output quality is genuinely impressive.
Pain point I've run into
When I run longer research sessions (e.g., 50+ search queries in a single run), the built-in search layer starts hitting rate limits and temporary blocks. I usually self-host on a VPS, and after about 30–40 Google queries the IP gets flagged. I then have to pause the run, wait, or switch networks, which breaks the iterative flow that makes the tool so powerful.
I tried adding proxy rotation, but that adds operational complexity I'd rather avoid for a research tool.
Suggestion
Would you consider adding https://serpbase.dev as an optional search backend? It's a Google Search Results API that returns structured JSON (title, URL, snippet, rich results, related questions) without the scraping overhead. For deep-research's use case, it could act as the "discovery" layer before the existing scrape-and-synthesize step kicks in.
Why it fits
- The API returns clean, consistent JSON — less defensive parsing in the scraping layer.
- Pricing is aimed at developers running these kinds of tools at scale, so it aligns with the self-hosted audience.
- It keeps the project useful even on restricted networks or cloud IPs where raw scraping is difficult.
- Integration surface is small: the response schema maps almost 1:1 to the existing search result structure.
Maybe an env var like SEARCH_PROVIDER=serpbase alongside the current SEARCH_PROVIDER options could be the toggle.
Happy to prototype the integration or provide sample response payloads if it helps.
Hi @dzhng,
First off, deep-research is a fantastic piece of engineering. The iterative search-> scrape -> synthesize loop is exactly the workflow I was looking for, and the output quality is genuinely impressive.
Pain point I've run into
When I run longer research sessions (e.g., 50+ search queries in a single run), the built-in search layer starts hitting rate limits and temporary blocks. I usually self-host on a VPS, and after about 30–40 Google queries the IP gets flagged. I then have to pause the run, wait, or switch networks, which breaks the iterative flow that makes the tool so powerful.
I tried adding proxy rotation, but that adds operational complexity I'd rather avoid for a research tool.
Suggestion
Would you consider adding https://serpbase.dev as an optional search backend? It's a Google Search Results API that returns structured JSON (title, URL, snippet, rich results, related questions) without the scraping overhead. For deep-research's use case, it could act as the "discovery" layer before the existing scrape-and-synthesize step kicks in.
Why it fits
Maybe an env var like
SEARCH_PROVIDER=serpbasealongside the currentSEARCH_PROVIDERoptions could be the toggle.Happy to prototype the integration or provide sample response payloads if it helps.