diff --git a/docs/assets/tools-scrapegraphai.png b/docs/assets/tools-scrapegraphai.png new file mode 100644 index 0000000000..130f761f8c Binary files /dev/null and b/docs/assets/tools-scrapegraphai.png differ diff --git a/docs/tools/index.md b/docs/tools/index.md index 978975ea13..194e54368c 100644 --- a/docs/tools/index.md +++ b/docs/tools/index.md @@ -223,6 +223,16 @@ Check out the following pre-built tools that you can use with ADK agents: + +
+ ScrapeGraphAI +
+
+

ScrapeGraphAI

+

AI-powered web scraping, crawling, and data extraction

+
+
+
Tavily diff --git a/docs/tools/third-party/index.md b/docs/tools/third-party/index.md index 60969c64e0..44f878693f 100644 --- a/docs/tools/third-party/index.md +++ b/docs/tools/third-party/index.md @@ -89,6 +89,16 @@ Check out the following third-party tools that you can use with ADK agents:
+ +
+ ScrapeGraphAI +
+
+

ScrapeGraphAI

+

AI-powered web scraping, crawling, and data extraction

+
+
+
Tavily diff --git a/docs/tools/third-party/scrapegraphai.md b/docs/tools/third-party/scrapegraphai.md new file mode 100644 index 0000000000..7f1d821ee6 --- /dev/null +++ b/docs/tools/third-party/scrapegraphai.md @@ -0,0 +1,91 @@ +# ScrapeGraphAI + +The +[ScrapeGraphAI MCP Server](https://github.com/ScrapeGraphAI/scrapegraph-mcp) +connects your ADK agent to [ScrapeGraphAI](https://scrapegraphai.com/). This +integration enables your agent to extract structured data using natural language +prompts, handle dynamic content like infinite scrolling, and convert complex +webpages into clean, usable JSON or Markdown. + +## Use cases + +- **Scalable Extraction & Crawling**: Extract structured data from single pages + or crawl entire websites, leveraging AI to handle dynamic content, infinite + scrolling, and large-scale asynchronous operations. + +- **Research and Summarization**: Execute AI-powered web searches to research + topics, aggregate data from multiple sources, and summarize findings. + +- **Agentic Workflows**: Run advanced agentic scraping workflows with + customizable steps, complex navigation (like authentication), and structured + output schemas. + +## Prerequisites + +- Create an [API Key](https://dashboard.scrapegraphai.com/register/) in + ScrapeGraphAI. Refer to the + [documentation](https://docs.scrapegraphai.com/api-reference/introduction) for more information. +- Install the [ScrapeGraphAI MCP server + package](https://pypi.org/project/scrapegraph-mcp/) (requires Python 3.13 or + higher): + + ```console + pip install scrapegraph-mcp + ``` + +## Use with agent + +=== "Local MCP Server" + + ```python + from google.adk.agents import Agent + from google.adk.tools.mcp_tool.mcp_toolset import MCPToolset + from google.adk.tools.mcp_tool.mcp_session_manager import StdioConnectionParams + from mcp import StdioServerParameters + + SGAI_API_KEY = "YOUR_SCRAPEGRAPHAI_API_KEY" + + root_agent = Agent( + model="gemini-2.5-pro", + name="scrapegraph_assistant_agent", + instruction="""Help the user with web scraping and data extraction using + ScrapeGraph AI. You can convert webpages to markdown, extract + structured data using AI, perform web searches, crawl + multiple pages, and automate complex scraping workflows.""", + tools=[ + MCPToolset( + connection_params=StdioConnectionParams( + server_params=StdioServerParameters( + # The following CLI command is available + # from `pip install scrapegraph-mcp` + command="scrapegraph-mcp", + env={ + "SGAI_API_KEY": SGAI_API_KEY, + }, + ), + timeout=300, + ), + # Optional: Filter which tools from the MCP server are exposed + # tool_filter=["markdownify", "smartscraper", "searchscraper"] + ), + ], + ) + ``` + +## Available tools + +Tool | Description +---- | ----------- +`markdownify` | Transform any webpage into clean, structured markdown format +`smartscraper` | Leverage AI to extract structured data from any webpage with support for infinite scrolling +`searchscraper` | Execute AI-powered web searches with structured, actionable results +`scrape` | Basic scraping endpoint to fetch page content with optional heavy JavaScript rendering +`sitemap` | Extract sitemap URLs and structure for any website +`smartcrawler_initiate` | Initiate intelligent multi-page web crawling (asynchronous operation) +`smartcrawler_fetch_results` | Retrieve results from asynchronous crawling operations +`agentic_scrapper` | Run advanced agentic scraping workflows with customizable steps and structured output schemas + +## Additional resources + +- [ScrapeGraphAI MCP Server Documentation](https://docs.scrapegraphai.com/services/mcp-server) +- [ScrapeGraphAI MCP Server Repository](https://github.com/ScrapeGraphAI/scrapegraph-mcp) diff --git a/mkdocs.yml b/mkdocs.yml index 2d731c2f56..b060a034d2 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -186,6 +186,7 @@ nav: - GitHub: tools/third-party/github.md - Hugging Face: tools/third-party/hugging-face.md - Notion: tools/third-party/notion.md + - ScrapeGraphAI: tools/third-party/scrapegraphai.md - Tavily: tools/third-party/tavily.md - Agentic UI (AG-UI): tools/third-party/ag-ui.md - Custom Tools: