Version: 3.0.4
Author: lexiismadd
This tool enables your Open WebUI instance to not only search the internet but to deeply crawl, extract, and summarise content from web pages. It combines the power of search engines (SearXNG and Open WebUI Native Search) with the advanced extraction capabilities of Crawl4AI.
- Dual-Engine Search: Simultaneously utilises SearXNG and OpenWebUI Native Search to find the most relevant URLs.
- Intelligent Crawling: Powered by Crawl4AI, it extracts clean, markdown-formatted content while stripping away ads, sidebars, and navigation clutter.
- Research Mode: Recursively follows links on discovered pages to perform "Deep Research" based on keyword relevance using Crawl4AI's Deep Crawl feature. (User-specific setting)
- LLM-Driven Extraction: Uses an OpenAI-compatible LLM (like GPT-4, Claude, or Ollama) to summarise and structure the crawled data before it reaches your chat.
- Rich UI Image Gallery: Automatically identifies and displays high-quality images in an interactive gallery using Open WebUI's Rich UI embedding.
- Smart Token Management: Includes automatic content truncation and token counting (via
tiktoken) to ensure responses stay within model limits. - Concurrent Validation: Validates image URLs in batches to ensure only "live" media is displayed.
- Highly Configurable: Offers many configuration valves to tune the tool to your preference and environment.
- Trigger: When you ask a question requiring real-time data, the tool is called.
- Gathering: It queries SearXNG or Native Search and combines those results with any specific URLs you provided.
- Crawling: It sends those URLs to a self-hosted Crawl4AI instance.
- Extraction: The LLM configured in the "Valves" processes the raw HTML/Markdown into a structured summary focusing on your core instructions.
- Delivery: The tool returns a clean list of summaries with citations, and (in rich mode) an interactive gallery of relevant images.
| Category | Key | Default | Description |
|---|---|---|---|
| General | Initial delta response |
"Searching..." | The message shown in chat when the tool starts working. |
| Search | Use Native Search |
true |
Enable/Disable OpenWebUI's internal search. |
| Search | Search with SearXNG |
false |
Enable/Disable SearXNG integration. |
| Search | SearXNG Search URL |
(see default) | SearXNG Search query URL with <query> placeholder. |
| Search | SearXNG API Token |
"" | The API token or Secret for your SearXNG instance. |
| Search | SearXNG HTTP Method |
"GET" | HTTP method to use for SearXNG API calls (GET or POST). |
| Search | SearXNG Timeout |
30 | The timeout (in seconds) for SearXNG API requests. |
| Search | SearXNG Max Results |
10 | The maximum number of results to return from SearXNG. |
| Crawl4AI | Crawl4AI Base URL |
"http://crawl4ai:11235" | The URL of your Crawl4AI Docker/Server instance. |
| Crawl4AI | Crawl4AI API Token |
"" | API token for authenticating with your Crawl4AI instance. |
| Crawl4AI | Crawl4AI User Agent |
(Chrome UA) | Custom User-Agent string for Crawl4AI. |
| Crawl4AI | Crawl4AI Timeout |
60 | The timeout (in seconds) for Crawl4AI requests. |
| Crawl4AI | Crawl4AI Batch |
5 | The number of URLs to send to Crawl4AI per batch. |
| Crawl4AI | Crawl4AI Maximum URLs to crawl |
20 | The maximum number of URLs to crawl with Crawl4AI. |
| Crawl4AI | Crawl External Domains |
false |
Allow Crawl4AI to crawl external/additional URL domains. |
| Crawl4AI | Excluded Domains |
"" | Comma-separated list of external domains to exclude from crawling. |
| Crawl4AI | Excluded Social Media Domains |
(common social sites) | Comma-separated list of social media domains to exclude. |
| Crawl4AI | Exclude Images |
"None" | Exclude images from crawling (None, External, All). |
| Crawl4AI | Word Count Threshold |
200 | The minimum word count threshold for content to be included. |
| Crawl4AI | Text Only |
false |
Only extract text content, excluding images and other media. |
| Crawl4AI | Max Images |
8 | Maximum number of images to include in results. Set to 0 for no images. |
| Crawl4AI | Min Image Score To Include |
6 | Minimum image score (0-10) from Crawl4AI to include. |
| Crawl4AI | Validate Image Links |
true |
Validate image links to ensure they are accessible. |
| Crawl4AI | Max Tokens used by web content |
0 | Maximum tokens for web content response (0 = unlimited). |
| Output | Response Mode |
"rich" | "rich" = Interactive HTML gallery. "text" = Plain markdown with citations. |
| Output | Max Output Characters |
50000 | Maximum characters in final output (0 = unlimited). |
| LLM | LLM Base URL |
"https://openrouter.ai/api/v1" | The base URL for your preferred OpenAI-compatible LLM. |
| LLM | LLM API Token |
"" | Optional API Token for your preferred OpenAI-compatible LLM. |
| LLM | LLM Provider |
"openrouter/auto" | The LLM provider and model to use. |
| LLM | LLM Temperature |
0.3 | The temperature to use for the LLM. |
| LLM | LLM Extraction Instruction |
(see code) | The instruction to use for the LLM when extracting from the webpage. |
| LLM | LLM Max Tokens |
4096 | The maximum number of tokens to use for the LLM. |
| LLM | LLM Top P |
null | The top_p value to use for the LLM. |
| LLM | LLM Frequency Penalty |
null | The frequency penalty to use for the LLM. |
| LLM | LLM Presence Penalty |
null | The presence penalty to use for the LLM. |
| Debug | More status updates |
false |
Show more status updates during web search and crawl. |
| Debug | Debug logging |
false |
Enable detailed debug logging. |
These settings can be configured per-user:
| Key | Default | Description |
|---|---|---|
SearXNG Max Results |
null | Override max results from SearXNG. |
Crawl4AI Maximum URLs to crawl |
null | Override max URLs to crawl. |
Max Images |
null | Override max images to include. |
Research Mode |
false |
Enable research mode using Crawl4AI with Deep Crawling. |
Research Crawl Mode |
"pseudo_adaptive" | Crawling strategy: pseudo_adaptive, llm_guided, bfs_deep, or research_filter. |
Keyword Relevance Weight |
0.7 | The keyword relevance weight when using Research mode. |
Max Depth |
2 | Maximum depth of links to follow (max 10). |
Max Pages |
15 | Maximum pages to crawl in Research mode (max 25). |
Research Batch Size |
5 | Number of URLs to process per batch during research. |
LLM Link Selection |
true |
Use LLM to select next links when in llm_guided mode. |
Include External |
false |
Allow following external domains during research crawling. |
To use this tool, your environment must have:
- Crawl4AI Server: A running instance of Crawl4AI (usually via Docker).
- SearXNG: An accessible instance of SearXNG (if using SearXNG).
- OpenWebUI: A recent version (v0.9.5 or higher) to support Native Search and Tools.
Standard Search:
"Find the latest news on SpaceX Starship and summarise the key findings."
Targeted Crawl:
"Crawl https://example.com and tell me their pricing structure."
Research Mode:
"Perform a deep research search on 'Ambient Computing' and find at least 10 sources."
Please log any issues on my Github repo