Ferret

Get the Claygent experience with your existing stack, outside of Clay.

A simple, open-source web research agent. Send a prompt + JSON Schema, get structured data back. Runs on Cloudflare Workers. Call it from anywhere — AI coding sessions, Clay HTTP Request columns, n8n, Make, or any tool that can POST JSON.

You need four things:

A Scraping API (or just use Cloudflare's free native fetch)
A SERP API (search engine — Google, Bing, Exa, Yandex, etc.)
An LLM API (any provider with tool calling)
A Cloudflare account (free tier works)

Point your coding agent (Claude Code, Codex, Cursor) to AGENT.md and tell it to fit Ferret to your stack.

Writing prompts and schemas that hold up at 10K-row scale: see PROMPTING.md — scope control, verification patterns, schema design rules, all tested. Ready-made recipes in examples/.

Claude Code users: the repo ships a ferret-prompt-test skill — open this repo in Claude Code and ask it to test your prompt; it runs your prompt against diverse real inputs, diagnoses failures from the agent_log, and iterates until results are reliable.

How It Works

POST { prompt, schema }
        │
        ▼
   ┌─────────┐
   │   LLM    │──── web_search (search engine) ───► search results
   │  agent   │──── fetch_page ───► native fetch ───► CF Browser ───► scraping API
   │  loop    │
   └────┬─────┘
        │
        ▼
  { structured JSON matching your schema }

The LLM decides what to search and what pages to read. It calls tools autonomously until it has enough information, then returns JSON matching your schema.

Page fetching cascades through 5 tiers to minimize cost:

Tier	Method	Cost
1	Native `fetch()` from Cloudflare edge	Free
2	Cloudflare Browser Rendering	Browser seconds (no external credits)
3+	Scraping API fallback (JS rendering, proxies)	Varies

Most pages resolve at tier 1. Ships with scrape.do as the fallback scraper, but you can replace it with your preferred provider (Zenrows, ScrapingBee, Spider.cloud, etc.) — ideally one with JS rendering and proxies for hard-to-reach sites.

Built-in cost and reliability controls:

Dead-URL stop + recovery — a 404/410/401 from the target stops the cascade immediately (escalating to paid tiers can't make a missing page appear), then the agent recovers: it searches for the live page, navigates from the homepage, or falls back to another reliable source — primary sources first, since third-party data goes stale
Site navigation — every fetched page ends with a list of same-site links, so the agent follows real URLs instead of guessing paths
Per-request cache — re-fetching a URL or repeating a search is free and doesn't count against the tool budget
Context compaction — older tool results are truncated so input tokens don't grow quadratically across rounds
Schema validation — required fields, types, and enums are checked on the final JSON; the agent gets one cheap retry round to fix any problems (no more "about 120" in your number columns)
Deadline support — a 120s default time budget (override per request with deadline_ms) makes the agent wrap up early instead of timing out your HTTP client

Prerequisites

You need three things:

1. Cloudflare account (free)

Sign up at cloudflare.com
Install Wrangler (Cloudflare's CLI):
```
npm install -g wrangler
```
Log in:
```
wrangler login
```
Find your Account ID — run wrangler whoami or find it in the Cloudflare dashboard URL: dash.cloudflare.com/<account-id>/...

The free Workers plan gives you 100,000 requests/day. Paid plan ($5/mo) unlocks higher limits and Browser Rendering.

2. LLM API key

Get an API key from any OpenAI-compatible provider:

Provider	Sign up	Cost
DeepSeek (default)	platform.deepseek.com	~$0.14/M input, $0.28/M output
OpenRouter	openrouter.ai	Varies by model, many free models
OpenAI	platform.openai.com	~$0.15/M input (gpt-4o-mini)
Groq	console.groq.com	Free tier available
Together	api.together.xyz	~$0.18/M input (Llama 3.3 70B)

If you use a provider other than DeepSeek, see AGENT.md to swap the URL (one line change).

3. Search API key

Get a RapidAPI key for Google Search:

Sign up at rapidapi.com
Subscribe to one of the search providers listed below
Your API key is in the RapidAPI dashboard under Apps → default-application → Authorization

Quick Start

git clone https://github.com/edsmkt/Ferret.git
cd Ferret

# Set your secrets (LLM API — works with any OpenAI-compatible provider)
echo "DEEPSEEK_API_KEY=sk-your-key" > .dev.vars
echo "RAPIDAPI_KEY=your-rapidapi-key" >> .dev.vars

# Optional (fallback scraping — see scrape.do pricing below)
echo "SCRAPE_DO_TOKEN=your-token" >> .dev.vars

# Run locally
wrangler dev

# Deploy to Cloudflare
wrangler deploy

After deploying, your worker runs at https://ferret.<your-subdomain>.workers.dev.

Setting secrets for production

Local development reads from .dev.vars. For production, set secrets via Wrangler:

wrangler secret put DEEPSEEK_API_KEY
wrangler secret put RAPIDAPI_KEY
wrangler secret put SCRAPE_DO_TOKEN        # optional
wrangler secret put WORKER_AUTH            # strongly recommended — see Production Hardening

Optional: Cloudflare Browser Rendering

This enables Tier 2 page fetching — a headless browser on Cloudflare's edge for JS-heavy sites. Requires the Workers paid plan ($5/mo).

Go to Cloudflare dashboard → My Profile → API Tokens → Create Token
Create a Custom Token with the permission: Account → Browser Rendering → Edit

Set the secrets:

# For local dev, add to .dev.vars:
echo "CF_ACCOUNT_ID=your-account-id" >> .dev.vars
echo "CF_API_TOKEN=your-token" >> .dev.vars

# For production:
wrangler secret put CF_ACCOUNT_ID
wrangler secret put CF_API_TOKEN

If not configured, Ferret simply skips this tier — native fetch → scrape.do still works.

API

Request

POST /
Content-Type: application/json

{
  "prompt": "Research Acme GmbH. Find what they do, their pricing, and who the CEO is.",
  "schema": {
    "type": "object",
    "required": ["company", "what_they_do", "ceo"],
    "properties": {
      "company": { "type": "string" },
      "what_they_do": { "type": "string", "description": "1-2 sentence summary" },
      "pricing_url": { "type": "string" },
      "ceo": { "type": "string" },
      "employee_count": { "type": "integer" }
    }
  }
}

The schema field accepts either a JSON Schema (with type, properties, required, description, enum) or a plain example object — Ferret handles both. With a JSON Schema, the final output is validated — required fields present, values matching their declared types, enum values legal — and the agent gets one retry round to fix any problems.

Optional fields:

Field	Description
`deadline_ms`	Soft time budget in milliseconds. The agent stops researching and returns its best answer before the deadline. Defaults to 120000 (2 min) — set it lower when calling from tools with HTTP timeouts (Clay ~45000), or pass `0` to disable the deadline entirely. The default is configurable via the `DEFAULT_DEADLINE_MS` var in `wrangler.toml`.
`max_fetches`	Per-request tool-call budget override (clamped to 1–20). Lets one deployed worker serve both light lookup columns (`5`) and deep research columns (`15`) without separate deploys. Defaults to the `MAX_FETCHES` var (10).

Response

{
  "result": {
    "company": "Acme GmbH",
    "what_they_do": "B2B industrial supplies manufacturer based in Munich.",
    "pricing_url": "https://acme.de/pricing",
    "ceo": "Hans Mueller",
    "employee_count": 120
  },
  "sources": ["https://acme.de", "https://acme.de/about"],
  "agent_log": [
    { "step": "web_search", "query": "Acme GmbH", "via": "rapidapi", "status": 200, "cost": 0 },
    { "step": "fetch_page", "url": "https://acme.de", "via": "native", "status": 200, "cost": 0, "chars": 8000, "purpose": "Find what the company does and locate pricing/about links" },
    { "step": "fetch_page", "url": "https://acme.de/about", "via": "native", "status": 200, "cost": 0, "chars": 5200, "purpose": "Find the CEO name and company size" },
    { "step": "done", "rounds_total": 3, "fetches_used": 4 }
  ],
  "scrape_credits_total": 0,
  "tokens_in": 14250,
  "tokens_out": 820,
  "duration_ms": 21400,
  "model": "deepseek-v4-flash"
}

Field	Description
`result`	The structured JSON matching your schema
`sources`	The pages the agent actually read — audit where answers came from
`agent_log`	Every step taken: every search query, every fetch with the agent's stated `purpose`, which tier handled it, the cost
`scrape_credits_total`	Paid scraping credits spent on this request
`tokens_in` / `tokens_out`	Total LLM tokens — multiply by your provider's rates for exact per-row cost
`duration_ms`	Wall-clock time for the whole research run

If the run fails mid-research (e.g. the LLM provider goes down), the response includes an error field plus everything gathered up to that point — the agent_log is never lost.

Calling Ferret from your stack

Ferret is a plain HTTP endpoint — anything that can POST JSON can use it. The sweet spot is workflows outside spreadsheet-style tools: n8n/Make automations, scripts, batch jobs, and AI agents, where nothing times out and the agent can research as thoroughly as the task needs (deadline_ms: 0, max_fetches up to 20).

n8n / Make: add an HTTP Request node — POST, JSON body with prompt + schema, header x-worker-key. n8n waits as long as needed, so thorough runs are fine. Map result.* fields from the response into the rest of your workflow.

Scripts / AI agents: see the curl examples in examples/ — each recipe is a complete request body. Coding agents can also use this repo's ferret-prompt-test skill to develop and validate prompts.

Clay (works, with one caveat — Clay's HTTP column times out around 30-60s):

Method: POST
URL: https://ferret.your-subdomain.workers.dev
Headers: Content-Type: application/json
Body:

{
  "prompt": "Research {{Company Name}} at {{Website}}. Classify their industry and find the CEO.",
  "schema": {
    "industry": "string",
    "ceo_name": "string",
    "employee_count": "number"
  },
  "deadline_ms": 45000
}

Clay resolves {{placeholders}} before sending — Ferret receives the full prompt with real values. Set deadline_ms to ~45000 so the agent answers before Clay's HTTP timeout instead of returning an error on slow research runs.

Production Hardening

Set WORKER_AUTH — an open endpoint is an open wallet

Without it, anyone who finds your worker URL can burn your LLM and scraping credits. Treat this as required for any deployed instance:

wrangler secret put WORKER_AUTH

Then include in every request:

x-worker-key: your-secret-key

Rate limiting

Even with auth, a leaked key or a runaway Clay table can hammer your worker. Add a Cloudflare rate limiting rule on your worker's route (e.g. 60 requests/minute per IP) — no code changes needed.

Subrequest limits (free tier)

Free-tier Workers allow 50 subrequests per invocation. A worst-case run (10 fetches × full 5-tier cascade + LLM retries) can exceed that and fail mid-research. The paid plan ($5/mo) allows 1,000, which Ferret can't realistically hit. If you stay on free tier, set MAX_FETCHES=5.

Configuration

Environment Variables

Variable	Required	Description
`DEEPSEEK_API_KEY`	Yes	LLM API key. Ships with DeepSeek but works with any OpenAI-compatible provider (OpenRouter, OpenAI, Groq, Together, Mistral) — just swap the URL in `worker.js`. See AGENT.md.
`RAPIDAPI_KEY`	Yes	RapidAPI key for Google Search
`SCRAPE_DO_TOKEN`	No	scrape.do token (fallback scraping)
`CF_ACCOUNT_ID`	No	Cloudflare account ID (Browser Rendering)
`CF_API_TOKEN`	No	Cloudflare API token with Browser Rendering permission
`WORKER_AUTH`	Strongly recommended	Secret key to protect your endpoint — without it anyone with the URL can spend your credits

Wrangler Vars

Variable	Default	Description
`MODEL`	`deepseek-v4-flash`	LLM model to use
`MAX_FETCHES`	`10`	Max tool calls per request — tune to your workload, see AGENT.md
`MAX_TOKENS`	`8000`	Max LLM output tokens — large schemas need headroom or the JSON truncates
`DEFAULT_DEADLINE_MS`	`120000`	Default soft time budget when the request doesn't pass `deadline_ms`

Search API Providers

Ferret uses RapidAPI Google Search by default. Here are providers ranked by cost:

RapidAPI Providers

Provider	Subs	Latency	Uptime	Rate Limit	100K Cost	Per 1K	Notes
ScraperLink	628	3,914ms	100%	1 req/s	$20	$0.04	Cheapest but slow, 5s per request. 1 req/s rate limit.
Winbay Tech	180	6,501ms	100%	10 req/s	$10	$0.10	Cheap. Low price plan. Bad latency. But 10 req/s rate limit.
Scrappa	221	739ms	100%	1 req/s	$25	$0.25	Good speed, but 1 req/s rate limit.
FlyByAPIs	83	435ms	100%	10 req/s	$50	$0.50	Good speed, but 2x Scrappa's cost. But 10 req/s rate limit.

Other Providers

Provider	Plan	100K Cost	Per 1K	Speed	Notes
Scrappa.co (PAYG)	$25 Basic (86K credits)	$27	$0.29	N/A	PAYG, credits valid 12 months. 80+ endpoints. $0.20/1K at $1,000 pack
Scrapingdog Light	$90/mo Standard	$45	$0.45	1.25-1.83s	Basic organic only (position, title, URL, snippet)
Scrapingdog Advanced	$90/mo Standard	$90	$0.90	1.25-1.83s	Full output: PAA, AI Overview, ads, local pack, related searches
HasData Light	$99/mo Business	$50	$0.50	1.75-2.3s	Organic only. 200K searches at $0.50/1K
HasData Full	$99/mo Business	$99	$0.99	1.75-2.3s	15+ features: PAA, AI Overview, ads, local pack, knowledge graph
SearchCans (PAYG)	$99 Starter	$75	$0.75	1-1.5s	PAYG, credits valid 6 months. Zero independent reviews
Serper.dev (PAYG)	$50	$100	$1.00	1.83-2.87s	PAYG, credits valid 6 months. Gets cheaper at 500K+ ($0.50/1K)

AI-Powered Search & Retrieval

These aren't traditional SERP scrapers — they're AI-native search APIs that return cleaner, more relevant results. Good alternatives if you want higher quality over raw Google results.

Provider	Pricing	Free Tier	Notes
Exa	$7/1K searches, $1/1K pages	1,000 free requests/mo	Neural search engine. Returns clean content, not just links. Great for finding similar companies or specific content types.
Tavily	$8/1K credits (PAYG)	1,000 free credits/mo	Built for AI agents. Returns pre-extracted content with each result — less need for follow-up page fetches.
Jina AI Reader	Token-based (10M free tokens)	Free without API key (20 RPM)	Not a search engine — converts any URL to clean markdown. Use as a `fetch_page` replacement or alongside search. `r.jina.ai/<url>` returns markdown directly.

To swap providers, give AGENT.md to your coding agent (Claude Code, Codex, Cursor, etc.) and ask it to swap to your preferred provider. It has the architecture map, contracts, and drop-in code for each component.

Scraping Providers (page fetching fallback)

Ferret uses native Cloudflare fetch first (free). These are fallback options for when sites block direct requests:

Provider	Plan	Credits	Per 1K	Concurrency	Notes
scrape.do Free	$0/mo	1,000	Free	5	Good for testing
scrape.do Hobby	$29/mo	250K	$0.11	10	Personal/non-commercial
scrape.do Pro	$99/mo	1.25M	$0.08	50	Teams and power users
scrape.do Business	$249/mo	3.5M	$0.07	100	Dedicated account manager
scrape.do Advanced	$699/mo	10M	$0.06	200	Custom WAF bypass, SLA

Note: JS rendering (render=true) costs 5 credits per call, super mode costs 25 credits. Most pages resolve via free native fetch — scrape.do is rarely needed.

Other Scraping Options

Provider	Pricing	Notes
Crawl4AI	Free (open source)	Self-hosted async web crawler. Returns clean markdown. Run it on your own server and point `fetchPage()` at it.
Spider.cloud	From $0.10/1K pages	Fast managed scraper with JS rendering, anti-bot bypass, and markdown output. Good drop-in for scrape.do.
Jina AI Reader	Free tier available	`r.jina.ai/<url>` returns any page as clean markdown — zero config, works as a one-liner `fetch_page` replacement.

Cost Comparison

Claygent

Locked to Clay — can only use inside Clay tables
Limited models to choose from

Ferret (self-hosted)

Search: $0.04-$1.00 per 1K searches depending on provider
Page fetching: Usually free (native fetch). Fallback scraping ~1-25 credits when needed
LLM: DeepSeek at ~$0.14/M input tokens, $0.28/M output tokens
Hosting: Cloudflare Workers free tier (100K requests/day) or $5/mo paid

Typical cost per research call: $0.001 - $0.01 depending on how many searches/pages the agent needs.

License

MIT

Ferret is an open-source alternative to Claygent, Clay's AI web research agent. If you're looking for a self-hosted Claygent replacement that works with any LLM, any search API, and any scraper — this is it.

Name		Name	Last commit message	Last commit date
Latest commit History 43 Commits
.claude/skills/ferret-prompt-test		.claude/skills/ferret-prompt-test
examples		examples
.gitignore		.gitignore
AGENT.md		AGENT.md
LICENSE		LICENSE
PROMPTING.md		PROMPTING.md
README.md		README.md
test.sh		test.sh
worker.js		worker.js
wrangler.toml		wrangler.toml

Folders and files

Latest commit

History

Repository files navigation

Ferret

How It Works

Prerequisites

1. Cloudflare account (free)

2. LLM API key

3. Search API key

Quick Start

Setting secrets for production

Optional: Cloudflare Browser Rendering

API

Request

Response

Calling Ferret from your stack

Production Hardening

Set WORKER_AUTH — an open endpoint is an open wallet

Rate limiting

Subrequest limits (free tier)

Configuration

Environment Variables

Wrangler Vars

Search API Providers

RapidAPI Providers

Other Providers

AI-Powered Search & Retrieval

Scraping Providers (page fetching fallback)

Other Scraping Options

Cost Comparison

Claygent

Ferret (self-hosted)

License

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages