[ChronodeAi]

Adds an opt-in firecrawl web provider ([web] provider = "firecrawl"). It uses the httpx dependency the core already has, so there is no new dependency. Defaults and every existing provider are unchanged.

What it does

• fetch calls Firecrawl v2 /scrape and returns markdown, rawHtml (for DOI extraction), images (as media) and the screenshot, which it downloads through safe_get.
• fetch_many submits one /batch/scrape job, polls it until it finishes, and follows next pages.
• search calls /search with scrapeOptions set, so each result carries its page's markdown.
• Without FIRECRAWL_API_KEY, fetch and search use Firecrawl's keyless tier, and batches go one URL at a time.

Fallback to crawl4ai. The crawl4ai fallback is built with the vault's profile, magic and settings. Firecrawl cannot use a local login profile, so the following go to crawl4ai:

• the visible-browser lane
• allow_private_hosts hosts, which are never sent to Firecrawl
• Firecrawl API errors (402, 429, network)
• pages where the target answered 401/403/407/429/5xx, or that look like a login wall or bot-detection page

A 404/410 is raised and not retried. If the fallback also fails on a page Firecrawl blocked on, the Firecrawl result is returned, so the existing junk gate and escalation queue still run.

Invariants kept

• PDFs go through the shared PDF lane first, so raw_file is still saved.
• The SSRF entry gate is applied.
• skipTlsVerification is set to not browser_verify_tls, because Firecrawl defaults to skipping verification (crawl4ai browser lanes fetch with TLS verification off #137).

Provenance. fetch_provider now comes from served_by(prov, result). That is the name of the provider that actually served the page (crawl4ai for a handed-off URL). For every other provider it equals prov.name, as before.

Tests. 18 offline tests in tests/test_web/test_firecrawl_provider.py use httpx.MockTransport, a fake fallback provider and stubbed DNS. The full suite passes (1737). A live smoke test against the keyless tier covered fetch (Wikipedia: 92k chars plus screenshot), search, and fetch-batch. In the batch, a 403 URL was handed to crawl4ai, matching crawl4ai's own batch behaviour.

Setup wizard. hyperresearch setup step 1 is now a menu listing crawl4ai, firecrawl and builtin. A vault on any other provider also gets that provider as an option, so re-running setup doesn't replace exa or tavily without asking. Choosing firecrawl reports whether FIRECRAWL_API_KEY is set. It also keeps the browser-profile step, stealth and the Chromium install, because the crawl4ai fallback uses them. Re-running setup defaults to the vault's current provider. Covered by 4 wizard tests with scripted prompts.

[jordan-gibbs]

Thanks for this. The provider code looks good with no new dependency, sensible batch scrape use, and rawHtml for DOI extraction. Could you split it so it's easier to take?

1. Keep the provider on its own without the setup-wizard rewrite in cli/setup.py, since that lists Firecrawl as option 2 for every user and steers people toward one vendor.
2. Put the served_by provenance in its own small PR since that's useful on its own.
3. Make the keyless tier an explicit opt-in. Sending URLs to Firecrawl shouldn't happen just because the provider is selected without a key.