feat: add fastCRW support#73
Conversation
|
@us is attempting to deploy a commit to the joschan21's projects Team on Vercel. A member of the Team first needs to authorize it. |
📝 WalkthroughWalkthroughA new Firecrawl-compatible web scraping integration is added. ChangesFastcrw client and environment wiring
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
environment.d.ts (1)
23-23: ⚡ Quick winAlign
CRW_API_URLtyping with actual optional runtime behavior.
CRW_API_URLis documented and consumed as optional (defaulted insrc/lib/crw.ts), but this type marks it required. Make it optional here (and insrc/server/jstack.ts) to keep the contract consistent.Suggested diff
- CRW_API_URL: string + CRW_API_URL?: string🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@environment.d.ts` at line 23, The CRW_API_URL property is typed as a required string in environment.d.ts, but it is actually optional at runtime with a default value provided in src/lib/crw.ts. Make CRW_API_URL optional in both environment.d.ts and src/server/jstack.ts by changing the property definition from CRW_API_URL: string to CRW_API_URL?: string to align the type signature with the actual optional runtime behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@environment.d.ts`:
- Line 23: The CRW_API_URL property is typed as a required string in
environment.d.ts, but it is actually optional at runtime with a default value
provided in src/lib/crw.ts. Make CRW_API_URL optional in both environment.d.ts
and src/server/jstack.ts by changing the property definition from CRW_API_URL:
string to CRW_API_URL?: string to align the type signature with the actual
optional runtime behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 4bf089ea-4aaf-49cb-99a7-27e6f9675ea3
📒 Files selected for processing (4)
.env.exampleenvironment.d.tssrc/lib/crw.tssrc/server/jstack.ts
1 participant
What
Adds fastCRW as a web scrape/search provider, alongside the existing Firecrawl integration — additive, Firecrawl untouched.
Why
fastCRW is a fully open-source web scraping and search engine (AGPL, single ~8MB Rust binary) that outperforms Firecrawl on Firecrawl's own benchmark dataset: truth-recall 63.74% vs 56.04%, faster median latency (p50 ~1.9s vs ~2.3s), and ~6MB RAM at idle.
Runs 100% locally — including on protected/JS-heavy sites. Anti-bot bypass (Cloudflare JS-challenge), BYO-proxy + rotation, UA rotation, and SPA rendering all ship in the open core. Firecrawl's OSS self-host gates its stealth engine (
fire-engine) behind a cloud-only flag, so self-hosted Firecrawl falls back to plain fetch/Playwright and can't reliably reach protected sites. fastCRW's open core handles all of it — one binary, no cloud required.On web search: crw is not an alternative to SearXNG — it is built on top of it. SearXNG is the metasearch aggregator underneath; crw adds a quality layer on top: query expansion (multi-variant rewrite), content-aware reranking (re-scoring by fetched content instead of SearXNG's content-blind ordering), a calibrated direct-answer mode, and category routing (research queries fan out to arxiv / semantic scholar / google scholar, code queries to GitHub). You get SearXNG's breadth plus a measurable accuracy layer — all open-source (AGPL) and self-hostable with configurable engines.
Flat pricing: 1 credit = 1 page; no 4x stealth surcharge, no billed-on-failure. Free tier at https://fastcrw.com/dashboard (
CRW_API_KEY); self-host base URL supported viaCRW_BASE_URL.The integration diff is intentionally small because fastCRW exposes a Firecrawl-compatible API — same endpoints, same request/response shape, so the wiring mirrors the existing Firecrawl provider with minimal delta.
Happy to adjust to your conventions — I maintain the integration and can provide free credits to evaluate.
Summary by CodeRabbit