feat: add fastCRW web tool#884
Conversation
📝 WalkthroughWalkthrough
ChangesfastCRW Integration
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@legacy/app/modules/intelligence/tools/web_tools/webpage_extractor_tool.py`:
- Around line 44-52: The issue is that when `CRW_API_URL` is set to an empty
string, `os.getenv("CRW_API_URL", "https://fastcrw.com/api")` returns the empty
string instead of applying the default value, causing an empty `api_url` to be
passed to `FirecrawlApp` initialization on line 51, which breaks extraction. To
fix this, modify the assignment of `self.api_url` on line 46 to treat both None
(unset) and empty string values as triggers for using the default fastCRW cloud
endpoint by using a pattern that falls back to the default
"https://fastcrw.com/api" when the environment variable is either unset or
evaluates to an empty string.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro
Run ID: b8326d43-3388-4ba3-aaca-d9e6b549a703
📒 Files selected for processing (2)
legacy/.env.templatelegacy/app/modules/intelligence/tools/web_tools/webpage_extractor_tool.py
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
legacy/app/modules/intelligence/tools/web_tools/webpage_extractor_tool.py (1)
119-125:⚠️ Potential issue | 🟠 Major | ⚡ Quick winNormalize env vars in factory gating to match constructor logic.
At Line 120, raw
os.getenv(...)truthiness allows whitespace-only keys to pass gating, whileWebpageExtractorTool.__init__strips those values (Lines 42-43). That can register a tool that always fails extraction. BecauseToolServiceregisters any non-Nonetool instance, this becomes user-visible misbehavior.Suggested patch
def webpage_extractor_tool(sql_db: Session, user_id: str) -> Optional[StructuredTool]: - if not os.getenv("CRW_API_KEY") and not os.getenv("FIRECRAWL_API_KEY"): + crw_api_key = (os.getenv("CRW_API_KEY") or "").strip() + firecrawl_api_key = (os.getenv("FIRECRAWL_API_KEY") or "").strip() + if not crw_api_key and not firecrawl_api_key: logger.warning( "Neither CRW_API_KEY nor FIRECRAWL_API_KEY set, webpage extractor tool " "will not be initialized" ) return NoneAs per coding guidelines, “cross-file and configuration contract breaks” should validate actual configuration values and keep behavior consistent across layers.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@legacy/app/modules/intelligence/tools/web_tools/webpage_extractor_tool.py` around lines 119 - 125, The gating logic in the webpage_extractor_tool factory function at line 120 uses raw os.getenv() calls without stripping whitespace, allowing whitespace-only API keys to pass and create a tool instance. However, the WebpageExtractorTool.__init__ method (lines 42-43) strips these values, making them effectively empty. Normalize the gating logic by stripping the environment variable values before checking their truthiness, just as the constructor does. This ensures only non-whitespace API keys pass the gate and the tool will function properly when created.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@legacy/app/modules/intelligence/tools/web_tools/webpage_extractor_tool.py`:
- Around line 119-125: The gating logic in the webpage_extractor_tool factory
function at line 120 uses raw os.getenv() calls without stripping whitespace,
allowing whitespace-only API keys to pass and create a tool instance. However,
the WebpageExtractorTool.__init__ method (lines 42-43) strips these values,
making them effectively empty. Normalize the gating logic by stripping the
environment variable values before checking their truthiness, just as the
constructor does. This ensures only non-whitespace API keys pass the gate and
the tool will function properly when created.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro
Run ID: 6f234866-6b37-468e-925a-a5861ccbae01
📒 Files selected for processing (1)
legacy/app/modules/intelligence/tools/web_tools/webpage_extractor_tool.py
1 participant
What
Adds fastCRW as a web scrape/search provider, alongside the existing Firecrawl integration — additive, mirrors the Firecrawl wiring (Firecrawl untouched).
Why
fastCRW is a self-hostable, fully open-source web engine (AGPL, single ~8MB Rust binary) that outperforms Firecrawl on Firecrawl's own benchmark dataset — 63.74% truth-recall vs 56.04%, faster median latency (p50 ~1.9s vs ~2.3s) — and runs 100% locally without gating any capability behind a cloud plan.
Where it's materially better than Firecrawl's self-host:
fire-engine) is a cloud-only flag; its self-hosted version falls back to plain fetch or Playwright and can't handle Cloudflare JS challenges or heavily protected pages. fastCRW ships Cloudflare JS-challenge bypass, UA rotation, SPA rendering, BYO-proxy + rotation, and an HTTP→headless→proxy fallback ladder in the open core.The integration diff is small because fastCRW exposes a Firecrawl-compatible API — that's the only reason it's listed here, not the value prop.
Key via
CRW_API_KEY(free tier at https://fastcrw.com/dashboard); self-host base URL supported. I maintain the integration and can provide free credits to evaluate — happy to adjust to your conventions.