feat(scraper): add pluggable provider with crawl4ai backend - #45
Merged
Conversation
Wire the registry from Tasks 1-3 into the existing pipeline. discover.py and fetch.py now take DiscoveryProvider/FetchProvider as a parameter and no longer import the Firecrawl SDK directly. cli.py adds a --provider flag, resolves both stages via resolve_provider_name() before any I/O, and surfaces clean exits: 2 for unknown provider names, 3 for ProviderUnavailableError with the install hint. config.py reads the new SCRAPE_PROVIDER / SCRAPE_DISCOVER_PROVIDER / SCRAPE_FETCH_PROVIDER envs. Existing tests drop the FirecrawlApp module-level patches (the symbol no longer exists in those modules) and inject fake providers instead. Adds tests/test_scraper/test_pipeline_injection.py with 5 cases: default firecrawl, flag sets env, stage env overrides flag, unknown provider exits 2, ProviderUnavailableError exits 3. Zero breaking change: with no env and no flag, the pipeline still resolves 'firecrawl' for both stages, identical to the prior behavior.
Task 1 of the pluggable scraper provider work moved firecrawl-py from core dependencies to the [firecrawl] extra. Without updating the npm installer, every npx @king-context/cli init and update would silently install a Firecrawl-less venv and break the default scraper. Switch both install and upgrade calls to PEP 508 direct reference form "king-context[all] @ git+https://...king-context.git" so init keeps pulling firecrawl-py and crawl4ai together as before.
New .github/workflows/test.yml runs pytest on the firecrawl install path and adds a gated smoke-crawl4ai job that installs the crawl4ai extra, runs crawl4ai-setup, and executes scripts/smoke-crawl4ai.py. The smoke job is intentionally blocking (ADR-0011): detecting API churn between crawl4ai>=0.8.5,<0.9 minor versions before release is the entire reason it exists.
README gains a Scraper providers section with the env-var table and links to the CLI guide. docs/CLI_GUIDE.md adds the --provider flag entry plus a subsection covering resolution rules, four scenario examples (default, Crawl4AI, mixing, resume), and a troubleshooting block for the four canonical errors. .env.example and installer/templates/env.example add commented SCRAPE_PROVIDER, SCRAPE_DISCOVER_PROVIDER, and SCRAPE_FETCH_PROVIDER entries, with em-dashes in pre-existing comments swept out. CHANGELOG records the abstraction, Crawl4AI extra, CI workflow, firecrawl-py move to extras, and the installer PEP 508 update under Unreleased.
The previous Scraper providers section showed only the standalone pip install path, which was misleading for the majority flow where the npm installer already bundles crawl4ai via the [all] extra. Adds: - A brief paragraph on why Crawl4AI was added (no external credits, no third-party HTML pass-through, transparent conversion, private docs friendly) - A beta notice with a link to file issues - Two distinct enable paths: one for npx-init users (just run crawl4ai-setup from the project venv) and one for standalone pip installs - Same env-var table and mixing example as before
king-context is not on PyPI yet (the standalone distribution is on the
roadmap), so 'pip install king-context[...]' was misleading users into
a command that does not work today.
Replaced with the two paths that actually work:
- npx-installed projects: the [all] extra is bundled by
'npx @king-context/cli init', so users only need 'crawl4ai-setup'.
- Cloned-repo developers: 'pip install -e ".[crawl4ai]" && crawl4ai-setup'
from the repo root.
Updated:
- README.md and docs/CLI_GUIDE.md (Scraper providers section,
troubleshooting block)
- .env.example and installer/templates/env.example (comments)
- CHANGELOG.md [Unreleased] entries
- src/scraper_providers/{firecrawl,crawl4ai}_provider.py: _INSTALL_HINT
strings now point at the npx and dev-clone paths instead of the fake
PyPI command. _SETUP_HINT for crawl4ai-setup is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a pluggable scraper provider abstraction to
king-scrape, with Crawl4AI as the first local opt-in backend. Firecrawl remains the zero-config default.Why?
king-scrapewas Firecrawl-only, contradicting the local-first positioning (ADR-0001). Users without Firecrawl credits, with private docs, or who wanted transparent HTML to Markdown conversion had no path. Mirrors the LLM provider pattern from ADR-0003.Decisions: ADR-0009 (provider abstraction), ADR-0010 (thin provider, pipeline-owned IO), ADR-0011 (Crawl4AI as first local backend).
What changed
src/scraper_providers/(new, 5 files): registry,DiscoveryProvider/FetchProviderProtocols,PageContentdataclass,ProviderUnavailableError, plus Firecrawl and Crawl4AI implementations with soft import.src/king_context/scraper/{cli,config,discover,fetch}.py):--providerCLI flag, stage-aware env resolution (SCRAPE_PROVIDER,SCRAPE_DISCOVER_PROVIDER,SCRAPE_FETCH_PROVIDER), provider injection. Checkpoint, slug paths, and manifest semantics preserved.pyproject.toml:firecrawl-pymoved to[firecrawl]extra. New[crawl4ai]and[all]extras.entry_pointsgroupking_context.scraper_providersfor future plugin discovery.installer/lib/python.js: install/upgrade now useking-context[all] @ git+...so init keeps pulling firecrawl-py after the extras move..github/workflows/test.ymlwithtest(pytest) and gatedsmoke-crawl4aijob (crawl4ai-setup+scripts/smoke-crawl4ai.py). Blocking by design (ADR-0011) to detect API churn acrosscrawl4ai>=0.8.5,<0.9minor versions..env.example+ installer template synced, CHANGELOG[Unreleased]entry.tests/test_scraper_providers/(registry, soft import, stage resolution, entry points, both provider clients with mocked SDKs). Existing scraper tests adapted for provider injection.Validation
pytest -q: 683 passedfirecrawland reaches the API (zero breaking change)kctx adr validate: passed