feat(scraper): SCRAPE_CACHE_MODE and --no-fetch-cache for crawl4ai (#50)

[nelsonmfinda]

Summary

Fix #50: adds --no-fetch-cache flag and SCRAPE_CACHE_MODE env var to bypass Crawl4AI's local cache (~/.crawl4ai/) without wiping the directory by hand. Maps to Crawl4AI's CacheMode enum and is plumbed into both the discover and fetch paths via CrawlerRunConfig. CLI flag is shorthand for SCRAPE_CACHE_MODE=bypass and uses setdefault semantics so an explicit pre-existing env value wins (mirrors --provider's precedence). main() restores the prior env value on exit so the flag does not leak into a test session or an embedding application.

Honoured by the crawl4ai provider; firecrawl ignores it (its API defaults to fresh-fetch). Knob is global today; per-stage variants (SCRAPE_DISCOVER_CACHE_MODE / SCRAPE_FETCH_CACHE_MODE) deferred to a follow-up if real configurations need them.

Related issue

Closes #50.

Type of change

• New feature (non-breaking change that adds functionality)

How it was tested

pytest -q
752 passed, 1 skipped

15 new tests:

• _resolve_cache_mode env mapping for bypass, disabled, read_only, write_only, case-insensitive, default/unset, unknown value (warns), missing library (silent), future enum-rename via getattr AttributeError fallback, warn-once per raw value across repeated calls
• Crawl4AIDiscoveryProvider.discover_urls passes cache_mode into the constructed CrawlerRunConfig when the env is set, omits it when unset
• Crawl4AIFetchProvider.fetch_one passes config=run_config when override is set, preserves the original crawler.arun(url=url) call shape when unset
• CLI: --no-fetch-cache sets SCRAPE_CACHE_MODE=bypass, the flag respects a pre-existing env (disabled survives), and main() restores the prior env (popped or restored to original) on every exit path including sys.exit

Checklist

• My code follows the project style (English-only code, comments, and identifiers)
• I ran the test suite locally and it passes (pytest)
• I added or updated tests where it made sense
• I updated the documentation in docs/ and/or README.md if behavior changed (new flag entry in docs/CLI_GUIDE.md; .env.example and installer/templates/env.example carry the
  value list; CHANGELOG entry added)
• I read the Contributing guide and the Code of Conduct
• My commits follow the project commit message style
• I confirmed there are no secrets or credentials in the diff

Additional notes

A few choices worth flagging:

• Env-driven, not constructor-arg. Provider factories take no args today (registry.py:_FETCH_FACTORIES stores lambda: Crawl4AIFetchProvider()). Adding constructor args would change the registry contract and break any third-party providers. Reading SCRAPE_CACHE_MODE at fetch time keeps the factory contract intact and matches the existing SCRAPE_PROVIDER / SCRAPE_DISCOVER_PROVIDER / SCRAPE_FETCH_PROVIDER pattern from ADR-0009. Trade-off: one more env var to maintain, but it composes naturally with how the rest of the scraper is configured.
• setdefault precedence. Mirrors --provider's behaviour at line 124 of the same file. A user who explicitly set SCRAPE_CACHE_MODE=read_only in their .env and then passes
  --no-fetch-cache thinking they're stacking gets to keep read_only rather than being silently downgraded. The "flag wins" alternative was considered and rejected for asymmetry with
  --provider.
• try/finally env restore. Without it, os.environ["SCRAPE_CACHE_MODE"] = ... would persist for the lifetime of the process. Fine for a one-shot CLI invocation; broken for tests
  that call main() directly and for any future embedding application that reuses the same Python process. The restore is unconditional and runs even on sys.exit(2) / sys.exit(3).
• Two distinct None paths in _resolve_cache_mode. Library missing → silent (the caller's _ensure_available surfaces a clearer ProviderUnavailableError). Value missing or
  unrecognised → warn once per raw value via a module-level _WARNED_CACHE_VALUES: set[str]. Without the dedupe, a typo would emit hundreds of duplicate stderr lines on a deep crawl.
• getattr(CacheMode, mapped) is wrapped in try/except AttributeError. A future Crawl4AI release that renames an enum value falls through to the warn-once branch instead of crashing
  the run.
• Firecrawl is a no-op. Firecrawl SDK's app.scrape(url, ...) defaults to fresh-fetch; the API has a maxAge parameter for opt-in caching that king-scrape doesn't use today. If
  Firecrawl ever gains a default cache, the same env can grow a Firecrawl-specific mapping.

This lays the primitive king-scrape update <name> (#46) needs to make force_refresh=True actually fetch from the network instead of being shadowed by Crawl4AI's local cache.

[deandevz]

Verified locally: 182 tests pass, env precedence is correct (existing SCRAPE_CACHE_MODE wins over the flag), the finally block restores or pops cleanly, _resolve_cache_mode maps all four cache modes case insensitively and warns once on garbage. Honoured by crawl4ai, firecrawl correctly ignores it.

One blocking issue and one merge mechanic worth knowing about, both in #55: --no-fetch-cache does not plumb into update_main or audit_main (it only exists on the main parser), and cli.py main() has a guaranteed conflict with #49 and #51 since all three touch the dispatcher. The issue also has consolidated feedback for your other three PRs and a suggested unification.

[deandevz]

Superseded by #56, merged as ae1f941. SCRAPE_CACHE_MODE
plus --no-fetch-cache landed in the unified stack, now
plumbed through all three subcommands (cli, audit, update)
via the shared _cache_mode helper. Thanks for the original
work, closing as superseded.