feat(core): fall back to challenge-bypass sidecar for article enrichment - #171
Merged
Merged
Conversation
joshpanka
pushed a commit
that referenced
this pull request
Jul 13, 2026
Address the PR #171 review: - SSRF blocker: reject non-http(s) / hostless external_url in _challenge_fallback BEFORE the sidecar handoff (file:// data: gopher: bypassed the pre-check because the host is empty, and the direct path's rejection is what triggers the fallback). - crash: destination_block_reason now treats an out-of-range port (ValueError on parts.port) as BLOCKED instead of raising, so an untrusted URL can't fail the run. - document the redirect-follow residual (public host 302 -> internal) alongside the DNS-rebinding TOCTOU, both accepted for the self-hosted, opt-in sidecar. - tests: non-web scheme rejected before sidecar, malformed port blocked not crash, disabled-sidecar skips the pre-check + solve, happy path never consults the sidecar. - nits: single DEFAULT_MAX_FETCH_BYTES in sources.connectors.base (was 3 literals); direct settings.SOURCE_CHALLENGE_BYPASS_URL (not getattr); clearer enum comment; drop a pre-existing `--` in the _external docstring.
…er-kind fetch target Article enrichment (semantic_filter / extract) now recovers the FULL article for challenge-protected sources and enriches RSS, so the judge/extract see the article body instead of just the feed teaser. - Per-kind fetch target: resolve_external_content fetches `payload.article_url`, a computed property on the connector SourcePayload -- base returns `external_url` (aggregators like HN point off-site), RssEntryPayload overrides to `url` (an RSS entry IS the article). Reddit / Ask HN / HN comments have neither, so they no-op. Chosen at evaluation time from stored fields, so it applies to existing feed items with no re-poll / backfill and adds nothing to the wire (plain @Property). - Challenge-bypass fallback: when the direct pinned-IP fetch hits a Cloudflare-style wall (UNAVAILABLE / MISSING) and SOURCE_CHALLENGE_BYPASS_URL is set, retry the article through the existing FlareSolverr sidecar, tagged INCLUDED_VIA_FALLBACK. - SSRF: the sidecar egresses via a real browser, so the untrusted article_url is pre-validated (http(s) scheme + host via _is_fetchable_web_url, then destination_block_reason with block_private_ips=True) before handoff. Residual gaps (DNS-rebind TOCTOU, redirect-follow, urlsplit-vs-WHATWG parser differential) are documented and accepted for the self-hosted, opt-in sidecar. - Robustness: primary fetch catches httpx.InvalidURL (a sibling of HTTPError); destination_block_reason treats a malformed URL / bad port as BLOCKED not a crash; challenge_bypass_fetch tolerates a non-object JSON body -- so a bad/crafted URL or a misconfigured sidecar degrades to UNAVAILABLE, never fails the run. - Refactors: extract module-level challenge_bypass_fetch (mixin delegates); one shared FETCH_DEFAULT_MAX_BYTES cap; new ExternalContentStatus.INCLUDED_VIA_FALLBACK. Off by default (empty SOURCE_CHALLENGE_BYPASS_URL). Adding a value to the strict ExternalContentStatus response enum means a pinned older CLI must upgrade to render fallback-enriched runs.
joshpanka
force-pushed
the
feat/enrichment-challenge-bypass-fallback
branch
from
July 13, 2026 16:46
bc2fe4f to
46570e5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Linked-article enrichment (
semantic_filter+extract) fetches an item'sexternal_urlvia the direct pinned-IP path. When that hits a Cloudflare-style wall — the fetch is refused (UNAVAILABLE) or returns an interstitial with no extractable article (MISSING) — retry the article through the existing FlareSolverr challenge-bypass sidecar whenSOURCE_CHALLENGE_BYPASS_URLis set, tagging the resultINCLUDED_VIA_FALLBACK.Off by default (empty setting). Reuses the RSS connector's existing sidecar client rather than adding a new one.
Why
RSS feeds of press releases (e.g. athletics coaching-change announcements) carry only the
<description>teaser; the judge/extract need the full article. Many of those publisher sites sit behind Cloudflare, so the direct pinned-IP fetch is blocked and enrichment silently falls back to the teaser. The sidecar (headless Chrome) passes the challenge and returns the real body.Security (SSRF)
The sidecar egresses via a real browser, so the pinned-IP guard that protects the direct path cannot apply, and
external_urlis untrusted (item-derived). The fallback therefore:SOURCE_BLOCK_PRIVATE_IPS) before handing off, thenThis leaves a DNS-rebinding TOCTOU the pinned path doesn't have (the sidecar re-resolves at fetch time) — accepted because the sidecar is a self-hosted, operator-enabled service. Documented in code + the settings comment.
Changes
watch_actions/_engine.py: newExternalContentStatus.INCLUDED_VIA_FALLBACK(schema.json + webgenerated.tsregenerated, additive enum value).sources/connectors/challenge_bypass.py: extracted a module-levelchallenge_bypass_fetch()(theChallengeBypassMixinnow delegates; RSS path unchanged) + a SECURITY note that untrusted callers must pre-validate the host.watches/actions/_external.py: the fallback branch +_challenge_fallback()with the SSRF pre-check.conf/settings/base.py: expanded theSOURCE_CHALLENGE_BYPASS_URLcomment (now also backs enrichment; states the trade-off).watches/tests_external.py: made existing tests hermetic (sidecar off) + addedChallengeBypassFallbackTests(fires on UNAVAILABLE + MISSING, SSRF pre-check blocks private hosts, sidecar-fail keeps status, solved-but-no-text stays MISSING).Verification
ruff + format + whitespace, ty, schema drift clean, 361 core tests, 145 CLI tests, web
check+smoke+@magpie/schema/@magpie/apptypecheck — all green.Follow-up (not in this PR)
The RSS connector still needs to populate
external_url(= the article link) for RSS items to have anything to enrich; that's a separate small change.