Skip to content

fix(web): add Cloudflare block detection and direct HTTP fallback - #480

Closed
nankingjing wants to merge 2 commits into
Panniantong:mainfrom
nankingjing:fix/cloudflare-block-fallback
Closed

fix(web): add Cloudflare block detection and direct HTTP fallback#480
nankingjing wants to merge 2 commits into
Panniantong:mainfrom
nankingjing:fix/cloudflare-block-fallback

Conversation

@nankingjing

@nankingjing nankingjing commented Jul 5, 2026

Copy link
Copy Markdown

Summary

When Jina Reader is blocked by Cloudflare or similar anti-bot protection (e.g., fetching imginn.com/instagram/), the response comes back as a security verification page instead of actual content.

This PR adds automatic Cloudflare block detection and a direct HTTP fallback using requests (already a project dependency) with realistic browser headers.

Changes

  • agent_reach/channels/web.py: Added _is_blocked() to detect Cloudflare/anti-bot block pages and _direct_fetch() as a fallback using browser-like headers
  • When Jina Reader returns a block page, auto-fallback to direct HTTP fetch
  • Only changes the web channel; all other channels unaffected

Test Results

184 passed, 11 skipped, 1 pre-existing failure (unrelated YouTube test)

Closes #428

… Jina Reader

When Jina Reader is blocked by Cloudflare or similar anti-bot
protection, auto-detect the block page and fall back to direct
HTTP fetch with browser-like headers.

Closes Panniantong#428

@moxxxskaifa moxxxskaifa left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice fix! 解决了一个长期痛点:

  • _CF_BLOCK_PATTERNS 覆盖了常见 CF 拦截页面特征 ✅
  • _is_blocked() 只检查前2000字,性能开销小 ✅
  • Jina Reader 被拦后自动降级直连,优雅降级 ✅

建议一个小优化:direct_fetch 失败时最好 fallback 回原始 text(而不是丢数据)。

整体 LGTM!

@nankingjing

Copy link
Copy Markdown
Author

您好 @moxxxskaifa,这个 PR 已经 approved,是否可以合并了?谢谢!

@nankingjing

Copy link
Copy Markdown
Author

Hi @moxxxskaifa, this PR has been approved — could you merge it when you have a moment? Thanks!

…h fails

Per review feedback: when the Cloudflare/anti-bot direct-fetch fallback
itself fails (returns None or empty), return the original Jina Reader
response instead of dropping the data (`direct or text`).
@nankingjing

Copy link
Copy Markdown
Author

@moxxxskaifa 感谢 review!已按您建议的小优化实现:

当 Jina Reader 被判定为被拦截、走 _direct_fetch 直连回退,而直连本身也失败(返回 None 或空响应)时,read() 现在会显式回退到原始 Jina 文本direct or text),不再有丢数据的风险;空字符串也会走回退。逻辑与原有降级链保持一致,未改变其它路径。

commit: f0fb2fd

@nankingjing

Copy link
Copy Markdown
Author

@moxxxskaifa the post-approval commit f0fb2fd implements your suggestion exactly: when _direct_fetch fails, read() now explicitly falls back to the original Jina text (direct or text) instead of losing data. No other paths changed. The earlier approval was on 39eecdc — would you mind confirming this still LGTM? Thanks!

@Panniantong

Copy link
Copy Markdown
Owner

感谢定位 Cloudflare challenge 页面。merged #576 已安全检测并明确失败;本 PR 的直接 HTTP fallback 可能绕过预期网络和安全边界,因此不采用并关闭。

@Panniantong Panniantong closed this Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Jina Reader blocked by Cloudflare

3 participants