fix(web): add Cloudflare block detection and direct HTTP fallback - #480
fix(web): add Cloudflare block detection and direct HTTP fallback#480nankingjing wants to merge 2 commits into
Conversation
… Jina Reader When Jina Reader is blocked by Cloudflare or similar anti-bot protection, auto-detect the block page and fall back to direct HTTP fetch with browser-like headers. Closes Panniantong#428
moxxxskaifa
left a comment
There was a problem hiding this comment.
Nice fix! 解决了一个长期痛点:
- _CF_BLOCK_PATTERNS 覆盖了常见 CF 拦截页面特征 ✅
- _is_blocked() 只检查前2000字,性能开销小 ✅
- Jina Reader 被拦后自动降级直连,优雅降级 ✅
建议一个小优化:direct_fetch 失败时最好 fallback 回原始 text(而不是丢数据)。
整体 LGTM!
|
您好 @moxxxskaifa,这个 PR 已经 approved,是否可以合并了?谢谢! |
|
Hi @moxxxskaifa, this PR has been approved — could you merge it when you have a moment? Thanks! |
…h fails Per review feedback: when the Cloudflare/anti-bot direct-fetch fallback itself fails (returns None or empty), return the original Jina Reader response instead of dropping the data (`direct or text`).
|
@moxxxskaifa 感谢 review!已按您建议的小优化实现: 当 Jina Reader 被判定为被拦截、走 commit: f0fb2fd |
|
@moxxxskaifa the post-approval commit f0fb2fd implements your suggestion exactly: when |
f0fb2fd to
9146d24
Compare
|
感谢定位 Cloudflare challenge 页面。merged #576 已安全检测并明确失败;本 PR 的直接 HTTP fallback 可能绕过预期网络和安全边界,因此不采用并关闭。 |
Summary
When Jina Reader is blocked by Cloudflare or similar anti-bot protection (e.g., fetching imginn.com/instagram/), the response comes back as a security verification page instead of actual content.
This PR adds automatic Cloudflare block detection and a direct HTTP fallback using
requests(already a project dependency) with realistic browser headers.Changes
agent_reach/channels/web.py: Added_is_blocked()to detect Cloudflare/anti-bot block pages and_direct_fetch()as a fallback using browser-like headersTest Results
Closes #428