PROD-10345 - Fix: Link preview fetch returns 403 for external URLs whose firewall blocks browser-like User-Agents (e.g., jeremyrhammond.com) - #5042
Open
sajib-bb wants to merge 2 commits into
Open
Conversation
bp_core_parse_url() used a hardcoded browser-style User-Agent for its server-side fetch. Some external sites run a firewall/WAF that blocks that signature while allow-listing known link-preview bots (facebookexternalhit, Twitterbot, Slackbot), so the preview request was rejected with a 403. - Add a bp_core_parse_url_http_args filter around both fetch call sites so the User-Agent (or any other request arg) can be overridden per-site. - Stop treating non-2xx responses as a successful fetch: a WAF block page's own HTML (e.g. containing a literal "403 Forbidden" title) was being scraped and cached as a real preview for 24 hours.
The link preview fetch previously always used a browser-style User-Agent, which some external sites' firewalls block while still allow-listing known link-preview bots (facebookexternalhit, Twitterbot, Slackbot). - bp_core_parse_url() now tries facebookexternalhit/1.1 first via bp_core_get_link_preview_user_agent() (filterable via bp_core_parse_url_user_agent). - If that request doesn't return a usable title/description/image, it automatically retries with the previous browser User-Agent via bp_core_get_link_preview_fallback_user_agent() (filterable via bp_core_parse_url_fallback_user_agent). This keeps working, with no custom code required, for sites that only serve full preview markup to browser-style requests. - Extracted the OG/meta-tag scraping logic into bp_core_extract_link_preview_data_from_html() so it can be reused for each User-Agent attempt without duplicating the parsing code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Jira Issue:
https://buddyboss.atlassian.net/browse/PROD-10345
General Note
Keep all conversations related to this PR in the associated Jira issue(s). Do NOT add comment on this PR or edit this PR’s description.
Notes to Developer
Notes to Reviewer