scraper: replace blocklisted default User-Agent (fixes RealJam VR / PornCorn VR 403)#2234
Open
make-it-fun wants to merge 1 commit into
Open
scraper: replace blocklisted default User-Agent (fixes RealJam VR / PornCorn VR 403)#2234make-it-fun wants to merge 1 commit into
make-it-fun wants to merge 1 commit into
Conversation
…ornCorn VR 403) realjamvr.com and porncornvr.com reject XBVR's default User-Agent "Chrome/73.0.3683.103" with HTTP 403. Both scrapers therefore fail on every scheduled run with Error visiting https://realjamvr.com/scenes Forbidden Error visiting https://porncornvr.com/scenes Forbidden and ingest nothing. The failure is easy to miss: the scraper logs the error, reports "Finished", and the sites simply stop producing new scenes. The block is keyed to the EXACT literal version string, not to "old browser" heuristics. Verified live against realjamvr.com: Chrome/73.0.3683.103 -> 403 (the shipped default) Chrome/73.0.3683.104 -> 200 (one digit different) Chrome/73.0.0.0 -> 200 Chrome/120.0.0.0 -> 200 Because the default is compiled in, every XBVR install sends that string verbatim, which makes it a reliable fingerprint for the client - and that is what makes it worth blocklisting. Nothing short of changing the string works; this is not something a user can configure around. Bump it to Chrome/120.0.0.0, in line with the Chrome/108 UA already used by the imageproxy in pkg/server/server.go. Verified after the change: both scene listings return 200, and full scrapes of both sites complete normally and save scenes again.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
realjamvr.comandporncornvr.comreject XBVR's default User-AgentChrome/73.0.3683.103with HTTP 403. Both scrapers therefore fail on every scheduled run and ingest nothing:This is easy to miss. The scraper logs the error, then reports
Finished ... completed=true, so from the UI the run looks like it happened — the site just quietly stops producing new scenes. I only found it while chasing an unrelated single-scene report; the scrapers had been dead for two days without anyone noticing.Root cause
The block is keyed to the exact literal version string, not to "old browser" heuristics. Verified live against
realjamvr.com:Chrome/73.0.3683.103(the shipped default)Chrome/73.0.3683.104(one digit different)Chrome/73.0.0.0Chrome/120.0.0.0Because the default is compiled in, every XBVR install sends that string verbatim — which makes it a reliable fingerprint for the client, and that is precisely what makes it worth blocklisting. Changing the string is the only remedy; there is no user-facing setting to work around it.
Fix
Bump the default to
Chrome/120.0.0.0, in line with theChrome/108UA already used by the imageproxy inpkg/server/server.go. One string, plus a comment recording the diagnosis so the next person hitting a 403 checks for a fingerprint block before assuming the site changed.Verification
Forbidden).ScrapeHtml) for these sites works again as a side effect, since it shares this UA.Notes
ScraperProxy) would be the real escape hatch, so a blocklisting doesn't require a new release. Happy to open that as a separate PR if there's interest.pkg/scrapecan't currently host tests, becausecommon.InitPaths()callsflag.Parse()at init and the test binary's-test.*flags abort it (flag provided but not defined: -test.testlogfile). Fixing that is a separate concern from this one-line change — glad to submit it on its own if you'd like the test.