Skip to content

scraper: replace blocklisted default User-Agent (fixes RealJam VR / PornCorn VR 403)#2234

Open
make-it-fun wants to merge 1 commit into
xbapps:masterfrom
make-it-fun:fix/blocklisted-default-user-agent
Open

scraper: replace blocklisted default User-Agent (fixes RealJam VR / PornCorn VR 403)#2234
make-it-fun wants to merge 1 commit into
xbapps:masterfrom
make-it-fun:fix/blocklisted-default-user-agent

Conversation

@make-it-fun

Copy link
Copy Markdown
Contributor

Problem

realjamvr.com and porncornvr.com reject XBVR's default User-Agent Chrome/73.0.3683.103 with HTTP 403. Both scrapers therefore fail on every scheduled run and ingest nothing:

level=error msg="Error visiting https://realjamvr.com/scenes Forbidden"
level=error msg="Error visiting https://porncornvr.com/scenes Forbidden"

This is easy to miss. The scraper logs the error, then reports Finished ... completed=true, so from the UI the run looks like it happened — the site just quietly stops producing new scenes. I only found it while chasing an unrelated single-scene report; the scrapers had been dead for two days without anyone noticing.

Root cause

The block is keyed to the exact literal version string, not to "old browser" heuristics. Verified live against realjamvr.com:

User-Agent Result
Chrome/73.0.3683.103 (the shipped default) 403
Chrome/73.0.3683.104 (one digit different) 200
Chrome/73.0.0.0 200
Chrome/120.0.0.0 200

Because the default is compiled in, every XBVR install sends that string verbatim — which makes it a reliable fingerprint for the client, and that is precisely what makes it worth blocklisting. Changing the string is the only remedy; there is no user-facing setting to work around it.

Fix

Bump the default to Chrome/120.0.0.0, in line with the Chrome/108 UA already used by the imageproxy in pkg/server/server.go. One string, plus a comment recording the diagnosis so the next person hitting a 403 checks for a fingerprint block before assuming the site changed.

Verification

  • Both scene listings return 200 with the new UA.
  • Full scrapes of both sites now complete normally and save scenes again (previously: instant Forbidden).
  • Trailer-source scraping (ScrapeHtml) for these sites works again as a side effect, since it shares this UA.

Notes

  • This will go stale again eventually. A user-settable User-Agent in Advanced config (next to ScraperProxy) would be the real escape hatch, so a blocklisting doesn't require a new release. Happy to open that as a separate PR if there's interest.
  • I wrote a regression test pinning the known-bad literal, but left it out of this PR: pkg/scrape can't currently host tests, because common.InitPaths() calls flag.Parse() at init and the test binary's -test.* flags abort it (flag provided but not defined: -test.testlogfile). Fixing that is a separate concern from this one-line change — glad to submit it on its own if you'd like the test.

…ornCorn VR 403)

realjamvr.com and porncornvr.com reject XBVR's default User-Agent
"Chrome/73.0.3683.103" with HTTP 403. Both scrapers therefore fail on every
scheduled run with

  Error visiting https://realjamvr.com/scenes Forbidden
  Error visiting https://porncornvr.com/scenes Forbidden

and ingest nothing. The failure is easy to miss: the scraper logs the error,
reports "Finished", and the sites simply stop producing new scenes.

The block is keyed to the EXACT literal version string, not to "old browser"
heuristics. Verified live against realjamvr.com:

  Chrome/73.0.3683.103  -> 403   (the shipped default)
  Chrome/73.0.3683.104  -> 200   (one digit different)
  Chrome/73.0.0.0       -> 200
  Chrome/120.0.0.0      -> 200

Because the default is compiled in, every XBVR install sends that string
verbatim, which makes it a reliable fingerprint for the client - and that is
what makes it worth blocklisting. Nothing short of changing the string works;
this is not something a user can configure around.

Bump it to Chrome/120.0.0.0, in line with the Chrome/108 UA already used by the
imageproxy in pkg/server/server.go. Verified after the change: both scene
listings return 200, and full scrapes of both sites complete normally and save
scenes again.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant