Summary
The native FuckPassVR scraper (fuckpassvr-native) only processes a partial set of scenes per run when the site returns a geo-gate / "safe-for-work" redirect for some requests. Once any request redirects to https://www.fuckpassvr.com/sfw/, Colly's visited-URL dedup adds /sfw/ to its visited list, and every subsequent scene whose request also redirects to /sfw/ errors out as "https://www.fuckpassvr.com/sfw/" already visited — the scraper skips those scenes instead of treating them as retry-able.
Observation
Single run after force-site-update on 374 flagged scenes:
- 191 scene URLs attempted
- 88 failed with
Error visiting <scene-url> Get "https://www.fuckpassvr.com/sfw/": "https://www.fuckpassvr.com/sfw/" already visited
- ~103 succeeded (89 existing scenes updated + 9 brand-new scenes discovered)
Log sample:
level=info msg="visiting https://www.fuckpassvr.com/video/the-night-is-young"
level=error msg="Error visiting https://www.fuckpassvr.com/video/the-night-is-young Get \"https://www.fuckpassvr.com/sfw/\": \"https://www.fuckpassvr.com/sfw/\" already visited"
level=info msg="visiting https://www.fuckpassvr.com/video/fucking-like-the-bulls"
level=error msg="Error visiting https://www.fuckpassvr.com/video/fucking-like-the-bulls Get \"https://www.fuckpassvr.com/sfw/\": \"https://www.fuckpassvr.com/sfw/\" already visited"
Why this surfaces with rotating proxies
Using Webshare's Rotating Proxy Endpoint (per-request IP rotation), each HTTP request comes from a different exit node. Some nodes pass the geo-gate cleanly; others trigger the /sfw/ redirect. With a static proxy you get one consistent outcome (all succeed or all fail). With rotation you get a mix: the first blocked request adds /sfw/ to Colly's visited list, then every later request that also tries to redirect to /sfw/ trips the dedup check and the scene is permanently skipped for that run.
Current workaround
Restart the container (clears Colly's in-memory visited set), re-run the scrape. Each cycle picks off another fraction of the remaining scenes. In practice it takes 2-3+ restart-and-rerun cycles to drain the backlog, and it leaves a lot of "already visited" noise in the logs.
Proposals
Not sure which is the right fit for the codebase — any of these would help:
- Scope Colly's dedup to the originally-requested URL, not redirect targets. The scene's canonical URL is the natural dedup key; intermediate redirects (
/sfw/, age gates, login walls, etc.) shouldn't pollute the visited set.
- Explicit geo-gate handling in the FPVR scraper. Detect the
/sfw/ redirect and either retry the original URL on the next IP rotation, or skip the scene silently without the dedup error.
- Preflight the gate with a cookie. If FPVR's geo-gate accepts an
age_verify=1 (or equivalent) cookie, set it proactively before scene visits so the redirect never fires.
(1) is the most general — it would help any site with an age/geo/region gate that redirects to a shared landing URL, not just FPVR.
Environment
- XBVR
0.4.38 (build 2026-01-20), upstream image ghcr.io/xbapps/xbvr:latest
- Proxy: Webshare Rotating Proxy Endpoint (
p.webshare.io:80), DE+JP country filter
- Affected scraper:
fuckpassvr-native. The SLR-aggregator path fuckpassvr-slr is not affected because SLR is the intermediary and doesn't redirect.
Related
Summary
The native FuckPassVR scraper (
fuckpassvr-native) only processes a partial set of scenes per run when the site returns a geo-gate / "safe-for-work" redirect for some requests. Once any request redirects tohttps://www.fuckpassvr.com/sfw/, Colly's visited-URL dedup adds/sfw/to its visited list, and every subsequent scene whose request also redirects to/sfw/errors out as"https://www.fuckpassvr.com/sfw/" already visited— the scraper skips those scenes instead of treating them as retry-able.Observation
Single run after
force-site-updateon 374 flagged scenes:Error visiting <scene-url> Get "https://www.fuckpassvr.com/sfw/": "https://www.fuckpassvr.com/sfw/" already visitedLog sample:
Why this surfaces with rotating proxies
Using Webshare's Rotating Proxy Endpoint (per-request IP rotation), each HTTP request comes from a different exit node. Some nodes pass the geo-gate cleanly; others trigger the
/sfw/redirect. With a static proxy you get one consistent outcome (all succeed or all fail). With rotation you get a mix: the first blocked request adds/sfw/to Colly's visited list, then every later request that also tries to redirect to/sfw/trips the dedup check and the scene is permanently skipped for that run.Current workaround
Restart the container (clears Colly's in-memory visited set), re-run the scrape. Each cycle picks off another fraction of the remaining scenes. In practice it takes 2-3+ restart-and-rerun cycles to drain the backlog, and it leaves a lot of "already visited" noise in the logs.
Proposals
Not sure which is the right fit for the codebase — any of these would help:
/sfw/, age gates, login walls, etc.) shouldn't pollute the visited set./sfw/redirect and either retry the original URL on the next IP rotation, or skip the scene silently without the dedup error.age_verify=1(or equivalent) cookie, set it proactively before scene visits so the redirect never fires.(1) is the most general — it would help any site with an age/geo/region gate that redirects to a shared landing URL, not just FPVR.
Environment
0.4.38(build2026-01-20), upstream imageghcr.io/xbapps/xbvr:latestp.webshare.io:80), DE+JP country filterfuckpassvr-native. The SLR-aggregator pathfuckpassvr-slris not affected because SLR is the intermediary and doesn't redirect.Related