Conversation
Two bugs made the FlareSolverr backend unable to serve any scraper that
fetches a JSON endpoint, so sites behind a Cloudflare managed challenge had
no working fallback at all.
1. `params` was silently dropped. FlareSolverr's API has no `params` field
and `FlareSolverrBackend.request` only forwarded `json`/`data`, and only
for POST. A `scraper.get(url, params={...})` therefore reached the site
with no query string. Fold them into the URL instead, before the URL is
logged so the request line shows what was actually fetched.
2. The response body was the rendered DOM, not the raw payload. Chrome shows
non-HTML documents in its built-in viewer, so FlareSolverr returns
`<html><head>..</head><body><pre>{json}</pre>..</html>` and any caller
doing `.json()` gets a JSONDecodeError. Unwrap that shell, matching the
whole document so a real page containing a `<pre>` is left untouched, and
unescape the entities Chrome introduces in the text node.
Verified against Babepedia, which is currently behind a managed challenge:
`performer-by-name` went from `JSONDecodeError` (and, once that was fixed, an
empty list from the dropped `params`) to returning correct search results.
`performer-by-url` and other HTML scrapes are byte-for-byte unaffected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ghost
closed this
ghost
deleted the
fix/flaresolverr-params-and-json
branch
August 24, 2026 23:45
ghost
changed the title
This pull request was closed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.