Scraping reachability, measured — 2026

Which review and travel sites answer an ordinary HTTP client, which need a particular TLS fingerprint, which need a short-lived token minted by a browser, and which are genuinely closed to an anonymous reader.

24 sites, 22 notes, 29 measured rows. Every line in this repository is a measurement with a date on it, taken against the live public site, mostly on 2026-08-26 to 2026-08-29. Nothing here is inferred from a blog post or from a vendor's marketing. Where a source is closed, that is written down as plainly as where it is open — including the cases where we spent a browser, a residential pool and half a day and still could not read one page.

We published this because the question "can this site be read without a browser?" is answered everywhere by guesswork, and the answer changes per site, per TLS profile, per IP pool and per month. The measurements below cost real proxy traffic and real time; they are more useful in public than in a private notebook.

What this is not

• Not a bypass guide. There is no captcha solving, no challenge-token minting recipe, no cookie or session sharing, no credential re-use, no advice for getting past a login or a paywall. Where a site requires sign-in, the finding is "requires sign-in — we stopped", and that is the end of the entry.
• Not a scraper. The only code here is a diagnostic (check.py) that reports what a site answers your client and names the block page it recognises. It does not solve anything.
• Not a claim about legality or terms. Reachability is a technical fact. Whether you may fetch a page is a separate question, and it is yours to answer.

Verdict key

verdict	meaning
OPEN	a plain HTTP client (httpx/requests, HTTP/2, ordinary UA) on cheap datacenter IPs is served
OPEN — TLS profile	served only by a client that presents a real browser's TLS/HTTP2 fingerprint (curl_cffi impersonation); the IP pool is not the issue
OPEN — residential	served only from residential IPs; datacenter ranges are refused whatever the client looks like
BROWSER TOKEN	pages are challenged, but a browser can mint a token once and ordinary clients are then served for as long as it lives
BROWSER ONLY	a real browser must run the challenge for every read; no HTTP client passes
CLOSED	not readable anonymously by anything we could bring, including a real headed browser from a clean home IP, or explicitly sign-in walled

Summary

source	verdict	what it needs	what one request returns	measured
TripAdvisor — hotels, restaurants, attractions	OPEN — residential + TLS profile	residential IP and a Firefox 147 fingerprint; Chrome fingerprints are challenged, datacenter IPs refused	10 reviews per -orN- page (15 on restaurants), full record inlined in the page	2026-08-29
Booking.com	OPEN — TLS profile	Chrome fingerprint; the ReviewList GraphQL call is outside the WAF that challenges every page	25 reviews per call, with sub-scores and the property reply	2026-08-26 … 29
Trustpilot	BROWSER TOKEN	one headless-browser challenge solve per ~5 minutes; then any client, any IP	20 reviews per page, hard stop at 200 per filtered view	2026-08-29
Hostelworld	OPEN	nothing — public keyless JSON API, datacenter, plain HTTP client	up to 50 reviews per call, self-describing paging	2026-08-29
Indeed — company reviews	OPEN — TLS profile	Chrome fingerprint (Firefox is a hard 403); datacenter is fine	20 reviews per page, server-rendered JSON, plus histogram and category averages	2026-08-29
Zoover	OPEN	TLS profile for the HTML page; the review API answers a plain client and is Allowed in robots.txt	every review of an accommodation in one call	2026-08-29
Kurzurlaub.de	OPEN	nothing — datacenter, plain HTTP client	20 ratings per AJAX page (scores and titles; no free text exists)	2026-08-26
Agoda	OPEN	HTTP/2 — over HTTP/1.1 the endpoint answers 200 with zero reviews	a page of reviews per call	2026-08-26
Trip.com	OPEN	the site's own head block in the request body, or 200/Success with nothing in it	a page of reviews per call	2026-08-26 … 29
Airbnb	OPEN	the public API key the site ships in its own pages	search, calendar, listing and reviews via persisted GraphQL	2026-08-26
Google Hotels / Flights	OPEN	nothing — the site's own batchexecute RPC / tfs URL parameter	hotel list with rates, or a page of itineraries	2026-08-25 … 26
App Store / Google Play	OPEN	pacing — both answer 429 to bursts	a page of reviews per call	2026-08-29
AliExpress	OPEN	nothing for the feedback endpoint; datacenter is fine	up to 500 reviews per call from page 2 on — page 1 always returns 20	2026-08-29
eBay — active listings	OPEN — TLS profile	Chrome fingerprint; and the marketplace host matters more than the IP	60 listings per search page (+2 decoy cards)	2026-08-29
Amazon	OPEN, with a low ceiling	rotation and a current Chrome UA string	product data + at most 13 reviews per product, ever	2026-08-27 … 29
HRS	OPEN — residential	residential IPs; the ratings API takes a client id that is a constant in the site's own bundle	the full rating list for a hotel	2026-08-29
Despegar	OPEN — residential + TLS profile, capped	residential and a Chrome fingerprint (Firefox is refused everywhere)	the score, six category scores, an AI summary and 4 comments — that is all that exists	2026-08-29
YouTube — captions	HALF OPEN	the caption list is free; the caption body needs a proof-of-origin token	list yes, transcript no	2026-08-29
Expedia / Hotels.com	BROWSER ONLY	a browser that runs the Akamai sensor and the captcha app	nothing to an HTTP client, including from a clean IP	2026-08-29
Yelp	CLOSED from our pools	a clean IP that is not on DataDome's list	403 on every commercial proxy pool we have; 200 from a home IP	2026-08-29
Glassdoor	CLOSED	—	403 to 23 of 23 HTTP attempts and to real headed Chrome from a clean IP; paging is sign-in walled	2026-08-29
eBay sold listings	CLOSED — sign-in walled	an account	0 results in ~115 attempts; it is an HTTP redirect, so a browser does not help	2026-08-29

Machine-readable: data/reachability.csv.

What generalises

Ten things that were not obvious before we measured them, and that cost the most time to learn:

1. The impersonation profile is per-site, not per-vendor. TripAdvisor and Despegar both sit behind DataDome. TripAdvisor is served to a Firefox 147 fingerprint and refuses every Chrome one; Despegar is the exact mirror image. Probe both before calling anything browser-only.
2. "Blocked" has at least four different causes, and they need different fixes — IP reputation (Yelp), TLS fingerprint (Booking pages, Zoover pages, eBay), a JS-computed token (Trustpilot, Expedia), and a sign-in wall (Glassdoor page 2, eBay sold, Amazon's review archive). Only the first is fixed by buying better proxies.
3. A site's front door and its API are usually behind different doors. Booking challenges every page and leaves the GraphQL call its own review list uses open. TripAdvisor's persisted-query gateway answers datacenter Chrome clients while its pages refuse them. Zoover 403s its HTML to a plain client and serves the JSON API to it. Trustpilot's widget host sits outside the WAF that covers the site.
4. Read robots.txt and the SSR config blob before writing a single request. Zoover's robots.txt contains Allow: /api/getReviews above Disallow: /api. Hostelworld's page config contains HOSTELWORLD_API_URL. Two of three sources handed over their endpoint that way.
5. The dangerous failure is a 200, not a 403. Agoda answers a throttled call 200 with no comments — identical to a hotel with no reviews. Indeed answers a page past the end with 200 and page 1's twenty reviews. AliExpress wraps a far page back to page 1. Zoover returns exactly one row if you omit size while still reporting the true total. A loop that stops "when the page is empty" never stops, and a scraper that bills per row bills the same rows forever.
6. Run the probe in the environment the code will run in. Same script, same proxy, same second: the macOS box resolved curl_cffi 0.13 (firefox → Firefox 135) and the Linux container resolved 0.16 (firefox → Firefox 147). One is refused by TripAdvisor and the other is served. The verdict flipped on a library alias, not on the operating system — so pin the profile version explicitly.
7. Test against something other than the most popular id on the internet. YouTube's ANDROID_VR client answered for dQw4w9WgXcQ and for nothing else — almost certainly an edge cache. Eleven other videos failed from up to 60 fresh residential IPs. A single lucky control makes a broken pipeline look finished.
8. Public ceilings are much lower than the numbers a site prints. Amazon shows at most 13 reviews per product whether it has 37 ratings or 60,321. Despegar renders 4 of 4,511. Trustpilot serves 200 per filtered view. HRS embeds 4. The count on the page is not the count you can read.
9. Per-host posture inside one brand. eBay's .de and .com.au served 5 of 5 while .co.uk, .ie, .ca, .ch and .com.hk served 0 of 18 on the same proxies in the same minute. Country domains are separate deployments; so are Indeed's, which each expose a different, disjoint subset of the same company's reviews.
10. A browser capture is worth it for the negative answer too. Two full headed-Chrome captures of a Despegar hotel page (109 and 101 requests) recorded zero review-bearing XHRs, and the page's own analytics beacon spelled out the ceiling: {"review_count":"4511","total_comments":"4"}. That is what "there is nothing more to fetch" looks like as evidence.

How this was measured

See METHODOLOGY.md — proxy pools, clients, what counted as a result, and what we deliberately did not do.

Reproduce it

check.py is a single-file diagnostic. It fetches one URL with a plain HTTP/2 client and with each TLS-impersonation profile you name, and reports the status, the size and which known block page it recognises (AWS WAF challenge, DataDome device-check / captcha / hard block, Akamai Access Denied or Bot or Not?, a Cloudflare managed challenge, an Indeed security check, a sign-in interstitial).

pip install httpx curl_cffi
python check.py https://www.tripadvisor.com/Hotel_Review-g189158-d228423-Reviews.html --profiles chrome,firefox
python check.py https://www.hostelworld.com/ --proxy http://user:pass@proxy:8000

It answers "what does this site do to my client, from my address, today" — which is the only form of this question that has a stable answer.

Contributing

Corrections are welcome, and re-measurements especially. The bar is the same one we held ourselves to: say what you requested, from where, with which client, what came back (status, size, and the marker you matched), and on which date. A claim without a measurement behind it will be turned down, however plausible. See CONTRIBUTING.md.

Licence

The findings are CC BY 4.0; check.py is MIT. Use it, quote it, correct it.

---

Built with this

These measurements came out of building production scrapers. If you want the working implementations rather than the map: the kestrel Apify actors and their documentation and runnable examples.