browserless provider always fails with HTTP 500 — sends Bearer header, API expects ?token= query param

[lloydsilvertwo]

Summary

BrowserlessProvider::scrape_url authenticates with an Authorization: Bearer <token> header. Browserless authenticates with a ?token= query parameter. The header form is rejected at their openresty edge proxy with a generic HTTP 500, so the browserless provider fails for every user on every request.

Symptoms:

• search doctor reports browserless: ok: false, category: "server"
• -m extract silently loses the last link in its fallback chain (Jina → Firecrawl → Browserless), which is precisely the Cloudflare / anti-bot case the earlier links can't serve

The part that cost me the most time: because the rejection happens at the proxy, the request never reaches the customer's account, so nothing appears in the Browserless API logs. From the user's side it looks like a Browserless outage or a bad key. I checked the key, regenerated it, and went looking at Browserless's status page before thinking to diff the auth mechanism.

Version: 0.9.0, and current master.
File: src/providers/browserless.rs:29-43.

Reproduction

import json, os, urllib.request, urllib.error

KEY = os.environ["BROWSERLESS_API_KEY"]
BODY = json.dumps({"url": "https://example.com"}).encode()

for host in ["production-sfo.browserless.io", "production-lon.browserless.io",
             "production-ams.browserless.io", "chrome.browserless.io"]:
    for label, url, headers in [
        ("Bearer header", f"https://{host}/content",
         {"Content-Type": "application/json", "Authorization": f"Bearer {KEY}"}),
        ("?token= param", f"https://{host}/content?token={KEY}",
         {"Content-Type": "application/json"}),
    ]:
        req = urllib.request.Request(url, data=BODY, headers=headers, method="POST")
        try:
            with urllib.request.urlopen(req, timeout=30) as r:
                print(f"{host:34} {label:16} {r.status}  {len(r.read())} bytes")
        except urllib.error.HTTPError as e:
            print(f"{host:34} {label:16} {e.code}")

Output with a valid key (2026-08-15):

production-sfo.browserless.io      Bearer header    500
production-sfo.browserless.io      ?token= param    200  560 bytes
production-lon.browserless.io      Bearer header    500
production-lon.browserless.io      ?token= param    200  560 bytes
production-ams.browserless.io      Bearer header    500
production-ams.browserless.io      ?token= param    200  560 bytes
chrome.browserless.io              Bearer header    500
chrome.browserless.io              ?token= param    200  560 bytes

The region is irrelevant. The auth mechanism is the whole variable.

Fix

-        let endpoint = "https://production-sfo.browserless.io/content";
-
         let body = serde_json::json!({
             "url": url,
             "waitForSelector": { "selector": "body", "timeout": 10000 }
         });
 
         let client = &self.ctx.client;
         let token = self.api_key().to_string();
 
+        let endpoint = format!("https://production-sfo.browserless.io/content?token={}", token);
+
         let resp = super::retry_request(|| async {
             let r = client
-                .post(endpoint)
+                .post(&endpoint)
                 .header("Content-Type", "application/json")
-                .header("Authorization", format!("Bearer {}", token))
                 .json(&body)
                 .send()
                 .await?;

token has to move above endpoint; otherwise the change is confined to those two lines. I've been running this locally and search doctor reports browserless: ok, with -m extract -p browserless returning real page content.

Happy to open a PR if you'd prefer that to a patch in an issue.

Two adjacent suggestions

Distinguish a malformed request from a provider outage. The 500 arrives as an HTML error page from openresty, and ok_or_api_error surfaces it as category: "server" — which reads as "the provider is down" and sends users to the wrong place. A 5xx whose body is HTML rather than JSON is a decent heuristic for "we sent something the edge didn't understand".

Make the region configurable. production-sfo is hardcoded. All four endpoints work, so a European or APAC user is currently paying a transatlantic round trip on the slowest provider in the extract chain. A keys.browserless_region (or providers.browserless.endpoint) config entry would be cheap and would help the exact case Browserless is there for.