fix(search): send auth cookies when fetching x.com home for ClientTransaction init (fixes SearchTimeline 404)

[socks1s]

Problem

twitter search ... returns HTTP 404 for every query (with or without filters), while user-posts / user / tweet / feed all work. Upgrading doesn't help (PyPI 0.8.5 and main 0.8.6 share this code).

Root cause

X enforces the x-client-transaction-id anti-bot header on SearchTimeline specifically (other operations tolerate its absence, which is why only search 404s).

_init_client_transaction() fetches https://x.com with guest headers only (no auth cookies) to locate the ondemand.s file:

ct_headers = _gen_ct_headers()
home_page = cffi_session.get("https://x.com", headers=ct_headers, timeout=10)
...
ondemand_url = get_ondemand_file_url(response=home_page_response)

X's current guest homepage (~35 KB) no longer embeds the ondemand.s reference, so get_ondemand_file_url() raises 'NoneType' object has no attribute 'group'. CT init then silently fails (the WARNING Failed to init ClientTransaction: 'NoneType' ... line), no x-client-transaction-id header is sent, and SearchTimeline returns 404.

The logged-in homepage (~271 KB) still embeds ondemand.s, so fetching it with the session cookies makes CT init succeed.

I also ruled out a stale SearchTimeline queryId: even injecting x.com's current live queryId still 404s until the transaction-id header is present — so the header is the real cause.

Fix

Send the auth cookies on the home / ondemand fetches (both use ct_headers). This mirrors the cookie idiom already used in _build_headers:

ct_headers = _gen_ct_headers()
ct_headers["Cookie"] = self._cookie_string or "auth_token=%s; ct0=%s" % (self._auth_token, self._ct0)

Verification

• Guest fetch: home = 35 KB, get_ondemand_file_url → NoneType.group.
• Authed fetch: home = 271 KB, ondemand resolved (e.g. .../ondemand.s.964027da.js), CT init OK.
• After the change, twitter search "openai" and twitter search "codex" --from thsottiaux --min-likes 100 return results instead of 404.

[juavazmor]

Thanks for tracking this down. I hit the same 404 and independently landed on the same root cause. One correction on the diagnosis though, because I think it changes which fix is the right one.

The comment says the logged-in homepage is what embeds ondemand.s, and the diff therefore attaches auth cookies to the https://x.com fetch. In my testing the deciding variable is the path, not authentication. https://x.com/home serves the classic responsive-web bundle even with no auth cookies at all.

Reproducible with a completely virgin session:

from curl_cffi import requests as cr

s = cr.Session(impersonate="chrome")
h = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
                   "AppleWebKit/537.36 (KHTML, like Gecko) "
                   "Chrome/133.0.0.0 Safari/537.36"}

for u in ["https://x.com", "https://x.com/home"]:
    r = s.get(u, headers=h, timeout=15)
    print(u, "->", r.status_code, len(r.text), "ondemand:", "ondemand.s" in r.text)

https://x.com      -> 200  34966 ondemand: False
https://x.com/home -> 200 268217 ondemand: True

No auth_token / ct0 involved. (The session does pick up 3 cookies, but those are guest cookies X sets during the request itself.)

What actually changed is that https://x.com now serves a new lightweight "x-web" logged-out shell, around 35 KB with a single entry-client-logged-out-*.js script, which dropped the ondemand.s reference. /home still serves the old responsive-web bundle (vendor.*.js, main.*.js, i18n/*.js) that contains it.

So the one-liner proposed by @Amirezamky9 in #69 is enough:

- "https://x.com", headers=ct_headers, timeout=10,
+ "https://x.com/home", headers=ct_headers, timeout=10,

I've been running this locally against 0.8.5 and search works again. CT init succeeds and generate_transaction_id returns a valid 94-char id.

Two reasons I'd prefer it over attaching cookies:

1. It keeps CT init working for guest/unauthenticated flows, instead of coupling it to a valid session.
2. It avoids sending auth_token on an HTML scrape that demonstrably doesn't need it. Minor, but it's a credential reaching a request that has no use for it.

Worth noting the current diff also assumes self._cookie_string / self._auth_token / self._ct0 exist on the client, which is a bit more surface than the path change needs.

Either way this should land soon. search is 100% broken on 0.8.5 and the failure is silent (it gets swallowed into a WARNING by the except in _ensure_client_transaction), so it reads like an auth problem when it isn't. Happy to open a PR with the path change if that's useful, or you can fold it into this one.