fix: bypass Google cookie consent wall breaking all searches#108
fix: bypass Google cookie consent wall breaking all searches#108robertojrdev wants to merge 1 commit into
Conversation
Google now shows a "Before you continue" cookie consent page before serving any content. This causes all flight searches to fail with "No flights found" since the HTML returned is the consent wall, not flight results. Fix: pre-set the SOCS cookie on the primp Client before making the request. This tells Google that consent has been given, and the actual flight results page is returned. Also removes a stray `print(data)` debug statement in parser.py that dumped the raw JSON payload to stdout on every search. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
📝 WalkthroughWalkthroughThis PR contains two targeted improvements to the flight-fetching system: a SOCS cookie is pre-set in the request handler to bypass Google's consent wall that blocks results, and a debug print statement is removed from the parser for cleaner output during extraction. ChangesFlight Fetching Improvements
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Warning Review ran into problems🔥 ProblemsGit: Failed to clone repository. Please run the Tip 💬 Introducing Slack Agent: The best way for teams to turn conversations into code.Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.
Built for teams:
One agent for your entire SDLC. Right inside Slack. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
fast_flights/fetcher.py (1)
86-90: ⚡ Quick winHarden consent-cookie handling to avoid repeat outages.
The hardcoded
SOCSpayload is fragile; if Google changes it, searches can silently fail again. Please move this value to a named constant/config override (env var) so it can be rotated without code edits.Suggested change
+import os + URL = "https://www.google.com/travel/flights" +DEFAULT_SOCS_COOKIE = "CAISNQgDEitib3FfaWRlbnRpdHlmcm9udGVuZHVpc2VydmVyXzIwMjMwODE1LjA3X3AxGgJlbiACGgYIgJnPpwY" +SOCS_COOKIE = os.getenv("FAST_FLIGHTS_SOCS_COOKIE", DEFAULT_SOCS_COOKIE) ... - client.set_cookies(URL, {"SOCS": "CAISNQgDEitib3FfaWRlbnRpdHlmcm9udGVuZHVpc2VydmVyXzIwMjMwODE1LjA3X3AxGgJlbiACGgYIgJnPpwY"}) + client.set_cookies(URL, {"SOCS": SOCS_COOKIE})🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@fast_flights/fetcher.py` around lines 86 - 90, The hardcoded SOCS cookie in the call to client.set_cookies with URL is fragile; change it to read from a named constant that can be overridden via an environment variable (e.g., FAST_FLIGHTS_SOCS or SOCS_COOKIE) so operators can rotate the payload without code edits; update the code around client.set_cookies to load process.env.FAST_FLIGHTS_SOCS (fall back to the existing literal only if env var is absent), surface a clear log/warning when falling back, and ensure the constant name is used wherever SOCS is referenced so the value can be changed centrally or injected by config.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@fast_flights/fetcher.py`:
- Around line 86-90: The hardcoded SOCS cookie in the call to client.set_cookies
with URL is fragile; change it to read from a named constant that can be
overridden via an environment variable (e.g., FAST_FLIGHTS_SOCS or SOCS_COOKIE)
so operators can rotate the payload without code edits; update the code around
client.set_cookies to load process.env.FAST_FLIGHTS_SOCS (fall back to the
existing literal only if env var is absent), surface a clear log/warning when
falling back, and ensure the constant name is used wherever SOCS is referenced
so the value can be changed centrally or injected by config.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: b4eb98b2-e76b-4d0a-a45a-0cdd9c0ae461
📒 Files selected for processing (2)
fast_flights/fetcher.pyfast_flights/parser.py
💤 Files with no reviewable changes (1)
- fast_flights/parser.py
Google now blocks requests behind a "Before you continue" consent wall, causing the upstream parser to crash on the ErrorResponse payload it gets back. Adapted from upstream PRs AWeirdDev#108 and AWeirdDev#105. - preset SOCS cookie on the Google domain so Google skips the consent wall - fall back to parsing and submitting the consent "Reject all" form if the wall still appears - raise typed FlightsError on ErrorResponse / missing ds:1 script instead of crashing with JSONDecodeError - drop debug print(data) from parser - bump example.py date past today so it actually runs Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Summary
SOCScookie on theprimpClient before making the request to Google Flights. Google now shows a "Before you continue" cookie consent page before serving any content, causing all flight searches to fail with "No flights found".print(data)debug statement inparser.pythat dumped the raw JSON payload to stdout on every search.Problem
Google changed their consent flow — all requests to
google.com/travel/flightsnow return a cookie consent wall instead of flight results. Theprimpclient receives HTML containing "Before you continue to Google" instead of the expected flight data page. This breaks every search regardless of route, date, or configuration.Fix
Setting the
SOCScookie tells Google that cookie consent has already been given. The consent wall is skipped and the actual flight results page is returned.Testing
Before fix:
After fix:
🤖 Generated with Claude Code
Summary by CodeRabbit
Bug Fixes
Chores