Don't stop scrolling on the first round that adds no results - #21
Open
Benjapieres wants to merge 1 commit into
Open
Don't stop scrolling on the first round that adds no results#21Benjapieres wants to merge 1 commit into
Benjapieres wants to merge 1 commit into
Conversation
Fixes zohaibbashir#15. The scroll loop breaks as soon as one round finds the same count as the previous one. Google lazy-loads the results panel, so a flat round normally means the next batch is still in flight, not that the list ended. The loop spins faster than results arrive and exits almost immediately. Measured on current main, scrolling until exhaustion with an unreachable target: query current tolerant "cafeterias en Buenos Aires" 22 54 "restaurantes en Cordoba Argentina" 18 72 Current main gives up after 3-5 rounds in under a second, before the first lazy batch has even landed. Only conclude the list is exhausted after MAX_STAGNANT_SCROLLS consecutive flat rounds, waiting SCROLL_SETTLE_MS between them. The wait is paid only when the count looks stagnant, so runs that keep finding results stay as fast as they are today and the wait-for-element approach from zohaibbashir#2 is preserved on the hot path. Verified end to end with -t 50: the count climbs 22 -> 28 -> 40 -> 43 -> 49 -> 55 and 49 places are scraped, where main would have stopped at the first 22. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #15.
Problem
The scroll loop breaks as soon as one round finds the same count as the previous one:
Google lazy-loads the results panel, so a flat round normally means the next batch is still in flight — not that the list ended. The loop spins faster than results arrive and exits almost immediately, which is what #15 reports.
Measurement
Scroll phase only (no extraction), same query, unreachable target so both run to exhaustion:
main"cafeterias en Buenos Aires""restaurantes en Cordoba Argentina"Current
maingives up in under a second — before the first lazy batch has even landed. Asking for-t 50silently gets you 22.Fix
Only conclude the list is exhausted after
MAX_STAGNANT_SCROLLSconsecutive flat rounds, waitingSCROLL_SETTLE_MSbetween them.The wait is paid only when the count looks stagnant. Runs that keep finding results never hit it, so the hot path stays exactly as fast as today and the wait-for-element approach from #2 is preserved — no blind sleep was added to the loop. Worst case cost when the list really is exhausted is ~3s.
Verification
End to end with
-t 50:49 places scraped (one listing had no name and was skipped by the existing guard), 33 with phone numbers, 35 with websites. On
mainthis same command stops at the first 22.Independent of #19 and #20 — different function, no overlap.
🤖 Generated with Claude Code