Great repository — this is a really solid reference for Playwright scraping patterns across Python and Node.js.
One thing I wanted to add from production scraping systems is that once proxy support is introduced, the main bottleneck usually shifts away from browser automation itself and becomes:
proxy stability + session continuity + recovery after soft bans / CAPTCHA triggers
In practice, “rotating proxies” too aggressively often reduces stability rather than improving it, especially when scraping anti-bot protected sites. A more reliable pattern is to treat each proxy as a session boundary, and rebuild the browser context only when failure signals appear.
Example pattern: proxy-per-context with failure recovery
from playwright.sync_api import sync_playwright
PROXIES = [
"http://user:pass@proxy1:port",
"http://user:pass@proxy2:port",
"http://user:pass@proxy3:port",
]
def fetch(url):
with sync_playwright() as p:
for proxy in PROXIES:
browser = None
try:
browser = p.chromium.launch(
headless=True,
proxy={"server": proxy}
)
context = browser.new_context()
page = context.new_page()
page.goto(url, timeout=30000)
# basic soft-block detection example
content = page.content()
if "captcha" in content.lower():
raise Exception("blocked")
print("success via:", proxy)
browser.close()
return content
except Exception:
if browser:
browser.close()
continue
return None
Key idea
In real-world scraping systems:
Proxy rotation is not the core solution
Recovery strategy is more important than rotation strategy
Session stability (cookie + IP consistency) often determines success rate more than browser automation choice
Open question
Curious how others here handle this:
Do you prefer sticky sessions per proxy?
Or frequent rotation with short-lived contexts?
How do you balance retry logic vs detection risk?
Great repository — this is a really solid reference for Playwright scraping patterns across Python and Node.js.
One thing I wanted to add from production scraping systems is that once proxy support is introduced, the main bottleneck usually shifts away from browser automation itself and becomes:
proxy stability + session continuity + recovery after soft bans / CAPTCHA triggers
In practice, “rotating proxies” too aggressively often reduces stability rather than improving it, especially when scraping anti-bot protected sites. A more reliable pattern is to treat each proxy as a session boundary, and rebuild the browser context only when failure signals appear.
Example pattern: proxy-per-context with failure recovery
from playwright.sync_api import sync_playwright
PROXIES = [
"http://user:pass@proxy1:port",
"http://user:pass@proxy2:port",
"http://user:pass@proxy3:port",
]
def fetch(url):
with sync_playwright() as p:
for proxy in PROXIES:
browser = None
try:
browser = p.chromium.launch(
headless=True,
proxy={"server": proxy}
)
Key idea
In real-world scraping systems:
Proxy rotation is not the core solution
Recovery strategy is more important than rotation strategy
Session stability (cookie + IP consistency) often determines success rate more than browser automation choice
Open question
Curious how others here handle this:
Do you prefer sticky sessions per proxy?
Or frequent rotation with short-lived contexts?
How do you balance retry logic vs detection risk?