Handle Google Scholar consent interstitials to prevent empty search results

[Copilot]

Google Scholar searches could return empty/null results even when Scholar was reachable, because some requests were served a consent/interstitial page (HTTP 200) instead of result HTML. This change makes the Google Scholar connector detect and recover from that response shape.

• Root-cause handling: consent/interstitial responses

  • Added explicit detection for Google consent pages (before you continue, consent form action patterns).
  • Initialized Scholar requests with a consent cookie (CONSENT=YES+) to reduce interstitial hits.
  • When a consent page is still returned, retries the same query once before aborting.

• Bot-detection signal coverage

  • Expanded captcha/bot-page detection to include additional Scholar anti-bot wording (unusual traffic from your computer network), avoiding false “successful but empty” flows.

• Regression coverage

  • Added focused tests for:
    • consent cookie initialization on GoogleScholarSearcher session.
    • one-time retry path when first response is a consent page and second response contains valid Scholar results.

if self._is_consent_page(soup, page_text):
    if not consent_retry_attempted:
        consent_retry_attempted = True
        if self.session.cookies.get("CONSENT", domain=".google.com") != self.CONSENT_COOKIE_VALUE:
            self.session.cookies.set("CONSENT", self.CONSENT_COOKIE_VALUE, domain=".google.com")
        continue
    break

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• api.archives-ouvertes.fr
  • Triggering command: /usr/bin/python python -m pytest -q (dns block)
• api.base-search.net
  • Triggering command: /usr/bin/python python -m pytest -q (dns block)
• api.biorxiv.org
  • Triggering command: /usr/bin/python python -m pytest -q (dns block)
• api.core.ac.uk
  • Triggering command: /usr/bin/python python -m pytest -q (dns block)
• api.crossref.org
  • Triggering command: /usr/bin/python python -m pytest -q (dns block)
• api.openaire.eu
  • Triggering command: /usr/bin/python python -m pytest -q (dns block)
• api.semanticscholar.org
  • Triggering command: /usr/bin/python python -m pytest -q (dns block)
• citeseerx.ist.psu.edu
  • Triggering command: /usr/bin/python python -m pytest -q (dns block)
• dblp.org
  • Triggering command: /usr/bin/python python -m pytest -q (dns block)
• doaj.org
  • Triggering command: /usr/bin/python python -m pytest -q (dns block)
• eprint.iacr.org
  • Triggering command: /usr/bin/python python -m pytest -q (dns block)
• eutils.ncbi.nlm.nih.gov
  • Triggering command: /usr/bin/python python -m pytest -q (dns block)
• export.arxiv.org
  • Triggering command: /usr/bin/python python -m pytest -q (dns block)
  • Triggering command: /usr/bin/python python -m pytest -q tests/test_server.py tests/test_config_env.py (dns block)
• papers.ssrn.com
  • Triggering command: /usr/bin/python python -m pytest -q (dns block)
• scholar.google.com
  • Triggering command: /usr/bin/python python -m pytest -q (dns block)
  • Triggering command: /usr/bin/python python -m pytest -q tests/test_google_scholar.py (dns block)
  • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (dns block)
• sci-hub.se
  • Triggering command: /usr/bin/python python -m pytest -q (dns block)
• www.ebi.ac.uk
  • Triggering command: /usr/bin/python python -m pytest -q (dns block)
• www.ssrn.com
  • Triggering command: /usr/bin/python python -m pytest -q (dns block)
• zenodo.org
  • Triggering command: /usr/bin/python python -m pytest -q (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)