Skip to content

Add looot web provider - #165

Open
walidboulanouar wants to merge 2 commits into
jordan-gibbs:mainfrom
walidboulanouar:feat/looot-web-provider
Open

walidboulanouar wants to merge 2 commits into
jordan-gibbs:mainfrom
walidboulanouar:feat/looot-web-provider

Conversation

@walidboulanouar

Copy link
Copy Markdown

Disclosure up front, since the template asks: I work on looot.

What this changes

Adds looot as a [web] provider: Google organic results with each result's page fetched as Markdown through the same token, so hyperresearch research gets page text rather than SERP snippets. fetch also works on its own. Opt-in only; nothing changes for anyone who does not set LOOOT_TOKEN.

Why

looot runs third-party data APIs through one prepaid token, billed per call, and a failed call is not charged. The provider pins two endpoints, Serper's Google search (serper-search) and Serper's page scrape (serper-webpage), so the response shape is fixed and does not move with looot's provider routing. Like serply, it is JSON over httpx, which the core already carries: no extra, test_packaging.py untouched.

Shape follows serply_provider.py (#121): one module, one branch in get_provider, a README bullet, an [Unreleased] changelog line. Same two safeguards: search keeps the SERP snippet when the fetched page trips looks_like_junk or the fetch fails, and fetched text is clipped to max_characters (8000). looot runs are asynchronous on the server, so the provider asks the API to wait up to 30 s and polls GET /v1/runs/{id} if a run outlives that, up to the 45 s client timeout.

How it was verified

$ python -m pytest tests/
1729 passed, 22 warnings in 206.62s (0:03:26)

$ ruff check src/ tests/
All checks passed!

$ mypy src/hyperresearch/web/looot_provider.py
Success: no issues found in 1 source file

tests/test_web/test_looot_provider.py (10 tests) goes through httpx.MockTransport: factory registration, missing token, the exact run request (pinned endpoint id, wait, bearer header, idempotency key), snippet fallback when the page fetch fails, fetch_content=False, clipping, empty page, a failed run surfacing the API's message, polling an unfinished run, and an HTTP 401. Without the change the file fails at import (ImportError: cannot import name 'looot_provider' from 'hyperresearch.web') and get_provider("looot") raises ValueError: Unknown web provider.

Not run: a live call. The mocked shapes follow Serper's documented responses and looot's run object (runId, status, result, error.message). I can add a live check to this PR if you want one.


Scope check

  • No unrelated reformatting, whitespace, or prose rewriting outside the lines this change needs
  • CHANGELOG.md touched only to add an entry under [Unreleased]
  • No edits to user-facing output strings, docstrings, or test fixtures unrelated to this fix
  • If you ran a formatter or an automated cleanup pass, its results are in a separate PR

Google organic results plus page fetch through one LOOOT_TOKEN, via two
pinned endpoints (serper-search, serper-webpage) run through looot's
POST /v1/runs. Opt-in with [web] provider = "looot"; no new dependency.
@walidboulanouar

Copy link
Copy Markdown
Author

Friendly check-in. I merged current main into the branch today (no conflicts), so it is ready to review against your latest changes. Happy to adjust the provider config or docs if you want it shaped differently.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant