Repository navigation
Add looot web provider - #165
Open
walidboulanouar wants to merge 2 commits into
Open
walidboulanouar wants to merge 2 commits into
walidboulanouar wants to merge 2 commits into
Conversation
Google organic results plus page fetch through one LOOOT_TOKEN, via two pinned endpoints (serper-search, serper-webpage) run through looot's POST /v1/runs. Opt-in with [web] provider = "looot"; no new dependency.
Author
|
Friendly check-in. I merged current main into the branch today (no conflicts), so it is ready to review against your latest changes. Happy to adjust the provider config or docs if you want it shaped differently. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Disclosure up front, since the template asks: I work on looot.
What this changes
Adds
loootas a[web] provider: Google organic results with each result's page fetched as Markdown through the same token, sohyperresearch researchgets page text rather than SERP snippets.fetchalso works on its own. Opt-in only; nothing changes for anyone who does not setLOOOT_TOKEN.Why
looot runs third-party data APIs through one prepaid token, billed per call, and a failed call is not charged. The provider pins two endpoints, Serper's Google search (
serper-search) and Serper's page scrape (serper-webpage), so the response shape is fixed and does not move with looot's provider routing. Likeserply, it is JSON over httpx, which the core already carries: no extra,test_packaging.pyuntouched.Shape follows
serply_provider.py(#121): one module, one branch inget_provider, a README bullet, an[Unreleased]changelog line. Same two safeguards:searchkeeps the SERP snippet when the fetched page tripslooks_like_junkor the fetch fails, and fetched text is clipped tomax_characters(8000). looot runs are asynchronous on the server, so the provider asks the API to wait up to 30 s and pollsGET /v1/runs/{id}if a run outlives that, up to the 45 s client timeout.How it was verified
tests/test_web/test_looot_provider.py(10 tests) goes throughhttpx.MockTransport: factory registration, missing token, the exact run request (pinned endpoint id,wait, bearer header, idempotency key), snippet fallback when the page fetch fails,fetch_content=False, clipping, empty page, a failed run surfacing the API's message, polling an unfinished run, and an HTTP 401. Without the change the file fails at import (ImportError: cannot import name 'looot_provider' from 'hyperresearch.web') andget_provider("looot")raisesValueError: Unknown web provider.Not run: a live call. The mocked shapes follow Serper's documented responses and looot's run object (
runId,status,result,error.message). I can add a live check to this PR if you want one.Scope check
CHANGELOG.mdtouched only to add an entry under[Unreleased]