chore(oxylabs): allow the 3.x oxylabs SDK - #7330
Closed
oxy-giedrius wants to merge 2 commits into
Closed
Conversation
The oxylabs SDK logs HTTP errors and returns an empty response rather than
raising, so the unchecked `response.results[0]` in every Oxylabs tool turned a
rejected request into `IndexError: list index out of range`. Invalid credentials
-- the most likely first-run mistake -- gave no indication of the cause. A
result carrying a non-2xx `status_code` had the same problem one level down: the
job ran, the page did not come back, and the tool returned its empty content as
though the scrape had succeeded, handing the agent "[]".
Both are now reported as a `ToolFailure` naming what went wrong, so the agent
gets something it can act on and the framework records the call as failed:
401 Unauthorized
400 Bad Request - Parameter `parsing_instructions` can be used just with
`parse` parameter set to `true`.
Because the SDK keeps the cause only in its own log, the failing call is run
with a handler attached to the `oxylabs` logger and the status, the API's
explanation and timeouts are read back off it. `code` and `retryable` are set
from the status, so 429 and 5xx are marked worth retrying. Nothing about the
caller's logging configuration is changed; an application that has silenced the
SDK still gets the generic failure.
Content that is neither a string nor a dict is also serialized properly:
`parsing_instructions` commonly yields a list, and the previous `str()`
fallback produced a Python repr with single quotes instead of JSON.
The client construction and response handling these four tools duplicated
verbatim now live in a shared `OxylabsBaseTool`, following the existing
`SerpApiBaseTool` pattern, so the handling above exists in one place. The
generated tool specs change only by the new `locale` field, confirming the
tools' public surface is otherwise untouched.
Also add the `locale` option to the Google Search config, which the docs
already documented but the config model silently dropped, and correct two
copy-paste errors in the docs across all four locales.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`oxylabs` was pinned to exactly 2.0.0, so consumers could not take 3.0.0, out since March. 3.x keeps the `RealtimeClient` surface these tools use, and all four tools plus their failure paths were verified against the live API on both 2.0.0 and 3.0.0. The lockfile keeps oxylabs at 2.0.0, so this permits the upgrade rather than forcing it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
Thanks for the pull request. First-time contributors need an associated open issue before we can review a PR.
See the contributing guide. |
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (13)
📝 WalkthroughWalkthroughOxylabs scraper tools now share ChangesOxylabs scraper consolidation
Sequence Diagram(s)sequenceDiagram
participant ScraperTool
participant OxylabsBaseTool
participant RealtimeClient
participant ToolFailure
ScraperTool->>OxylabsBaseTool: call _scrape with a scraper endpoint
OxylabsBaseTool->>RealtimeClient: send authenticated request
RealtimeClient-->>OxylabsBaseTool: return response or SDK logs
OxylabsBaseTool->>ToolFailure: report diagnosed errors
Suggested reviewers: ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related to #7306.
Stacked on #7044, as requested in review — the failure-handling and base-tool work
stays there, this carries only the dependency bump.
Summary
oxylabsis pinned to exactly2.0.0inlib/crewai-tools/pyproject.toml, soconsumers can't take 3.0.0, released in March. This relaxes it to
oxylabs>=2.0.0,<4.3.x keeps the
RealtimeClientsurface these tools use (RealtimeClient(username, password, **kwargs)withsdk_type,results[0].content,results[0].status_code,and the
universal/google/amazonnamespaces), and adds sources rather thanremoving any.
The lockfile keeps
oxylabsat 2.0.0, so this permits the upgrade without forcingit on anyone.
Testing
All four Oxylabs tools were verified against the live Oxylabs API on
oxylabs2.0.0 and 3.0.0, along with each failure path from #7044 (invalid credentials, a
config the source rejects, an upstream 404, and a timeout):
OxylabsUniversalScraperToolOxylabsGoogleSearchScraperToolOxylabsAmazonSearchScraperToolOxylabsAmazonProductScraperTooluv lock --checkpasses with uv 0.11.3, the version pinned in.pre-commit-config.yamland the CI workflows. The lockfile change is a single line —the recorded specifier. Regenerating it wholesale also rewrote ~60 lines of unrelated
platform-marker churn and re-stamped the relative
exclude-newercutoff, so I appliedonly the specifier change and verified the result is consistent.
Note on the diff
Because a PR from a fork can't target another fork branch, this is opened against
mainand therefore also shows #7044's commit. Once #7044 merges, the diff herereduces to the two-line dependency change. Happy to rebase it whenever that lands.
AI-generated contribution
Per
CONTRIBUTING.md, this was authored with an AI coding assistant (Claude Code) andneeds the
llm-generatedlabel, which I can't apply from a fork — could a maintaineradd it?