[BUG] Oxylabs tools raise IndexError on any failed request and return failed scrapes as successes

[oxy-giedrius]

Opening this at @Vidit-Ostwal's request to track the problem behind #7044.

Description

All four Oxylabs tools (OxylabsUniversalScraperTool,
OxylabsGoogleSearchScraperTool, OxylabsAmazonSearchScraperTool,
OxylabsAmazonProductScraperTool) mishandle every Web Scraper API response that
isn't a success. Each _run does an unchecked response.results[0].

The oxylabs SDK logs transport and validation errors and returns an empty
Response rather than raising, so any rejected request reaches that line with
results == []. Three distinct failures follow:

1. Rejected requests raise IndexError: list index out of range. Invalid
   credentials — the most likely first-run mistake — produce this, with no
   indication of the cause unless the app happens to surface the
   oxylabs.internal.api logger.
2. Failed scrapes are returned as successes. A result with a non-2xx
   status_code (e.g. a 404 for a nonexistent ASIN) carries empty content. The
   tools return it, so the agent receives "[]" as though the scrape worked.
   The status_code is present on the result object and never read.
3. Non-dict content is returned as a Python repr. Only dict is
   JSON-encoded, with a str() fallback. parsing_instructions commonly yields
   a list, so agents get [{'title': '...'}] — single-quoted and not parseable.

Steps to Reproduce

No valid Oxylabs account is needed to see (1) — invalid credentials are enough:

1. pip install 'crewai[tools]' oxylabs
2. Run the snippet below (deliberately invalid credentials).
3. Observe IndexError instead of an authentication error.

import logging
logging.disable(logging.CRITICAL)  # a normal app that doesn't surface library logs

from crewai_tools import OxylabsUniversalScraperTool

tool = OxylabsUniversalScraperTool(username="invalid-user", password="invalid-pass")
print(tool.run(url="https://ip.oxylabs.io/location"))

For (2), with valid credentials, request a nonexistent ASIN
(OxylabsAmazonProductScraperTool(config={"parse": True}).run(query="B0BW4SWWLZ"))
— the upstream 404 comes back as the string "[]".

Expected behavior

The tool should report that the request failed and why — ideally as a
ToolFailure, so the agent gets an actionable message and the framework records
the call as unsuccessful rather than counting it as a success. A non-2xx result
should likewise be reported instead of returned as empty content.

Screenshots/Code snippets

Against current main:

Traceback (most recent call last):
  File "<stdin>", line 5, in <module>
  File ".../crewai/tools/base_tool.py", line 338, in run
    result = self._run(*args, **kwargs)
  File ".../tools/oxylabs_universal_scraper_tool/oxylabs_universal_scraper_tool.py", line 157, in _run
    content = response.results[0].content
              ~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

Operating System

macOS (26.6) — not OS-specific; the code path is platform-independent.

Python Version

3.13 (not offered in the template dropdown; also reproduces on 3.10–3.12, as the
code path has no version-specific behavior)

crewAI Version

1.15.16

crewAI Tools Version

1.15.16

Virtual Environment

Venv

Evidence

The relevant line, identical in all four tools — e.g.
lib/crewai-tools/src/crewai_tools/tools/oxylabs_universal_scraper_tool/oxylabs_universal_scraper_tool.py:

response = self.oxylabs_api.universal.scrape_url(
    url,
    **self.config.model_dump(exclude_none=True),
)

content = response.results[0].content   # <-- results == [] on any rejected request

if isinstance(content, dict):
    return json.dumps(content)

return str(content)                      # <-- a list becomes a Python repr

With invalid credentials the SDK emits (log only, then returns an empty
response):

ERROR:oxylabs.internal.api:HTTP error occurred: 401 Client Error: Unauthorized

Two secondary problems found while investigating:

• oxylabs is pinned ==2.0.0 in lib/crewai-tools/pyproject.toml, while 3.0.0
  has been out since March. 3.x keeps the RealtimeClient surface these tools
  use; I verified all four tools against the live API on both versions.
• OxylabsGoogleSearchScraperConfig has no locale field, though the docs list
  locale as a supported parameter — so it is silently dropped today.

Possible Solution

#7044 fixes all of the above and is ready to go:

• rejected requests and non-2xx results return a ToolFailure (retryable set
  for 429/5xx), following the existing precedent in
  crewai_platform_action_tool.py
• non-string content is JSON-serialized
• adds the missing locale option
• relaxes the pin to oxylabs>=2.0.0,<4 (the lockfile keeps 2.0.0, so this
  permits the upgrade without forcing it)
• moves the client construction and response handling that all four tools
  duplicated verbatim into a shared OxylabsBaseTool, following the existing
  SerpApiBaseTool pattern, so the handling lives in one place instead of being
  copy-pasted four times

All four tools were verified against the live Oxylabs API on oxylabs 2.0.0
and 3.0.0, each failure mode reproduced before and after, plus an end-to-end
Crew run confirming the agent both receives scraped data on success and a
usable message on failure. 24 tests added for the failure paths; ruff, mypy
and the full lib/crewai-tools/ suite pass. The regenerated tool.specs.json
changes only by the new locale field, which is evidence the refactor left the
tools' public surface untouched.

Happy to split it into two PRs (fixes / refactor) or rebase onto current main
— whichever is easier to review.

Additional context

I work at Oxylabs and can verify changes against the live API, so I'm able to
help maintain this integration going forward. Taking over from
@oxy-rostyslav, who added these tools in #2905 and has since left the company.

Note #7044 is currently closed and I can't reopen it (that needs write access);
reopening it, or telling me to raise a fresh PR, both work for me.

Per CONTRIBUTING.md: this report was prepared with the help of an AI coding
assistant and so needs the llm-generated label, which I can't apply from
outside the repo — could a maintainer add it?

[linear]

OSS-147