Smolagents Sandboxed Visual QA Tool

English · 简体中文 · 日本語 · Español · Português · 한국어

A narrow tool callable that combines a sandbox-safe payload, authorization reference, and per-run call budget.

Introduction

CapSolver is wrapped behind a mock-first boundary in this reference implementation. This repository applies the official Hugging Face smolagents extension surface to a JSON-safe tool contract with a strict per-run budget for sandboxed agent execution. The example is intentionally limited to a generated visual checkpoint from an owned QA environment.

Features

• Uses only JSON-safe strings and dictionaries at the agent boundary.
• Reserves a per-run call slot before adapter execution.
• Rejects unknown or blank authorization references.
• Keeps the mock adapter independent of any code-execution sandbox.
• Returns remaining allowance so the agent can stop predictably.

How It Works

The tool receives Base64, an authorization reference, and a stable run ID. It validates scope and reserves budget before calling the adapter. Since the boundary uses JSON-safe values and no filesystem access, it can be exposed to a ToolCallingAgent without granting broader sandbox capabilities.

The live adapter is opt-in and maps only the documented synchronous recognition result into the tool response. The field names and synchronous behavior are grounded in the official ImageToTextTask request and response contract and createTask operation.

Architecture

sandboxed agent -> JSON-safe tool -> authorization -> call budget -> OCR adapter

Smolagents supports tool-calling and code agents; this example is intentionally compatible with the narrower JSON tool path.

Quick Start

python -m venv .venv
source .venv/bin/activate
pip install -e '.[framework]'
PYTHONPATH=src python -m unittest discover -s tests -v
PYTHONPATH=src python examples/basic_usage.py
PYTHONPATH=src python -c 'import smolagents_qa_tool; print(smolagents_qa_tool.__version__)'

The Hugging Face smolagents quick start uses tests/fixtures/owned-qa.png; it does not load a browser session, API key, or external page.

Usage

Create one budget for the approved run and wrap the callable as a smolagents tool:

tool = build_sandboxed_tool(MockGateway(), CallBudget(limit_per_run=2))
# agent = ToolCallingAgent(tools=[tool], model=model)

The runnable Hugging Face smolagents example in examples/basic_usage.py remains fully offline. A live client cannot be constructed until the operator supplies CAPSOLVER_API_KEY locally and deliberately sets CAPSOLVER_ALLOW_LIVE=1. Store the key outside the repository and agent context; the adapter still expects an approved authorization reference.

Example Output

{'text': 'SMOL-QA', 'remaining_calls': 1, 'mode': 'mock'}

Project Structure

src/smolagents_qa_tool/          core policy, client, tool, and workflow adapters
examples/           runnable mock-first demonstration
tests/unit/         rejection and configuration checks
tests/integration/  end-to-end flow with the bundled synthetic image
docs/               five localized README files
assets/cover.png    shared English repository cover

Testing

Run the full offline suite with:

PYTHONPATH=src python -m unittest tests.unit.test_config tests.integration.test_mock_flow.BudgetedFlowTests -v
python -m compileall -q src

Tests cover independent run budgets, over-budget rejection, and an approved JSON-safe mock result. No test instantiates an LLM, opens a browser, contacts the recognition endpoint, or reads a real credential.

Troubleshooting

• Do not expose filesystem paths as tool arguments.
• Reuse the same run ID only within one approved job.
• Treat exhausted budget as a stop condition.

Responsible Use

Keep tool permissions narrower than the surrounding sandbox, use generated or owned images only, cap calls per run, and never enable network or filesystem access merely to broaden collection. Stop immediately when authorization is withdrawn or the observed scope differs from the approved test plan.

Contributing

Read CONTRIBUTING.md before changing the Hugging Face smolagents adapter. Contributions must keep mock-first tests, explicit authorization, and bounded live execution.

Security

Follow SECURITY.md when reporting an issue in the Hugging Face smolagents integration. Treat image payloads, task identifiers, agent traces, and browser sessions as potentially sensitive even when the fixture is synthetic.

Conclusion

The tool stays small enough for sandbox review while making authorization and cost limits deterministic, while the recognition contract stays aligned with CapSolver.

Disclosure

Developer sharing CapSolver integration examples.

License

MIT. See LICENSE.