langchain-spidra

This package contains the LangChain integration with Spidra, an AI-powered web scraping and crawling API. It lets you scrape, crawl, and batch-scrape the web — as a document loader for RAG pipelines or as agent tools.

Quick Install

pip install langchain-spidra

Get an API key from spidra.io and set it as the SPIDRA_API_KEY environment variable (or pass api_key=...).

export SPIDRA_API_KEY="spd-your-api-key"

Document loader

SpidraLoader loads web content as LangChain Documents. Pick a mode: scrape (one page), crawl (a whole site), or batch (many URLs in parallel).

from langchain_spidra import SpidraLoader

# Scrape a single page
loader = SpidraLoader(url="https://example.com", mode="scrape")
docs = loader.load()
print(docs[0].page_content[:200])
print(docs[0].metadata)

# Crawl an entire docs site
loader = SpidraLoader(url="https://docs.example.com", mode="crawl")
docs = loader.load()

# Batch scrape many URLs
loader = SpidraLoader(
    urls=["https://a.com", "https://b.com", "https://c.com"],
    mode="batch",
)
docs = loader.load()

Tools

Each Spidra capability is also available as a BaseTool you can bind to an agent:

from langchain_spidra import SpidraScrape, SpidraCrawl, SpidraBatchScrape

scrape = SpidraScrape()
result = scrape.invoke({"url": "https://example.com"})
print(result["content"])

crawl = SpidraCrawl()
pages = crawl.invoke({"url": "https://docs.example.com", "max_pages": 20})
for page in pages:
    print(page["url"], page["data"][:100])

batch = SpidraBatchScrape()
items = batch.invoke({"urls": ["https://a.com", "https://b.com"]})

Documentation

• Spidra docs: docs.spidra.io
• Spidra homepage: spidra.io