langchain-lyrenth

Read the web through Lyrenth from LangChain. A page arrives as clean Markdown instead of raw HTML, typically 80 to 90 percent fewer input tokens for the same page, and carries its canonical source URL so an answer built on it can be attributed.

Pages are served from a standing index of over 2 billion documents, so when many agents read the same URL it does not become many requests to that website.

pip install langchain-lyrenth

Get a free API key at https://lyrenth.com/signup (2,000 reads a month, no card). The components read LYRENTH_API_KEY from the environment.

A tool, for an agent

Give an agent the ability to read any page it decides it needs.

from langchain.agents import create_agent
from langchain_lyrenth import LyrenthReadTool

agent = create_agent(
    model="openai:gpt-5.5",
    tools=[LyrenthReadTool()],
    system_prompt="You are a research assistant.",
)

result = agent.invoke({"messages": [
    {"role": "user", "content": "What does https://example.com/pricing say?"}
]})

The tool is called read_url, takes a single url, and returns the page as Markdown with its title and source URL at the top. It implements the async path too, so it does not block the event loop an agent runs on.

Options: LyrenthReadTool(fresh=True) forces a live re-fetch instead of the stored copy, and max_tokens=4000 caps a long page at roughly that many tokens, trimmed at a clean boundary.

A loader, for a pipeline

When the URLs are known ahead of time, load them as Documents.

from langchain_lyrenth import LyrenthLoader

docs = LyrenthLoader([
    "https://example.com/a",
    "https://example.com/b",
]).load()

docs[0].page_content   # the cleaned Markdown
docs[0].metadata       # {"source", "title", "description", "word_count"}

lazy_load() streams them one at a time. The same fresh and max_tokens options apply, and client= reuses an existing lyrenth.Lyrenth client.

Notes

Reads resolve through a shared cache, so a page many callers want is fetched from its origin a minimal number of times. Lyrenth identifies its crawler, honors robots.txt, and does not train foundation models on crawled content; the policy is at https://lyrenth.com/bot.

MIT licensed. Issues and questions: https://github.com/lyrenth/langchain-lyrenth/issues.