diff --git a/README.md b/README.md index eb11565..fbd9194 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # langchain-wellmarked -The official [LangChain](https://www.langchain.com/) integration for **[WellMarked](https://wellmarked.io)** — load any URL as clean Markdown `Document`s. +The official [LangChain](https://www.langchain.com/) integration for **[WellMarked](https://wellmarked.io)** — load any URL as clean Markdown `Document`s, or retrieve them straight from a web search. ```bash pip install langchain-wellmarked @@ -50,6 +50,28 @@ docs[0].metadata["depth"] # how far from the root URL this page sits In `crawl` mode, pages that fail to extract (timeouts, robots-disallowed, no content) are skipped; only successful pages become `Document`s. +## Retriever + +`WellMarkedRetriever` is the natural surface for RAG chains where the context should come from the live web: each retrieval searches the web, extracts the top results to clean Markdown, and returns them as `Document`s — search and extraction in one round trip, no URLs to wrangle. Search requires a Pro plan or above. + +```python +from langchain_wellmarked import WellMarkedRetriever + +retriever = WellMarkedRetriever(num_results=5) +docs = retriever.invoke("best open-source vector databases") + +docs[0].page_content # clean Markdown of a result page +docs[0].metadata # {"source": ..., "title": ..., "snippet": ...} +``` + +| Parameter | Default | Description | +|---------------|-----------|----------------------------------------------------------------| +| `api_key` | env var | WellMarked API key; falls back to `WELLMARKED_API_KEY` | +| `num_results` | `5` | How many results to fetch + extract (clamped to 1..10) | +| `render_js` | `False` | Render JS-heavy result pages with a headless browser (Pro and above) | + +Only successfully extracted results become `Document`s; a result whose page timed out or was blocked is skipped. + ## Related - [`wellmarked`](https://pypi.org/project/wellmarked/) — the underlying Python SDK (this package wraps it) diff --git a/pyproject.toml b/pyproject.toml index c3903cc..a14e433 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -40,6 +40,15 @@ dev = [ line-length = 100 target-version = "py39" +[tool.ruff.lint] +# Pin ruff's classic default rule set (pyflakes + the pycodestyle subset). +# CI installs ruff unpinned, so without an explicit selection `ruff check` +# follows whatever the *current* ruff release defaults to — and newer releases +# broadened those defaults (pyupgrade/simplify/isort/ruff-native), which turns +# CI red on unchanged code. Adopting those rule families is a deliberate, +# separate migration, not an accident of an upstream release. +select = ["E4", "E7", "E9", "F"] + [project.urls] Homepage = "https://wellmarked.io" Documentation = "https://wellmarked.io/docs"