From 7306a67bd6c034f8f9e305ac6287b549b8e8cf3c Mon Sep 17 00:00:00 2001 From: Brian Date: Sun, 2 Aug 2026 14:42:26 -0700 Subject: [PATCH] OpenAPI and Environment info in README --- README.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/README.md b/README.md index 93bb618..f4fd853 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,9 @@ # `ccnget` -- common crawl news get +Search over 1.4 Billion archived news URLs from CC-NEWS and retrieve contents from WARC files. + +`/lookup?` Endpoint OpenAPI: [🛠️ Try it out (swagger)](https://brian-learns-cc-news-cdx-server.hf.space/docs#/default/lookup_endpoint_lookup_get) | [📖 Redoc](https://brian-learns-cc-news-cdx-server.hf.space/redoc) | [📄 OpenAPI JSON](https://brian-learns-cc-news-cdx-server.hf.space/openapi.json) | [🤗 Hugging Face Space](https://huggingface.co/spaces/brian-learns/cc-news-cdx-server) + ## Quickstart Running the command ```bash @@ -64,6 +68,16 @@ Lookup and retrieve the first result in one step: uv run ccnget fetch "http://example.com" -o article.html ``` +## Environment + + * **`CDX_LOOKUP_URL`** The URL for the CDX server lookup. + * Default: *`https://brian-learns-cc-news-cdx-server.hf.space/lookup`* + * **`CC_CRAWL_BASE_URL`** The base URL for downloading Common Crawl data. + * Default: *`https://data.commoncrawl.org`* + +These variables can be set directly in your shell environment or defined +in a local *.env* file. + ## Diagram While most of this was vibe coded, I drew this architecture diagram in monodraw and came up with the basic approach. Numbers are as of the first test retrospective build. I'm not sure if I'm going to do prospective maintenance.