Ccnget library / add python api

.ai/notes6-answers.txt

@@ -0,0 +1,17 @@
+Answers:
+
+1. API surface: (lookup, retrieve_record) as clean public APIs that callers compose themselves
+
+2. What should the return type look like? Something like a dataclass FetchResult(payload, http_headers, warc_headers, surt_key, timestamp)
+
+3. HTML parser integration: 
+
+```python pseudocode
+import ccnget 
+from selectolax.lexbor import LexborHTMLParser
+resp = ccnget.fetch("http://example.com")
+html = resp.payload
+tree = LexborHTMLParser(html)
+```
+
+4. Backward compatibility: I like the CLI interface pretty well now, would prefer to keep it the same

.ai/notes6.txt

@@ -0,0 +1,9 @@
+source: src/ccnget/geturl.py
+run: uv run ccnget ...
+test: make test
+
+This command line is working well, but now I want to use it as a library.
+
+Please analyze how this might be refactored to better support use as a module with various html parsers such as https://github.com/rushter/selectolax
+
+as a caller of the API, I might like to get access to the response headers, or the WARC headers

Makefile

@@ -8,7 +8,7 @@ help:
 	@echo "  make test       Run static checks followed immediately by pytest"
 	@echo "  make clean      Wipe out test tool cache tracking footprints"
 	@echo "  make init       Initialize new project with uv and test setup"
-	@echo "  make man        Create man page"
+	@echo "  make mandoc     Create man page and pydoc markdown for api.md"
 
 check:
 	@echo "\n— [An extremely fast Python linter and code formatter](https://docs.astral.sh/ruff/)"
@@ -48,6 +48,8 @@ checkdeps:
 testpackages:
 	uv add --dev ruff bandit vulture refurb ty pytest interrogate argparse-manpage
 
+mandoc: man doc
+
 man:
 	mkdir -p man
 	uv run argparse-manpage \
@@ -59,6 +61,9 @@ man:
 		--include man/__envars.inc \
 		> man/ccnget.1
 
+doc:
+	uvx pydoc-markdown
+
 export GIT_CEILING_DIRECTORIES	# can influence `uv init` behaviour
 pyproject.toml:
 	uv init --package .

README.md

@@ -116,6 +116,11 @@ Besides the code in this repository, code needed to make this work is in
  * [`brian-learns/cdx-cc-news` dataset Files tab](https://huggingface.co/datasets/brian-learns/cdx-cc-news/tree/main) to build cdxj and rocksdb indexes
  * [`brian-learns/cc-news-cdx-server` hf spaces Files tab](https://huggingface.co/spaces/brian-learns/cc-news-cdx-server/tree/main) for the lookup endpoint
 
+## See Also
+ * [`samples`](./samples/) directory with example using `duckdb` to query the parquet files, and sort of random samples of the data
+ * [`man`](./man/) man page for the command line
+ * [`api.md`](./api.md) pydoc markdown for use as a python module
+
 ## License
 
 BSD 3-Clause for the code in this revision control repository.

api.md

@@ -0,0 +1,292 @@
+<a id="ccnget"></a>
+
+# ccnget
+
+ccnget -- lookup URLs and get archived pages from Common Crawl News.
+
+Library usage
+-------------
+>>> import ccnget
+>>> result = ccnget.fetch("http://example.com")
+>>> print(result.surt_key, result.timestamp)
+>>> # Parse with any HTML parser
+>>> from selectolax.lexbor import LexborHTMLParser
+>>> tree = LexborHTMLParser(result.payload)
+
+Or use the lower-level API:
+>>> lr = ccnget.lookup("http://example.com", limit=5)
+>>> for entry in lr.entries:
+...     result = ccnget.retrieve(entry.warc_path, entry.offset, entry.length)
+
+<a id="ccnget.geturl"></a>
+
+# ccnget.geturl
+
+CLI entry-point for ccnget.
+
+Uses the library API (ccnget.api) for all logic.
+
+<a id="ccnget.geturl.limited_int"></a>
+
+#### limited\_int
+
+```python
+def limited_int(val_str: str) -> int
+```
+
+Checks that input is an integer between 1 and 100.
+
+<a id="ccnget.geturl.lookup_cmd"></a>
+
+#### lookup\_cmd
+
+```python
+def lookup_cmd(args: argparse.Namespace) -> None
+```
+
+Execute the lookup subcommand.
+
+<a id="ccnget.geturl.retrieve_cmd"></a>
+
+#### retrieve\_cmd
+
+```python
+def retrieve_cmd(args: argparse.Namespace) -> None
+```
+
+Execute the retrieve subcommand.
+
+<a id="ccnget.geturl.fetch_cmd"></a>
+
+#### fetch\_cmd
+
+```python
+def fetch_cmd(args: argparse.Namespace) -> None
+```
+
+Execute the fetch subcommand: lookup then retrieve the first result.
+
+<a id="ccnget.geturl.get_version"></a>
+
+#### get\_version
+
+```python
+def get_version() -> str
+```
+
+Get version from pyproject.toml
+
+<a id="ccnget.geturl.get_parser"></a>
+
+#### get\_parser
+
+```python
+def get_parser() -> argparse.ArgumentParser
+```
+
+Build and return the ArgumentParser for ccnget.
+
+<a id="ccnget.geturl.main"></a>
+
+#### main
+
+```python
+def main(argv: Optional[list[str]] = None) -> None
+```
+
+Parse CLI arguments and dispatch to subcommands.
+
+<a id="ccnget.api"></a>
+
+# ccnget.api
+
+Public library API for ccnget.
+
+Lookup and retrieve archived web pages from the Common Crawl News dataset.
+
+Example
+-------
+>>> import ccnget
+>>> result = ccnget.fetch("http://example.com")
+>>> print(result.surt_key, result.timestamp)
+>>> tree = LexborHTMLParser(result.payload)
+
+<a id="ccnget.api.LookupEntry"></a>
+
+## LookupEntry Objects
+
+```python
+@dataclass
+class LookupEntry()
+```
+
+One CDX index hit returned by the lookup API.
+
+<a id="ccnget.api.LookupResult"></a>
+
+## LookupResult Objects
+
+```python
+@dataclass
+class LookupResult()
+```
+
+Result of a CDX index lookup.
+
+<a id="ccnget.api.FetchResult"></a>
+
+## FetchResult Objects
+
+```python
+@dataclass
+class FetchResult()
+```
+
+Result of fetching an archived page.
+
+Attributes
+----------
+payload : bytes
+    Raw response body (typically HTML).
+http_headers : dict[str, str]
+    HTTP response headers from inside the WARC record.
+warc_headers : dict[str, str]
+    WARC record headers.
+surt_key : str
+    SURT-formatted URL key from the CDX index.
+timestamp : str
+    WARC timestamp (YYYYMMDDhhmmss).
+warc_path : str
+    Path to the WARC file on Common Crawl storage.
+
+<a id="ccnget.api.lookup"></a>
+
+#### lookup
+
+```python
+def lookup(url: str,
+           *,
+           exact: bool = False,
+           limit: int = 10,
+           at: str | None = None,
+           cdx_url: str = CDX_LOOKUP_URL) -> LookupResult
+```
+
+Search the CC-NEWS CDX index for *url*.
+
+Parameters
+----------
+url : str
+    URL to search for.
+exact : bool
+    Require exact match.
+limit : int
+    Maximum number of results (1-100).
+at : str | None
+    Timestamp filter (YYYYMMDDhhmmss).
+cdx_url : str
+    Override the CDX lookup endpoint.
+
+Returns
+-------
+LookupResult
+
+<a id="ccnget.api.retrieve"></a>
+
+#### retrieve
+
+```python
+def retrieve(warc_path: str,
+             offset: int,
+             length: int,
+             *,
+             base_url: str = CC_CRAWL_BASE_URL,
+             surt_key: str = "",
+             timestamp: str = "") -> FetchResult
+```
+
+Retrieve a single WARC record via byte-range request.
+
+Parameters
+----------
+warc_path : str
+    Path within Common Crawl storage (e.g. ``crawl-data/CC-NEWS/...``).
+offset : int
+    Byte offset of the record.
+length : int
+    Byte length of the record.
+base_url : str
+    Override the Common Crawl base URL.
+surt_key : str
+    SURT key from the CDX index (populated by ``fetch()``).
+timestamp : str
+    Timestamp from the CDX index (populated by ``fetch()``).
+
+Returns
+-------
+FetchResult
+
+<a id="ccnget.api.fetch"></a>
+
+#### fetch
+
+```python
+def fetch(url: str,
+          *,
+          exact: bool = False,
+          at: str | None = None,
+          cdx_url: str = CDX_LOOKUP_URL,
+          base_url: str = CC_CRAWL_BASE_URL) -> FetchResult
+```
+
+Lookup *url* in the CDX index and retrieve the first archived result.
+
+Convenience wrapper around :func:`lookup` + :func:`retrieve`.
+
+Parameters
+----------
+url : str
+    URL to search for.
+exact : bool
+    Require exact match.
+at : str | None
+    Timestamp filter (YYYYMMDDhhmmss).
+cdx_url : str
+    Override the CDX lookup endpoint.
+base_url : str
+    Override the Common Crawl base URL.
+
+Returns
+-------
+FetchResult
+
+<a id="ccnget.api.CcngetError"></a>
+
+## CcngetError Objects
+
+```python
+class CcngetError(Exception)
+```
+
+Base exception for ccnget.
+
+<a id="ccnget.api.NotFoundError"></a>
+
+## NotFoundError Objects
+
+```python
+class NotFoundError(CcngetError)
+```
+
+Raised when a URL has no matches in the CDX index.
+
+<a id="ccnget.api.NoRecordError"></a>
+
+## NoRecordError Objects
+
+```python
+class NoRecordError(CcngetError)
+```
+
+Raised when a WARC segment contains no response record.
+

man/ccnget.1

@@ -1,4 +1,4 @@
-.TH CCNGET "1" "2026\-08\-02" "ccnget 0.1.2" "Generated Python Manual"
+.TH CCNGET "1" "2026\-08\-02" "ccnget 0.1.3" "Generated Python Manual"
 .SH NAME
 ccnget
 .SH SYNOPSIS

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ccnget"
-version = "0.1.2"
+version = "0.1.3"
 description = "lookup urls and get files from Common Crawl News"
 readme = "README.md"
 authors = [
@@ -33,6 +33,10 @@ dev = [
     "vulture>=2.16",
 ]
 
+[tool.pydoc-markdown.renderer]
+type = "markdown"
+filename = "api.md"
+
 [tool.build_manpages]
 manpages = [
     "man/ccnget.1:function=get_parser:module=ccnget.geturl",