ccnget -- common crawl news get

Search over 1.4 Billion archived news URLs from CC-NEWS and retrieve contents from WARC files.

/lookup? Endpoint OpenAPI: 🛠️ Try it out (swagger) | 📖 Redoc | 📄 OpenAPI JSON | 🤗 Hugging Face Space

Quickstart

Running the command

uvx --from git+https://github.com/brian-learns/ccnget ccnget fetch http://example.com/ | uvx trafilatura

will lookup http://example.com/ in an index; get the WARC file, offset, and size; get the archived web page from S3; then extract some text with trafilatura -- resulting in:

This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.
More information...

Background

Common Crawl Announced a News Dataset October 4th, 2016, "containing news articles from news sites all over the world." Between then end of June 2016 over 1.4 billion news articles have been archived in the set.

This repository contains a python command for looking up and retrieving URLs from the WARC files on S3.

webrecorder/cdxj-indexer was used to create a Hugging Face Dataset brian-learns/cdx-cc-news with CDXj sorted by month and parquet files. Rocks DB was used to create a bloom filter index that powers a URL lookup tool served from a HuggingFace Space that provides a simple FastAPI /lookup endpoint.

Install

uv add git+https://github.com/brian-learns/ccnget

Usage

Lookup

Search a CC-NEWS index for a URL:

uv run ccnget lookup "http://example.com" --limit 5

Retrieve

Download a specific WARC record by offset and length:

uv run ccnget retrieve \
  --warc-path "crawl-data/CC-NEWS/2017/01/CC-NEWS-20170101071327-00034.warc.gz" \
  --offset 696229346 \
  --length 29897

Save to file instead of stdout:

uv run ccnget retrieve \
  --warc-path "crawl-data/CC-NEWS/2017/01/CC-NEWS-20170101071327-00034.warc.gz" \
  --offset 696229346 \
  --length 29897 \
  -o article.html

Fetch

Lookup and retrieve the first result in one step:

uv run ccnget fetch "http://example.com" -o article.html

Environment

• CDX_LOOKUP_URL The URL for the CDX server lookup.
  • Default: https://brian-learns-cc-news-cdx-server.hf.space/lookup
• CC_CRAWL_BASE_URL The base URL for downloading Common Crawl data.
  • Default: https://data.commoncrawl.org

These variables can be set directly in your shell environment or defined in a local .env file.

Diagram

While most of this was vibe coded, I drew this architecture diagram in monodraw and came up with the basic approach. Numbers are as of the first test retrospective build. I'm not sure if I'm going to do prospective maintenance.

 ┌────────────────────────────────────────┐       
 │  s3://commoncrawl/crawl-data/CC-NEWS/  │       
 │           49.4 TiB raw WARC            │       
 └───────────────┬────────────────────┬▲──┘       
  streamed WARC  │                    ││          
      files      │                    ││          
                 ▼                    ││          
 ┌────────────────────────────────┐   ││          
 │Huggingface Dataset             │   ││          
 │ - cdxj file per month          │   ││          
 │ - 115 GB, 119 files            │   ││          
 └───────────────┬────────────────┘   ││  range   
                 │                    ││ request  
                 │                    ││          
                 ▼                    ││          
 ┌────────────────────────────────┐   ││          
 │Huggingface Space               │   ││          
 │ - rocksdb (75.1 GB, 1116 files)│   ││          
 │ - fastapi /lookup?             │   ││          
 └────┬▲──────────────────────────┘   ││          
      ││                              ││          
      ││                              ││          
      ▼│                              ▼│          
 ┌────────────────────────────────────────┐       
 │                 ccnget                 │       
 │   lookup                    retrieve   │       
 └────────────────────────────────────────┘       

Supporting Code

Besides the code in this repository, code needed to make this work is in

• brian-learns/cdx-cc-news dataset Files tab to build cdxj and rocksdb indexes
• brian-learns/cc-news-cdx-server hf spaces Files tab for the lookup endpoint

See Also

• samples directory with example using duckdb to query the parquet files, and sort of random samples of the data
• man man page for the command line
• api.md pydoc markdown for use as a python module

License

BSD 3-Clause for the code in this revision control repository.

Files retrieved from Common Crawl are subject to Common Crawl Terms of Use and the original publisher's copyright.