Webchild Scripts

This repository stores small utility scripts used by the WEBCHILD project. Each script is a standalone tool for common tasks related to the project.

Available scripts

• dir_to_warc.py

  • Description: Walks a directory and archives every file into a WARC (Web ARChive) file. Each file is written as a WARC "response" record with a synthetic URL based on a provided base URL. Useful for preserving old sites, static sites or file trees in a format compatible with web archiving tools.

  • Basic usage:

    python3 dir_to_warc.py <input_dir> <output_file.warc.gz> <base_url> [--date ISO_DATE]

    Example:

    python3 dir_to_warc.py site/ site.warc.gz https://example.com --date 2025-10-17T12:00:00Z

  • Notes:

    • If --date is not provided the script uses the current UTC time as the WARC date.
    • The script attempts to guess MIME types using Python's mimetypes module; unknown types default to application/octet-stream.

• merge_warcs.py

  • Description: Recursively finds .warc and .warc.gz files in a directory and merges their records into a single gzipped WARC file. Useful for consolidating many small WARC archives into one file for easier storage or processing.

  • Basic usage:

    python3 merge_warcs.py <input_dir> <output_file.warc.gz>

    Example:

    python3 merge_warcs.py warcs/ merged.warc.gz

  • Notes:

    • The script preserves original records and headers. It writes records exactly as read using warcio's ArchiveIterator and WARCWriter.
    • Input files are detected case-insensitively by the extensions .warc and .warc.gz.
    • If you need regenerated WARC-Record-IDs, normalized timestamps, or header changes, the script can be extended to transform records before writing.
    • Ensure the warcio package is installed (pip install warcio). If warcio isn't available the script will raise an error and print installation advice.

• warc_content_pie.py

  • Description: Analyze a WARC file and create a pie chart of MIME type distribution.

  • Basic usage:

    python3 warc_content_pie.py <input.warc.gz> [--min-percent 1.0] [--output pie.png]

    Example (save chart):

    python3 warc_content_pie.py example.warc.gz --min-percent 0.5 --output mimetypes.png

  • Notes:

    • MIME types that individually account for less than --min-percent percent of the total are grouped into a single "Other" slice.

• count_warc.py

  • Description: Count WARC response records in a WARC/WARC.GZ file using warcio's ArchiveIterator.

  • Basic usage:

    python3 count_warc.py <input.warc.gz>

    Example:

    python3 count_warc.py example.warc.gz
    # -> Response records: 12345

  • Notes:

    • Requires the warcio package. Install with pip install warcio.
    • The script prints the number of response records (common representation of archived HTTP responses).
    • Works with compressed (.warc.gz) and uncompressed WARC files when passed as a filename to the script (if your Python open() supports reading the compressed file directly, or use gunzip -c file.warc.gz | python3 count_warc.py -).

• validate_warc.py

  • Description: Validate the integrity of WARC/WARC.GZ files by checking record digests and reading content streams.

  • Basic usage:

    python3 validate_warc.py <input.warc.gz>

  • Notes:

    • Requires the warcio package. Install with pip install warcio.
    • Validates WARC integrity by reading record content streams and checking digests.
    • Works with both compressed (.warc.gz) and uncompressed WARC files.

• markdown_render_html_from_warc.py

  • Description: Convert HTML pages in a WARC to Markdown files.

  • Basic usage:

    python3 markdown_render_html_from_warc.py <input.warc.gz> [--output-dir markdown_pages]

    Example:

    python3 markdown_render_html_from_warc.py example.warc.gz --output-dir md_out/

  • Notes:

    • The script uses html2text to convert HTML to Markdown. Install it with pip install html2text.

• harvest_comparator.py

  • Description: Test script which primary use is intended for testing the limiter function in our fork of python-wayback-machine-downloader during implementation. In general it compares two harvest directories and visualizes the distribution of captured snapshots over time. Analyzes file statistics and generates histograms comparing harvest patterns between two directories (e.g., with and without rate limiting).

  • Basic usage:

    python3 harvest_comparator.py <original_dir> <modified_dir>

    Example:

    python3 harvest_comparator.py harvest_unlimited/ harvest_limited/

  • Features:

    • Collects and analyzes file information (count, total size, file types) for both directories
    • Extracts snapshot dates from waybackup_snapshots folder structure
    • Generates side-by-side histograms showing harvest date distribution
    • Useful for comparing harvest results before/after applying rate limiters or other modifications

  • Notes:

    • Requires pandas and matplotlib packages. Install with pip install pandas matplotlib.
    • Expects directories to contain a waybackup_snapshots subdirectory with snapshot data organized by site and timestamp.
    • Snapshot timestamps are expected to be 14-digit strings in YYYYMMDDHHmmss format.

• ia_stats_fetcher.py

  • Description: First VERY RAW and unpolished stab at fetching Internet Archive (Wayback Machine) statistics for a list of URLs. Queries the CDX API to retrieve capture counts by MIME type and unique URL counts for each site.

  • Basic usage:

    python3 ia_stats_fetcher.py <input.csv> <output.csv> [--parallel] [--delay SECONDS] [--workers N]

    Example:

    python3 ia_stats_fetcher.py urls.csv ia_stats.csv --parallel --delay 1 --workers 5

  • Features:

    • Queries the Internet Archive CDX API for URL captures
    • Counts captures by MIME type (text/html, image/jpeg, image/png)
    • Tracks unique URLs captured for each site
    • Supports sequential or parallel processing with configurable delays and worker threads
    • Outputs results to CSV format

  • Notes:

    • Requires pandas and requests packages. Install with pip install pandas requests.
    • Input CSV should have URLs in the first column.
    • Disclaimer: This script has known timeout issues and may be buggy. The CDX API requests can timeout intermittently, especially with large batches of URLs, when processing multiple sites in parallel, or with very large sites.

• wayback_solrwayback_query_gen.js

  • Description: Tampermonkey/UserScript that adds a draggable SolrWayback helper panel on Internet Archive pages. It extracts timestamp and original URL from Wayback playback URLs and provides quick actions for SolrWayback queries.

  • Installation:

    1. Install a browser extension that supports UserScripts (e.g., Tampermonkey for Chrome, Firefox, Edge, or Safari).
    2. Copy the contents of wayback_solrwayback_query_gen.js into a new script in the extension.
    3. The script will automatically run on any Internet Archive Wayback Machine URL matching https://web.archive.org/web/*.

  • How it works:

    • Navigate to a Wayback playback URL (example: https://web.archive.org/web/20231015120000/https://example.com/)
    • A floating panel named SolrWayback appears in the bottom-right corner
    • Available actions include:
      • Open Exact: opens url + crawl_date query in SolrWayback
      • Open Nearest: opens url query in SolrWayback
      • Copy Query: copies the exact query string to clipboard
      • Open Domain: opens a domain-level SolrWayback query
    • The panel can be dragged by its header

  • Prerequisites:

    • Critical: You must have a SolrWayback instance bootstrapped and running for this script to work.
    • SolrWayback can run:
      • Locally on your machine (default: http://localhost:8080/solrwayback/)
      • On a remote server (requires port forwarding or SSH tunnel to the server)

  • Setup instructions:

    Custom Configuration

    • To change the SolrWayback base URL or port, edit the SOLRWAYBACK_BASE variable in the script:

      const SOLRWAYBACK_BASE = "http://localhost:8080/solrwayback/search?query=";

• solrwayback-playback_to_wayback_link.js

  • Description: Tampermonkey/UserScript that adds an Open in Wayback button when you view local SolrWayback playback pages. It converts the SolrWayback playback URL into the equivalent Internet Archive playback URL and opens it in a new tab.

  • Installation:

    1. Install a browser extension that supports UserScripts (e.g., Tampermonkey). Free for most browsers.
    2. Copy the contents of solrwayback-playback_to_wayback_link.js into a new script in the extension.
    3. The script runs on local playback paths matching:
       • http://localhost:8080/solrwayback/services/web/*
       • https://localhost:8080/solrwayback/services/web/*
       • http://127.0.0.1:8080/solrwayback/services/web/*
       • https://127.0.0.1:8080/solrwayback/services/web/*

  • How it works:

    • Open a local SolrWayback playback URL, for example:
      • http://localhost:8080/solrwayback/services/web/20021220002707//http://naver.atriax.dk:80/
    • The script extracts:
      • the 14-digit capture timestamp
      • the original URL after the timestamp
    • It builds the matching Internet Archive URL:
      • https://web.archive.org/web/<timestamp>/<original-url>
    • Click Open in Wayback to open that URL in a new browser tab.

• create_build_ia-harvester_virtual_pyenv.sh

  • Description: Helper shell script to create a pyenv virtualenv, clone the local development forks of InternetArchiveExtractor and python-wayback-machine-downloader, build/install the wayback package locally, patch the InternetArchiveExtractor requirements to point at the local wayback package, and install dependencies.

  • Basic usage:

    # Make executable (if necessary) and run from repository root
    chmod +x create_build_ia-harvester_virtual_pyenv.sh
    ./create_build_ia-harvester_virtual_pyenv.sh

    The script will create a pyenv virtualenv named IA-Extractor_<folder> (where <folder> is the current directory name), clone the two repos into the current directory, install the local wayback package, and install requirements for InternetArchiveExtractor.

  • Features / notes:

    • Uses pyenv + pyenv-virtualenv to create and activate a named virtual environment.
    • Clones InternetArchiveExtractor and python-wayback-machine-downloader from the project's GitHub forks.
    • Installs python-wayback-machine-downloader locally (pip install .) and updates InternetArchiveExtractor/requirements.txt to reference the local pywaybackup package via a file:// URL. A backup of requirements.txt is created as requirements.txt.bak before modification.
    • Installs InternetArchiveExtractor dependencies into the created virtualenv.
    • Exits on errors (set -euo pipefail) and prints progress messages.

  • Requirements / prerequisites:

    • pyenv and pyenv-virtualenv installed and configured in your shell.
    • git and pip available in PATH.
    • Ability to build/compile Python (system build tools / headers may be required for some Python versions).
    • The script uses PYTHON_VERSION="3.13.7" by default — change the script if a different interpreter is needed.

• multible_catagory_matcher.sql

  • Description: SQLite SQL script for matching and categorizing URLs across multiple category labels in the corpus database. Intended to be run interactively inside a SQLite GUI (for example DB Browser for SQLite) against the corpus-120b-wayback_url.db schema.

  • Basic usage:

    1. Open DB Browser for SQLite.
    2. Open the database file corpus-120b-wayback_url.db (see Requirements below for location).
    3. Switch to the "Execute SQL" tab and run the contents of multible_catagory_matcher.sql.

    Example (run inside DB Browser for SQLite):

    -- Paste the SQL file contents into the Execute SQL tab and run

  • Notes / Requirements for running:

    • Runs in DB Browser for SQLite (or any SQLite client that can execute multi-statement SQL scripts).
    • Built to work with the structure of the corpus-120b-wayback_url.db database. Do not run on different schemas without adapting the queries.
    • The corpus-120b-wayback_url.db can be retrieved from OneDrive under /Data/Databases/ in the project's storage.
    • Always backup the database before running scripts that modify data.

Requirements

• Python 3.7 or newer
• The warcio package (install with pip install warcio)
• DB Browser for SQLite (or any SQLite client) — required to run multible_catagory_matcher.sql. The corpus-120b-wayback_url.db used by the script can be retrieved from OneDrive under /Data/Databases/.

It's recommended to run the scripts inside a virtual environment:

python3 -m venv .venv
source .venv/bin/activate
pip install warcio

Contributing

Add new scripts at the repository root and update this README with a short description and usage example for each new file.