ny-gov-web-archiver

🚧 Work in progress. Under active development; interfaces and defaults may change. Issues and contributions welcome.

A throttled Internet Archive (Wayback Machine) harvester for NYC / NYS government web properties. Given a government entity's domains, it enumerates that domain's historical Wayback captures over time and — only when explicitly told to — downloads them to a local cache, writing a provenance manifest either way.

This is the middle layer of a three-layer architecture:

ny-gov-web-registry   →   ny-gov-web-archiver   →   downstream applications
(what exists: entities     (this repo: given domains,     (e.g. the NYC mayoral
 & their domains)           enumerate/fetch their          executive-orders corpus)
                            Wayback captures)

It is the general tool. It knows nothing about executive orders or any specific document type — it harvests captures for whatever domains it's given. Application-specific URL patterns (e.g. the old nyc.gov/html/records/... EO PDF paths, or a mayor's-office listing API) belong to the downstream application, not here.

Built on EDGI wayback

The Wayback access is EDGI's wayback library (docs: https://wayback.readthedocs.io/). We deliberately do not reimplement CDX querying, retry/backoff, or 429 handling — the library already does all of that. This repo is a thin, registry-fed orchestrator on top of it:

• wayback_client.py — builds a WaybackClient configured for a conservative go-slow rate (see "Rate posture").
• harvest.py — enumerates captures per domain via WaybackClient.search() (CDX), applies MIME / status / year filters, and optionally fetches via WaybackClient.get_memento(). Also the CLI.
• manifest.py — writes manifest.csv + manifest.jsonl (one row per capture) and a gaps.md errors/empties report.
• registry_input.py — loads the sibling ny-gov-web-registry and yields (betanyc_id, domain, role) targets.

Rate posture (read before running anything live)

Internet Archive is a nonprofit on constrained infrastructure and is consolidating its endpoints behind a shared ~30 requests/minute budget (EDGI issue #205). BetaNYC's standing etiquette lives in the workspace at team/reference/external-data-source-access.md. This tool encodes it:

• Go slow. config.SEARCH_CALLS_PER_SECOND = 0.40 (1 CDX query / 2.5s) and config.MEMENTO_CALLS_PER_SECOND = 0.33 (1 download / ~3s). Both sit well under the shared 30/min ceiling.
• One CDX query per domain, then throttle downloads; the library backs off and honors Retry-After on a 429.
• Cache locally, never re-fetch. A capture already on disk is not re-downloaded (idempotent re-runs).
• Descriptive User-Agent identifying BetaNYC + this project.

No live pulls in CI or from agents

The entire test suite is offline — it mocks the wayback client and asserts that a dry-run issues zero fetch calls. A guard fixture makes any accidental real call fail the test. No agent and no CI job hits the live Internet Archive.

Live access happens in exactly one place: scripts/smoke_test_live.py, a supervised, human-run script (see below). It refuses to run without an explicit acknowledgement flag.

Install

# From the repo root:
uv pip install -e .            # or: pip install -e .

Requires Python >=3.11,<3.15. The only runtime dependency is wayback.

Usage

Dry-run (enumerate only — the default, never fetches):

python -m ny_gov_web_archiver.harvest --domain mta.info --from 2010 --to 2012 --dry-run

Registry-wide dry-run (every domain in ny-gov-web-registry):

python -m ny_gov_web_archiver.harvest --registry-all --from 2010 --to 2012 --dry-run

Filter by MIME type and status, and actually download (explicit --fetch):

python -m ny_gov_web_archiver.harvest \
  --domain nycedc.com --from 2008 --to 2014 \
  --mime application/pdf --status 200 \
  --fetch

Key flags:

Flag	Default	Meaning
--domain D (repeatable) / --registry-all	— (one required)	Targets: ad-hoc domains, or the whole registry
--registry PATH	../ny-gov-web-registry/data/registry.json	Registry file for --registry-all
--from / --to YEAR	none	Inclusive capture-year window
--match-type	host	CDX granularity: host (safe, exact host), domain, prefix, exact
--mime M (repeatable)	none	Keep only these MIME types
--status C (repeatable)	none	Keep only these HTTP status codes
--collapse	digest	CDX collapse field (dedupe identical content)
--limit	5000	Max CDX rows per domain
--fetch / --dry-run	dry-run	--fetch downloads; dry-run never fetches
--cache-dir / --out-dir	cache/ / out/	Local fetched content / manifest output

--match-type host is the default on purpose. A registry entity like OTI or the Mayor's Office lives at a subpath of the shared nyc.gov host; a domain match on nyc.gov would try to enumerate all of nyc.gov. Opt into the wider net explicitly.

A domain shared by several entities (e.g. nyc.gov) is enumerated once per entity, each recording under its own betanyc_id.

Output

Written to --out-dir (default out/):

• manifest.csv / manifest.jsonl — one row per capture: betanyc_id, role, domain, original_url, wayback_timestamp, digest, mime_type, status_code, wayback_url, local_path. local_path is populated only when fetched.
• gaps.md — domains that returned zero captures, plus any per-capture fetch errors.

Fetched content caches under --cache-dir (default cache/), git-ignored and re-fetchable — the manifest, not the blobs, is the durable artifact.

Supervised live smoke test (human only)

scripts/smoke_test_live.py is the only code that touches the live Internet Archive. It is not run by CI, agents, or the test suite. Run it as a human, under BetaNYC's go-slow authorization:

python scripts/smoke_test_live.py \
  --domain mta.info --from 2012 --to 2012 \
  --i-am-a-human-running-this-supervised

It enumerates one domain over a tiny window and prints the first few captures. Without the acknowledgement flag it refuses to run.

Out of scope for v1 (OCR deferred)

v1 is fetch + manifest only. It does not OCR PDFs or extract text. Turning harvested PDFs into searchable text is a downstream (EO-corpus) concern — it belongs to the application consuming this harvester's output, not to the general archiver. The manifest records mime_type so a downstream OCR/parse stage can select what it needs.

Testing

uv run --with pytest --with wayback python -m pytest

All tests run offline — no network, no live Internet Archive traffic.

Related

• Input: ny-gov-web-registry — the entity/domain registry.
• Client library: wayback (EDGI).
• First downstream consumer: nyc-executive-orders.

Part of BetaNYC's civic data tools

This harvester is one layer of a family of free, open civic data assets BetaNYC builds and stewards:

• ny-gov-web-registry — which NY government entities exist, and where they live on the web (this tool's input).
• nyc-executive-orders — the open archive of NYC mayoral executive orders, 1974–present (this tool's first downstream application).
• nyc-boundaries — NYC administrative boundaries, mapped and queryable.
• Seven MCP servers giving AI agents direct access to NYC/NYS civic data: Council legislation, City Record, Checkbook spending, 311, Charter/Code/Rules, NYS legislation, and the Council budget.

AI use in this project

BetaNYC uses AI tools openly and with human accountability. This harvester was developed with AI assistance (Anthropic's Claude) under the direction and review of BetaNYC staff — the specification and rate posture were human-set, and live harvests are an explicit, supervised, human-run step (see "Supervised live smoke test").

The tool does not generate content: it downloads historical Wayback captures verbatim as retrieved from the Internet Archive, writing a provenance manifest either way. All harvesting is throttled, identifies itself with a contact address, and honors the Internet Archive's access guidelines rather than circumventing them.

Questions about our approach: hello@beta.nyc.

License

Two licenses, split by what the file is:

• Code (src/, scripts/, tests/): MIT © 2026 BetaNYC — use and modify freely with attribution preserved.
• Documentation (README.md, CONTRIBUTING.md): Creative Commons Attribution-ShareAlike 4.0 — reuse and adapt with credit to BetaNYC, and share adaptations under the same license.

Downloaded captures remain the property of their original publishers and the Internet Archive; this repository's licenses cover the tool and its documentation, not the harvested material.