ArcGIS Parcel Harvester

A resumable bulk downloader for public ArcGIS REST servers, built to assemble farmland parcel research datasets for any US state.

Point it at a state and it walks that state's GIS portal plus the national USDA NRCS and BLM servers, classifies every feature layer it finds into one of twelve farmland data categories, and downloads the relevant ones as GeoPackages in EPSG:4326. A full state harvest is thousands of requests over several hours, so the tool is built to survive that: every write is atomic, progress is committed after each layer, and re-running the same command picks up exactly where it stopped.

Built during a summer internship researching agricultural land in South Dakota, then generalised to all 50 states.

Quick start

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -e .

arcgis-harvest states                          # what is configured
arcgis-harvest harvest --state SD --dry-run    # plan without downloading
arcgis-harvest harvest --state SD              # confirm, then download

Nothing is downloaded until you confirm. --dry-run runs the identical enumeration, filtering, and file-naming logic as a real run and prints exactly what would be written.

Commands

Command	Purpose
arcgis-harvest states [STATE]	List configured states, or show one state's servers and bounding box
arcgis-harvest probe URL	Enumerate an unfamiliar ArcGIS server and summarise what it holds
arcgis-harvest harvest --state SD	Download a state's layers
arcgis-harvest merge --state SD	Combine one category's GeoPackages into a single file

Examples

# Plan a run and see the per-category layer counts
arcgis-harvest harvest --state SD --dry-run

# Only the layers that matter for a title search
arcgis-harvest harvest --state SD --categories parcels plss boundaries

# Unattended, gentler on the server, more parallelism per layer
arcgis-harvest harvest --state "North Dakota" --yes --rate-limit 1.0 --workers 8

# Inspect a county server before adding it to the config
arcgis-harvest probe https://gis.example.gov/arcgis/rest/services

# Merge every parcel file into one statewide layer
arcgis-harvest merge --state SD --category parcels

Run arcgis-harvest <command> --help for the full flag list.

Data categories

Layers are classified by name into twelve categories, aligned to the four sections of a standard farmland parcel record.

Category	Contents
parcels	Parcel boundaries, ownership records, tax districts and rates
plss	Public Land Survey System grid: sections, townships, quarter-quarters
boundaries	County, state, city, zip code, administrative boundaries
census	Census tracts, block groups, demographic areas
soils	USDA soil surveys (SSURGO, STATSGO, gSSURGO)
agricultural	Cropland, grazing allotments, farm tracts, irrigation
conservation	Conservation easements, districts, reserve programmes
landcover	Land use and cover classification, federal surface management
flood	FEMA flood hazard zones
hydro	Wetlands (NWI), rivers, streams, lakes, watersheds
schools	School district and special tax district boundaries
tribal	Tribal lands and reservation boundaries

Layers that match nothing are classified other, skipped, and deliberately not recorded as complete, so improving the vocabulary and re-running picks them up without a full re-harvest.

Output

data/<STATE>/
  parcels/
    <service_slug>_<layer_id>_<layer_name>.gpkg
  plss/
  boundaries/
  ...
  progress.json

Everything is written as GeoPackage in EPSG:4326, so layers from different sources join without reprojection. progress.json records each completed layer with its feature count and timestamp, which doubles as an audit trail of what a run actually produced.

How it works

Four decisions account for most of the design.

Object-ID batching rather than offset paging. The ArcGIS query API caps features per response. Offset paging (resultOffset) is simpler, but many older deployments silently ignore it and return the first page for every request: no error, wrong data. Fetching the object-ID list once and then requesting explicit ID ranges works on every ArcGIS version, is order-independent, and makes each request individually retryable. Batch size is clamped to the server's advertised maxRecordCount, because exceeding it truncates the response instead of failing.

ArcGIS errors are detected, not assumed away. The API reports application errors in a JSON body returned with HTTP 200. Code that checks only the status code reads {"error": {"code": 499, "message": "Token Required"}} as an empty result set and records the layer as done. Every response is inspected.

Atomic writes everywhere. GeoPackages and the progress file are written to a temporary path and renamed into place. Rename is atomic within a filesystem, so an interrupted download can never leave a truncated file that the next run would mistake for finished work. This is the difference between resumable and merely restartable.

One shared, rate-limited HTTP client. Public GIS servers are run by agencies on modest budgets. A single client enforces a minimum interval between requests to each host, retries only what is worth retrying (timeouts, connection resets, 429, 5xx) with exponential backoff and jitter, honours Retry-After, and identifies itself honestly rather than impersonating a browser. Batches within a layer are fetched concurrently; the per-host rate limit still governs the real request rate, so concurrency buys latency without costing politeness.

Override the User-Agent to include your own contact address:

export ARCGIS_HARVESTER_USER_AGENT="my-research/1.0 (+me@university.edu)"

Adding a state

The national NRCS and BLM servers are queried for every state, clipped to its bounding box, so any state works out of the box. To also pull from a state's own GIS portal, find it, check it, then add it.

arcgis-harvest probe https://txgis.example.gov/arcgis/rest/services

If the probe shows layers worth having, add the server to src/arcgis_harvester/data/states.json:

"TX": {
  "name": "Texas",
  "abbrev": "TX",
  "bbox": [-106.65, 25.84, -93.51, 36.50],
  "state_servers": [
    {
      "crawl_root": "https://txgis.example.gov/arcgis/rest/services",
      "server_base": null,
      "label": "TX-GIS",
      "description": "Texas GIS Portal",
      "bbox_filter": false
    }
  ]
}

server_base is only needed when crawl_root points at a sub-folder. ArcGIS folder listings name their services relative to the server root, so without it the built service URLs come out with a doubled path. bbox_filter should stay false for a state portal, which already holds only that state's data.

The config is validated on load: bounding boxes are checked for ordering, range, and transposed coordinates, and server_base must be an ancestor of crawl_root. A typo fails at startup rather than after an hour of downloading the wrong extent.

Development

pip install -e ".[dev]"
pytest                       # 231 tests
pytest --cov                 # with coverage
ruff check . && ruff format --check .

The test suite never opens a socket. The HTTP client is tested through intercepted requests traffic, and everything above it against canned ArcGIS payloads, so the suite is fast and does not depend on whether a state GIS server happens to be up. CI runs lint, tests on Python 3.11 through 3.13, and a packaging check that the bundled states.json actually ships inside the wheel.

src/arcgis_harvester/
  client.py       shared HTTP session: rate limiting, retries, error detection
  config.py       states.json loading, validation, state resolution
  catalog.py      recursive server enumeration, layer listing
  categorize.py   layer name to data category
  download.py     object-ID paging for a single layer
  harvest.py      planning, resume state, atomic writes, the download loop
  merge.py        combining harvested files
  cli.py          argument parsing and the four subcommands

Data sources and limitations

Server	Coverage	Status
SD GIS (sdgis.sd.gov/arcgis1)	South Dakota state layers	Working
SD Game, Fish and Parks (ert.gfp.sd.gov)	Wildlife and environmental	Working
SD Hosted (sdgis.sd.gov/host)	DOR tax data, PLSS, boundaries	Working
BLM Cadastral, Lands, Range (gis.blm.gov)	PLSS grid, federal land, grazing	Working
NRCS Soils, Easements, Land Use (nrcsgeoservices.sc.egov.usda.gov)	Soil surveys, easements	Blocking automated requests

The NRCS server resets the connection for automated clients regardless of the headers sent. The harvester reports this as an unreachable server and continues with the rest rather than failing the run. Use the Web Soil Survey bulk download or the gSSURGO state files for soils data in the meantime.

South Dakota does not publish a statewide parcel polygon layer, and owner, assessed value, and APN fields live with the 66 individual county assessors rather than in any state GIS system. docs/south-dakota-data-status.md covers what the harvest actually obtained, what is missing, and what each remaining field would cost to acquire.

Only public, unauthenticated endpoints are queried. Check the terms of service for any server you add, and keep the rate limit conservative.

License

MIT. See LICENSE.