WebsitesScraperAgent

A WAT-framework agent for scraping hospitality company websites into structured contact info + company profiles + raw page text, delivered as local JSON/CSV.

Layout

workflows/   # Markdown SOPs (what to do + how)
tools/       # Python execution scripts
.tmp/        # Disposable intermediates (.tmp/raw/<slug>.json per site)
.env         # Secrets (gitignored) — copy from .env.example
.venv/       # Local Python env (gitignored)
target-company-websites.json   # Input: ~180 {name, website} records

Setup

python3 -m venv .venv
./.venv/bin/pip install -r requirements.txt
cp .env.example .env        # then add your FIRECRAWL_API_KEY

Run the scrape workflow

See workflows/scrape_company_data.md. In short:

# 1. Test on ONE site first (Firecrawl is paid per extract)
./.venv/bin/python tools/scrape_single_site.py --url https://www.sagehospitalitygroup.com/ --name "Sage Hospitality Group"

# 2. After sign-off, batch-run all companies (resumable), then aggregate
./.venv/bin/python tools/run_all.py                 # --concurrency / --limit / --retry-failed
./.venv/bin/python tools/aggregate_results.py       # -> company-data.json + company-data.csv

How it works

1. scrape_single_site.py — submits the whole site (site.com/*) to Firecrawl /extract, which crawls the site and returns one merged schema-structured record (contacts + profile) plus per-field sources; saved to .tmp/raw/<slug>.json.
2. run_all.py — resumable batch runner over all companies with a small concurrency pool; skips null sites and already-completed companies.
3. aggregate_results.py — merges all raw results into company-data.json (full) and company-data.csv (flattened for spreadsheets).