diff --git a/.github/workflows/nhs-crawler.yml b/.github/workflows/nhs-crawler.yml new file mode 100644 index 0000000..28af45b --- /dev/null +++ b/.github/workflows/nhs-crawler.yml @@ -0,0 +1,35 @@ +name: nhs-crawler + +on: + pull_request: + paths: + - "nhs-crawler/**" + - ".github/workflows/nhs-crawler.yml" + +jobs: + nhs-crawler: + name: nhs-crawler + runs-on: ubuntu-latest + defaults: + run: + working-directory: nhs-crawler + steps: + - uses: actions/checkout@v7.0.0 + + - name: Install uv + uses: astral-sh/setup-uv@v8.2.0 + + - name: Set up Python + run: uv python install 3.11 + + - name: Install dependencies + run: uv sync --extra dev + + - name: Lint (ruff) + run: uv run ruff check . + + - name: Format check (ruff) + run: uv run ruff format --check . + + - name: Type check (pyright) + run: uv run pyright diff --git a/nhs-crawler/README.md b/nhs-crawler/README.md new file mode 100644 index 0000000..f44ecaf --- /dev/null +++ b/nhs-crawler/README.md @@ -0,0 +1,164 @@ +# NHS Conditions Scraper + +A Python scraper for all [NHS Health A to Z condition +pages](https://www.nhs.uk/health-a-to-z/conditions/). It discovers every +condition listed on the index (198 conditions), crawls all subpages for +each condition (254 pages total), and saves the content as clean +Markdown to `data/`. + +## Prerequisites + +- **[uv](https://docs.astral.sh/uv/)** — install with: + + ``` bash + curl -LsSf https://astral.sh/uv/install.sh | sh + ``` + +- **Python ≥ 3.11** — uv manages this automatically. + +## Quick Start + +``` bash +cd nhs-crawler +uv sync + +# Scrape ALL conditions (198 conditions, 254 pages) +uv run nhs-crawler + +# Scrape a single condition (discovers subpages automatically) +uv run nhs-crawler --url https://www.nhs.uk/conditions/type-2-diabetes/ + +# Scrape a single specific page +uv run nhs-crawler --page https://www.nhs.uk/conditions/type-2-diabetes/treatment/ + +# Show all options +uv run nhs-crawler --help +``` + +## Output + +Markdown files are saved to `data//.md`: + + data/ + ├── asthma/ + │ └── index.md # Inline hub page — no subpages + ├── type-2-diabetes/ + │ ├── what-is-type-2-diabetes.md + │ ├── symptoms.md + │ ├── treatment.md + │ ├── complications.md + │ └── support.md + ├── covid-19/ + │ ├── covid-19-symptoms-and-what-to-do.md + │ ├── how-to-avoid-catching-and-spreading-covid-19.md + │ └── treatments-for-covid-19.md + └── ... # 198 conditions, 254 Markdown files + +Each file starts with a title and source URL header: + +``` markdown +# Asthma + +> Source: https://www.nhs.uk/conditions/asthma/ + +Asthma is a common condition that affects your breathing... +``` + +### Reports + +After a full crawl, two files are written to `reports/`: + +- **`condition_urls.json`** — ordered list of all discovered condition + hub URLs. +- **`summary.json`** — aggregate statistics for the run: + +``` json +{ + "run_at": "2026-06-20T18:38:08.937007+00:00", + "index_url": "https://www.nhs.uk/health-a-to-z/conditions/", + "conditions_discovered": 198, + "pages_discovered": 254, + "pages_scraped": 254, + "error_count": 0, + "total_markdown_chars": 1136966, + "empty_pages": [], + "errors": [] +} +``` + +### Logs + +A detailed crawl log is appended to `logs/crawl.log` on every run. + +> **Note:** The `data/`, `logs/`, and `reports/` directories are created +> automatically at runtime by `src/constants.py`. They do not need to +> exist beforehand. Consider adding `logs/` and `reports/` to +> `.gitignore` if you don’t want to track generated output. + +## How It Works + +1. **Discover conditions** — fetches the A-to-Z index page and extracts + all `/conditions//` links using a regex filter. +2. **Discover subpages** — for each condition hub page, looks for a + `