Created nhs-crawler

.github/workflows/nhs-crawler.yml

@@ -0,0 +1,35 @@
+name: nhs-crawler
+
+on:
+  pull_request:
+    paths:
+      - "nhs-crawler/**"
+      - ".github/workflows/nhs-crawler.yml"
+
+jobs:
+  nhs-crawler:
+    name: nhs-crawler
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: nhs-crawler
+    steps:
+      - uses: actions/checkout@v7.0.0
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v8.2.0
+
+      - name: Set up Python
+        run: uv python install 3.11
+
+      - name: Install dependencies
+        run: uv sync --extra dev
+
+      - name: Lint (ruff)
+        run: uv run ruff check .
+
+      - name: Format check (ruff)
+        run: uv run ruff format --check .
+
+      - name: Type check (pyright)
+        run: uv run pyright

nhs-crawler/README.md

@@ -0,0 +1,164 @@
+# NHS Conditions Scraper
+
+A Python scraper for all [NHS Health A to Z condition
+pages](https://www.nhs.uk/health-a-to-z/conditions/). It discovers every
+condition listed on the index (198 conditions), crawls all subpages for
+each condition (254 pages total), and saves the content as clean
+Markdown to `data/`.
+
+## Prerequisites
+
+- **[uv](https://docs.astral.sh/uv/)** — install with:
+
+  ``` bash
+  curl -LsSf https://astral.sh/uv/install.sh | sh
+  ```
+
+- **Python ≥ 3.11** — uv manages this automatically.
+
+## Quick Start
+
+``` bash
+cd nhs-crawler
+uv sync
+
+# Scrape ALL conditions (198 conditions, 254 pages)
+uv run nhs-crawler
+
+# Scrape a single condition (discovers subpages automatically)
+uv run nhs-crawler --url https://www.nhs.uk/conditions/type-2-diabetes/
+
+# Scrape a single specific page
+uv run nhs-crawler --page https://www.nhs.uk/conditions/type-2-diabetes/treatment/
+
+# Show all options
+uv run nhs-crawler --help
+```
+
+## Output
+
+Markdown files are saved to `data/<condition-slug>/<subpage>.md`:
+
+    data/
+    ├── asthma/
+    │   └── index.md              # Inline hub page — no subpages
+    ├── type-2-diabetes/
+    │   ├── what-is-type-2-diabetes.md
+    │   ├── symptoms.md
+    │   ├── treatment.md
+    │   ├── complications.md
+    │   └── support.md
+    ├── covid-19/
+    │   ├── covid-19-symptoms-and-what-to-do.md
+    │   ├── how-to-avoid-catching-and-spreading-covid-19.md
+    │   └── treatments-for-covid-19.md
+    └── ...                       # 198 conditions, 254 Markdown files
+
+Each file starts with a title and source URL header:
+
+``` markdown
+# Asthma
+
+> Source: https://www.nhs.uk/conditions/asthma/
+
+Asthma is a common condition that affects your breathing...
+```
+
+### Reports
+
+After a full crawl, two files are written to `reports/`:
+
+- **`condition_urls.json`** — ordered list of all discovered condition
+  hub URLs.
+- **`summary.json`** — aggregate statistics for the run:
+
+``` json
+{
+  "run_at": "2026-06-20T18:38:08.937007+00:00",
+  "index_url": "https://www.nhs.uk/health-a-to-z/conditions/",
+  "conditions_discovered": 198,
+  "pages_discovered": 254,
+  "pages_scraped": 254,
+  "error_count": 0,
+  "total_markdown_chars": 1136966,
+  "empty_pages": [],
+  "errors": []
+}
+```
+
+### Logs
+
+A detailed crawl log is appended to `logs/crawl.log` on every run.
+
+> **Note:** The `data/`, `logs/`, and `reports/` directories are created
+> automatically at runtime by `src/constants.py`. They do not need to
+> exist beforehand. Consider adding `logs/` and `reports/` to
+> `.gitignore` if you don’t want to track generated output.
+
+## How It Works
+
+1.  **Discover conditions** — fetches the A-to-Z index page and extracts
+    all `/conditions/<slug>/` links using a regex filter.
+2.  **Discover subpages** — for each condition hub page, looks for a
+    `<ul class="nhsuk-hub-key-links">` navigation element to find
+    subpages. Some conditions (like asthma) are single-page with inline
+    content — the hub page itself is saved as `index.md`. Others (like
+    type-2-diabetes) have dedicated subpages for Symptoms, Treatment,
+    etc. — in this case only the subpages are saved (the hub is excluded
+    since the subpages contain all the content).
+3.  **Fetch & convert** — each page is fetched and its main content
+    element (`<div class="nhsuk-grid-column-two-thirds">` inside
+    `<article>`) is converted to clean Markdown using a custom NHS-aware
+    converter that handles:
+    - **Care cards** → blockquotes with emoji prefixes (🚨 ⚠️ 📋)
+    - **Do/don’t lists** → bullet lists with ✅/❌ markers
+    - **Inset text** → blockquotes with 💡 prefix
+    - **Clutter removal** — nav, breadcrumbs, feedback banners, SVGs,
+      etc.
+4.  **Save** — Markdown is written to `data/<condition>/<subpage>.md`
+    with a title and source URL header.
+5.  **Report** — `reports/summary.json` with aggregate statistics and
+    `reports/condition_urls.json` with the full condition URL list.
+
+### HTTP Strategy
+
+All requests go through a shared `requests.Session` with:
+
+- **Retry**: up to 4 retries with exponential backoff (factor 1.2) on
+  HTTP 429, 500, 502, 503, 504.
+- **Rate limiting**: 0.5s delay before each request (politeness).
+- **Connection pooling**: 10 connections, 20 max pool size.
+- **User-Agent**: identifies the scraper with a reference to nhs.uk.
+- **Concurrency**: pages are fetched in parallel using a thread pool (4
+  workers by default).
+
+## CLI Reference
+
+    usage: nhs-crawler [-h] [--url URL] [--page PAGE]
+
+    NHS Conditions scraper
+
+    options:
+      -h, --help   show this help message and exit
+      --url URL    Scrape a single condition URL (including subpages).
+                   If omitted, scrapes all conditions from the A-to-Z index.
+      --page PAGE  Scrape a single specific page URL (no subpage discovery).
+
+## Configuration
+
+Key settings in `src/constants.py`:
+
+| Constant      | Default | Description                           |
+|---------------|---------|---------------------------------------|
+| `DELAY`       | `0.5`   | Seconds between requests (politeness) |
+| `TIMEOUT`     | `30`    | HTTP request timeout (seconds)        |
+| `MAX_WORKERS` | `4`     | Parallel threads for page fetching    |
+
+## Development
+
+``` bash
+uv sync --all-extras
+uv run ruff format src/
+uv run ruff check src/
+uv run pyright src/
+```

nhs-crawler/data/acute-pancreatitis/index.md

@@ -0,0 +1,103 @@
+# Acute pancreatitis
+
+> Source: https://www.nhs.uk/conditions/acute-pancreatitis/
+
+![Diagram of the stomach area with labels showing the liver, stomach, gallbladder and pancreas. The pancreas is highlighted just below the stomach.](https://assets.nhs.uk/nhsuk-cms/images/Pancreatic_Cancer_NEW_copy.width-320.jpg)
+
+The pancreas is an organ in the middle of your tummy. It helps you digest food and makes hormones such as insulin.
+
+> **💡 Information:** Acute pancreatitis is different to chronic pancreatitis , a long-term condition where the pancreas has become permanently damaged.
+
+## Symptoms of acute pancreatitis
+
+The main symptom of acute pancreatitis is pain in your tummy (abdomen). You may also have a high temperature and feel sick or be sick (nausea and vomiting).
+
+Tummy pain may:
+
+- start suddenly and not go away
+- be severe, sharp or knife-like
+- affect the upper part of your tummy (between your ribs), one side of your tummy or your whole tummy
+- spread to your sides and back
+- feel worse after you eat, move around or lie down
+- feel better when you lean forward or bring your knees to your chest (fetal position)
+
+> **💡 Information:** Acute pancreatitis symptoms can be similar to other conditions such as appendicitis or stomach ulcer . Find out about what else can cause stomach ache.
+
+> **⚠️ Urgent advice:**
+>
+> You get sudden, severe pain in your tummy and:
+>
+> - it does not go away or keeps coming back
+> - you have a high temperature, or you feel hot, cold or shivery
+>
+> You can call 111 or [get help from 111 online](https://111.nhs.uk/triage/check-your-symptoms).
+>
+> If a GP thinks you have acute pancreatitis they will refer you to hospital for tests straight away.
+
+> **🚨 Immediate action required:**
+>
+> You get sudden, severe pain in your tummy and:
+>
+> - the pain is spreading to your back
+> - you have bloating that does not go away or keeps coming back
+> - you have a fast heartbeat or difficulty breathing
+> - the skin around your belly button, waist or upper outer thigh appears blue or bruised – this may be more difficult to see on black or brown skin
+>
+> [Find your nearest A&E](https://www.nhs.uk/service-search/find-an-accident-and-emergency-service/)
+
+> **💡 Information:** Do not drive to A&E. Ask someone to drive you or call 999 and ask for an ambulance. Bring any medicines you take with you.
+
+## Treatment for acute pancreatitis
+
+Acute pancreatitis is usually diagnosed using blood tests and sometimes a CT scan. It's a serious condition that needs treatment in hospital straight away.
+
+You'll be monitored to see how serious your condition is and if it's causing any other problems, such as an infection.
+
+Hospital treatment may include:
+
+- fluids and nutrients – given through a tube into a vein
+- painkillers
+- antibiotics – if you have an infection
+
+You may also need treatment for what is causing your acute pancreatitis, such as surgery for gallstones or support to stop drinking alcohol.
+
+Most people with acute pancreatitis start to get better within a week and can leave hospital in 5 to 10 days.
+
+If you have severe pancreatitis or it's causing other problems, you may need to stay in hospital for longer.
+
+## Problems caused by acute pancreatitis
+
+Most people with acute pancreatitis recover fully. But some people develop serious complications that will need treatment.
+
+Complications of acute pancreatitis include:
+
+- small growths in your pancreas (cysts) – these often go away on their own but may need to be removed if they become infected
+- pancreatic necrosis – where some of the tissue in the pancreas dies and you need surgery and antibiotics to prevent a serious condition called [sepsis](/conditions/sepsis/)
+- [chronic pancreatitis](/conditions/chronic-pancreatitis/) – if you keep getting acute pancreatitis it can develop into a serious long-term condition
+
+Acute pancreatitis can be life-threatening. You'll be monitored while you're in hospital, to check for any problems caused by acute pancreatitis.
+
+## Causes of acute pancreatitis
+
+The most common causes of acute pancreatitis are:
+
+- [gallstones](/conditions/gallstones/) – which can block the opening of the pancreas
+- drinking a lot of alcohol
+
+Less common causes include:
+
+- injury to the pancreas, such as during surgery
+- medicines, including certain steroids, heart and epilepsy medicines
+- other conditions including [lupus](/conditions/lupus/), [mumps](/conditions/mumps/), [pancreatic cancer](/conditions/pancreatic-cancer/) or having high levels of calcium in your blood (hypercalcaemia)
+
+## How to prevent acute pancreatitis
+
+If you have had acute pancreatitis once, it's possible to get it again.
+
+There are some things you can do to help stop it coming back.
+
+It's a good idea to:
+
+- drink less alcohol, or not drink any alcohol at all
+- stop smoking
+- eat a healthy, low-fat diet