Created healf-crawler

.github/workflows/healf-crawler.yml

@@ -0,0 +1,35 @@
+name: healf-crawler
+
+on:
+  pull_request:
+    paths:
+      - "healf-crawler/**"
+      - ".github/workflows/healf-crawler.yml"
+
+jobs:
+  healf-crawler:
+    name: healf-crawler
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: healf-crawler
+    steps:
+      - uses: actions/checkout@v7.0.0
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v8.2.0
+
+      - name: Set up Python
+        run: uv python install 3.11
+
+      - name: Install dependencies
+        run: uv sync --extra dev
+
+      - name: Lint (ruff)
+        run: uv run ruff check .
+
+      - name: Format check (ruff)
+        run: uv run ruff format --check .
+
+      - name: Type check (pyright)
+        run: uv run pyright

healf-crawler/README.md

@@ -0,0 +1,182 @@
+# Healf Product Crawler
+
+A Python crawler for a limited set of [Healf](https://healf.com)
+products from three brands — **Terranova**, **Life Extension**, and
+**NOW Foods**. It discovers every matching product from the collection
+listing (~175 products), fetches structured data and rendered page
+content, and saves each product as clean Markdown to `data/`.
+
+## Prerequisites
+
+- **[uv](https://docs.astral.sh/uv/)** — install with:
+
+  ``` bash
+  curl -LsSf https://astral.sh/uv/install.sh | sh
+  ```
+
+- **Python ≥ 3.11** — uv manages this automatically.
+
+## Quick Start
+
+``` bash
+cd healf-crawler
+uv sync
+
+# Crawl ALL target products (~175 products)
+uv run healf-crawler
+
+# Scrape a single product by handle or URL
+uv run healf-crawler --url terranova-magnesium-complex-50s
+uv run healf-crawler --url https://healf.com/products/terranova-magnesium-complex-50s
+
+# List discovered product handles without scraping
+uv run healf-crawler --list-only
+
+# Show all options
+uv run healf-crawler --help
+```
+
+## Dependencies
+
+| Package          | Purpose                               |
+|------------------|---------------------------------------|
+| `requests`       | HTTP fetching with retry/backoff      |
+| `beautifulsoup4` | HTML parsing and DOM manipulation     |
+| `lxml`           | Fast parser backend for BeautifulSoup |
+| `markdownify`    | HTML → Markdown conversion            |
+
+Dev dependencies (optional, install with `uv sync --all-extras`):
+
+| Package                | Purpose                            |
+|------------------------|------------------------------------|
+| `ruff`                 | Linting and formatting             |
+| `pyright`              | Static type checking (strict mode) |
+| `types-requests`       | Type stubs for requests            |
+| `types-beautifulsoup4` | Type stubs for BeautifulSoup       |
+
+## Output
+
+Markdown files are saved to `data/<product-handle>.md`:
+
+    data/
+    ├── terranova-magnesium-complex-50s.md
+    ├── life-extension-neuro-mag-magnesium-l-threonate.md
+    ├── now-foods-magnesium-glycinate.md
+    └── ...                       # ~175 Markdown files
+
+Each file contains the product name, source URL, brand, price,
+description, ingredients, and suggested use:
+
+``` markdown
+# Magnesium Complex
+
+> Source: https://healf.com/products/terranova-magnesium-complex-50s
+
+**Brand:** Terranova | **Price:** £13.00
+
+## Description
+
+**Key benefits**
+...
+```
+
+### Reports
+
+After a full crawl, two files are written to `reports/`:
+
+- **`product_handles.json`** — all discovered product handles with
+  vendor and product type.
+- **`summary.json`** — aggregate statistics for the run:
+
+``` json
+{
+  "run_at": "2026-06-20T18:38:08.937007+00:00",
+  "target_vendors": ["Life Extension", "NOW Foods", "Terranova"],
+  "products_discovered": 175,
+  "products_by_vendor": {
+    "Terranova": 60,
+    "Life Extension": 75,
+    "NOW Foods": 40
+  },
+  "products_scraped": 175,
+  "error_count": 0,
+  "total_markdown_chars": 980432,
+  "empty_pages": [],
+  "errors": []
+}
+```
+
+### Logs
+
+A detailed crawl log is appended to `logs/crawl.log` on every run.
+
+> **Note:** The `data/`, `logs/`, and `reports/` directories are created
+> automatically at runtime by `src/constants.py`. They do not need to
+> exist beforehand. Consider adding `logs/` and `reports/` to
+> `.gitignore` if you don’t want to track generated output.
+
+## How It Works
+
+1.  **Discover products** — queries the Shopify Storefront API for the
+    `all-products-1` collection, paginating through all results and
+    filtering by vendor (Terranova, Life Extension, NOW Foods).
+2.  **Fetch structured data** — for each product, the Storefront API
+    returns structured fields: title, vendor, price, and
+    `descriptionHtml`.
+3.  **Extract metafields from RSC** — the product detail page embeds
+    private metafields (`ingredients`, `suggested_use`) inside
+    `self.__next_f.push(...)` chunks in the React Server Components
+    payload. These are not exposed via the Storefront API, so they are
+    parsed from the rendered page’s HTML.
+4.  **Convert** — all HTML content fragments (description, ingredients,
+    suggested use) are converted to clean Markdown. Clutter tags
+    (buttons, forms, inputs, SVGs, scripts, styles, iframes) are
+    stripped and blank lines are collapsed.
+5.  **Save** — Markdown is written to `data/<product-handle>.md` with a
+    title, source URL, and structured metadata header.
+6.  **Report** — `reports/summary.json` with aggregate statistics and
+    `reports/product_handles.json` with the full product handle list.
+
+### HTTP Strategy
+
+All requests go through a shared `requests.Session` with:
+
+- **Retry**: up to 4 retries with exponential backoff (factor 1.2) on
+  HTTP 429, 500, 502, 503, 504.
+- **Connection pooling**: 10 connections, 20 max pool size.
+- **User-Agent**: identifies the crawler with a reference to healf.com.
+- **Concurrency**: products are fetched in parallel using a thread pool
+  (4 workers by default).
+
+## CLI Reference
+
+    usage: healf-crawler [-h] [--url URL] [--list-only]
+
+    Healf product crawler
+
+    options:
+      -h, --help     show this help message and exit
+      --url URL      Scrape a single product by its Healf product page URL or
+                     Shopify handle. If omitted, crawls all target products from
+                     the collection listing.
+      --list-only    Only discover and list target product handles, then exit.
+
+## Configuration
+
+Key settings in `src/constants.py`:
+
+| Constant | Default | Description |
+|----|----|----|
+| `MAX_WORKERS` | `4` | Parallel threads for product fetching |
+| `PAGE_SIZE` | `250` | Products per Storefront API page |
+| `TARGET_VENDORS` | `Terranova, Life Extension, NOW Foods` | Vendors to filter from the collection |
+| `START_URL` | `.../collections/all-products-1` | Healf collection page URL |
+
+## Development
+
+``` bash
+uv sync --all-extras
+uv run ruff format src/
+uv run ruff check src/
+uv run pyright src/
+```

healf-crawler/data/life-extension-advanced-curcumin-elite-turmeric-extract-ginger-turmerones.md

@@ -0,0 +1,39 @@
+# Advanced Curcumin Elite™ Turmeric Extract, Ginger & Turmerones
+
+> Source: https://healf.com/products/life-extension-advanced-curcumin-elite-turmeric-extract-ginger-turmerones
+
+**Brand:** Life Extension | **Price:** £20.99
+
+## Description
+
+**Key Benefits**
+
+- Experience a unique blend of turmeric, ginger, and turmerones.
+- Formulated for 45x greater absorption than standard curcumin.
+
+Start your day with a turmeric formula that’s made to go further. Curcumin, the active compound in turmeric, is naturally hard for the body to absorb. Advanced Curcumin Elite™ combines curcumin with fenugreek fibre, making it 45.5 times more bioavailable than standard curcumin—so you get more from every softgel. This carefully crafted blend also features ginger extract and turmerones from turmeric oil, bringing together traditional botanicals in a modern, easy-to-take format.  
+
+Life Extension’s Advanced Curcumin Elite™ uses FenuMAT™ technology, a water-based process that binds curcumin to fenugreek fibre for superior absorption. Responsibly sourced from India, this formula delivers free curcuminoids efficiently—helping you make the most of your daily routine.
+
+## Ingredients
+
+Curcumin Elite™ [Proprietary CGM Blend Providing 40% Curcuminoids (200 mg), 3% Turmerones (15 mg) from Turmeric (Rhizome), 30% Galactomannans (150 mg) from Fenugreek (Seed)], Gingerols [from Ginger CO₂ Extract (Root)], Turmerones [from Turmeric Extract Oil (Rhizome)], Extra Virgin Olive Oil, Capsule Shell (Gelatin), Glycerin, Purified Water, Emulsifier (Sunflower Lecithin), Colour (Carob Colour), Beeswax
+
+## Suggested Use
+
+Take one (1) softgel daily, or as recommended by a healthcare practitioner.  
+
+Warnings   
+ KEEP OUT OF REACH OF CHILDREN   
+DO NOT EXCEED RECOMMENDED DOSE   
+Do not purchase if outer seal is broken or damaged.   
+When using nutritional supplements, please consult with your physician if you are undergoing treatment for a medical condition or if you are pregnant or lactating.  
+
+
+**Additional Information:**
+
+Food supplements and foods sold by Healf should not be used as a substitute for a varied, balanced diet and healthy lifestyle.
+
+If you are pregnant, breastfeeding, have a medical condition, or are taking any medications, please consult with a healthcare professional before use. Use products only if the seal is intact. Store in a cool, dry place, out of the reach of young children. Do not exceed the recommended daily intake.
+
+We make every effort to ensure that product information on our website is accurate and up to date, but packaging and ingredients may occasionally vary from images shown on site. Please refer to the product label and contact Healf before use if you have any questions regarding your specific allergies or intolerances.

healf-crawler/data/life-extension-advanced-milk-thistle.md

@@ -0,0 +1,40 @@
+# Advanced Milk Thistle
+
+> Source: https://healf.com/products/life-extension-advanced-milk-thistle
+
+**Brand:** Life Extension | **Price:** £19.99
+
+## Description
+
+**Key Benefits**
+
+- Enhanced absorption for a full spectrum of milk thistle actives.
+- Features silymarin and silybin—signature compounds of milk thistle.
+- Phospholipid complex supports effective delivery of key ingredients.
+- Perfect for those seeking a high-quality botanical supplement.
+
+Start your wellness routine with a thoughtfully crafted milk thistle formula. Advanced Milk Thistle from Life Extension brings together silymarin, silybin, and isosilybin A and B—compounds found in milk thistle fruit—combined with phospholipids to help support absorption. This unique blend is designed to deliver a premium supplement experience, making it easy to add the benefits of milk thistle to your daily routine.   
+
+Choose Advanced Milk Thistle for a modern approach to traditional botanicals, ideal for those who value quality and innovation in their supplements.
+
+## Ingredients
+
+Milk Thistle Phospholipid Proprietary Blend: Milk Thistle Extract (Fruit) [Providing 480 mg Silymarin, 180 mg Silybin, 48 mg Isosilybin A and Isosilybin B], Phospholipids, Siliphos® Phytosome Milk Thistle Extract (Fruit) [Providing 47.52 mg Silybin], Sunflower Oil, Capsule Shell (Gelatin), Glycerin, Purified Water, Colour (Carob Colour), Beeswax
+
+## Suggested Use
+
+Take two (2) softgels daily, in divided doses, or as recommended by a healthcare practitioner.  
+
+ Warnings:  
+ KEEP OUT OF REACH OF CHILDREN.  
+ DO NOT EXCEED RECOMMENDED DOSE.  
+ Do not purchase if outer seal is broken or damaged. When using nutritional supplements, please consult with your physician if you are undergoing treatment for a medical condition or if you are pregnant or lactating.  
+
+
+**Additional Information:**
+
+Food supplements and foods sold by Healf should not be used as a substitute for a varied, balanced diet and healthy lifestyle.
+
+If you are pregnant, breastfeeding, have a medical condition, or are taking any medications, please consult with a healthcare professional before use. Use products only if the seal is intact. Store in a cool, dry place, out of the reach of young children. Do not exceed the recommended daily intake.
+
+We make every effort to ensure that product information on our website is accurate and up to date, but packaging and ingredients may occasionally vary from images shown on site. Please refer to the product label and contact Healf before use if you have any questions regarding your specific allergies or intolerances.

healf-crawler/data/life-extension-ampk-metabolic-activator.md

@@ -0,0 +1,40 @@
+# AMPK Metabolic Activator
+
+> Source: https://healf.com/products/life-extension-ampk-metabolic-activator
+
+**Brand:** Life Extension | **Price:** £32.99
+
+## Description
+
+**Key Benefits**
+
+- With calcium to support normal energy-yielding metabolism.
+- Features a carefully selected blend of botanicals and minerals.
+- Designed for those who want to maintain their daily vitality.
+- Non-GMO and made with quality-assured ingredients.
+
+Start your day with confidence—AMPK Metabolic Activator brings together G. pentaphyllum (Jiaogulan) extract, hesperidin (a citrus flavonoid), and calcium. Calcium contributes to normal energy-yielding metabolism, helping you stay on top of your routine. This thoughtfully crafted formula is ideal for anyone looking to support their energy levels and overall wellbeing with a blend of botanicals and minerals you can trust.
+
+## Ingredients
+
+Hesperidin [from Orange Extract (Fruit)], Actiponin® Gynostemma Extract (Leaf), Calcium (as Calcium Carbonate), Bulking Agent (Microcrystalline Cellulose), Anti-Caking Agent (Stearic Acid), Croscarmellose Sodium, Capsule Shell (Hydroxypropyl Cellulose), Aqueous Film Coating (Hypromellose, Glycerin, Purified Water), Anti-Caking Agent (Silicon Dioxide), Anti-Caking Agent (Vegetable Stearate)
+
+## Suggested Use
+
+Take one (1) tablet daily.  
+
+ CAUTION:  
+
+ Not recommended for pregnant women and persons undergoing antidiabetic and anticoagulant treatment. Consult your physician in case of kidney disorder.  
+
+ WARNINGS:  
+ Keep out of reach of children. Do not exceed recommended daily dose. Do not purchase if outer seal is broken or damaged. When using nutritional supplements, please consult with your physician if you are undergoing treatment for a medical condition or if you are pregnant or lactating. A food supplement should not be used as a substitute for a varied and balanced diet and a healthy lifestyle. Store tightly closed in a cool, dry place.  
+
+
+**Additional Information:**
+
+Food supplements and foods sold by Healf should not be used as a substitute for a varied, balanced diet and healthy lifestyle.
+
+If you are pregnant, breastfeeding, have a medical condition, or are taking any medications, please consult with a healthcare professional before use. Use products only if the seal is intact. Store in a cool, dry place, out of the reach of young children. Do not exceed the recommended daily intake.
+
+We make every effort to ensure that product information on our website is accurate and up to date, but packaging and ingredients may occasionally vary from images shown on site. Please refer to the product label and contact Healf before use if you have any questions regarding your specific allergies or intolerances.

healf-crawler/data/life-extension-bioactive-complete-b-complex.md

@@ -0,0 +1,44 @@
+# BioActive Complete B-Complex
+
+> Source: https://healf.com/products/life-extension-bioactive-complete-b-complex
+
+**Brand:** Life Extension | **Price:** £9.99
+
+## Description
+
+**Key Benefits**
+
+- Helps unlock energy from your food with thiamine, riboflavin, niacin, B6, biotin, B12, and pantothenic acid—all contributing to normal energy-yielding metabolism.
+- Supports your mind and mood—B6, B12, niacin, thiamine, biotin, and folate contribute to normal psychological function and nervous system health.
+- Promotes healthy red blood cell formation and homocysteine metabolism with B6, B12, folate, and riboflavin.
+- Features bioactive forms of B vitamins for optimal absorption and utilisation.
+
+
+
+Give your body the B vitamin essentials it needs to thrive. BioActive Complete B-Complex brings together all eight B vitamins in their most usable forms, so you can feel confident you’re getting comprehensive support. Thiamine, riboflavin, niacin, B6, biotin, B12, and pantothenic acid all contribute to normal energy-yielding metabolism—helping you turn food into fuel and reduce tiredness and fatigue. B6, B12, niacin, thiamine, biotin, and folate support your nervous system and psychological function, while folate and B12 also play a role in normal blood formation. With active forms for better absorption, this formula is designed to fit seamlessly into your daily routine—so you can feel ready for whatever the day brings.
+
+## Ingredients
+
+Pantothenic Acid (as D-Calcium Pantothenate), Thiamine (Vitamin B1) (as Thiamine HCl), Niacin (as Niacinamide And Niacin), Vitamin B6 (as Pyridoxine HCl And Pyridoxal 5’-Phosphate), Inositol, Riboflavin (Vitamin B2) (as Riboflavin And Riboflavin 5’-Phosphate), Calcium (as D-Calcium Pantothenate, Dicalcium Phosphate), PABA (Para-Aminobenzoic Acid), Folate (as L-5-Methyltetrahydrofolate Calcium Salt), Vitamin B12 (as Methylcobalamin), Vegetable Cellulose (Capsule), Bulking Agent (Microcrystalline Cellulose), Anti-Caking Agent (Stearic Acid), Purified Water, Anti-Caking Agent (Silicon Dioxide)
+
+## Suggested Use
+
+Take two (2) capsules daily with food, or as recommended by a healthcare practitioner.  
+
+Caution   
+Temporary flushing, itching, rash, or gastric disturbances may occur.  
+
+Warnings   
+KEEP OUT OF REACH OF CHILDREN   
+ DO NOT EXCEED RECOMMENDED DOSE   
+Do not purchase if outer seal is broken or damaged.   
+When using nutritional supplements, please consult with your physician if you are undergoing treatment for a medical condition or if you are pregnant or lactating.  
+
+
+**Additional Information:**
+
+Food supplements and foods sold by Healf should not be used as a substitute for a varied, balanced diet and healthy lifestyle.
+
+If you are pregnant, breastfeeding, have a medical condition, or are taking any medications, please consult with a healthcare professional before use. Use products only if the seal is intact. Store in a cool, dry place, out of the reach of young children. Do not exceed the recommended daily intake.
+
+We make every effort to ensure that product information on our website is accurate and up to date, but packaging and ingredients may occasionally vary from images shown on site. Please refer to the product label and contact Healf before use if you have any questions regarding your specific allergies or intolerances.