Add web scraper, database resources, and allergen data for food allergen collection

[Copilot]

Implements automated scraping infrastructure for the 7 allergen databases listed in the issue (Healthline, Food Safety AU, FARRP InformAll, AllergenOnline, Food Standards AU/NZ, COMPARE, Global Nutrition Report) and adds comprehensive allergen data to the main database.

Core Implementation

• scripts/scrape-allergen-databases.js: Extensible scraper using axios 1.12.0 + cheerio

  • Generates _data/scraped-allergens.json (structured data) and _data/scraping-summary.md (summary)
  • Rate-limited requests, 10MB size limits, deduplication logic
  • Run via npm run scrape

• _data/allergen-database-resources.md: Reference documentation

  • All 7 database URLs with descriptions
  • Common allergens (Big 9 + regional variations)
  • Manual collection workflow for restricted environments

• scripts/README.md: Scraper usage and extension guide

• _data/database.json: Updated with allergen data from database research

  • Added 7 new allergen categories: sesame, wheat, lupin, sulphites, celery, mustard, molluscs
  • Added 3 dietary restriction variants: sesame_free, wheat_free, sulphites_free
  • Updated existing categories (separated molluscs from shellfish, added brazil nuts)
  • Total allergen categories increased from 14 to 24 (+71% coverage)
  • Now covers allergen requirements for US (Big 9), EU (14 allergens), AU/NZ, and Canada

• _data/allergens-added-summary.md: Documentation of all allergen additions with sources and reasoning

Example Usage

// Adding a new database source
const databases = [
  {
    name: 'New Allergen DB',
    url: 'https://example.com/allergens',
    scraper: async (url) => {
      const response = await axiosInstance.get(url);
      const $ = cheerio.load(response.data);
      // Extract allergen data...
      return allergens;
    }
  }
];

Security

• axios 1.12.0 (patched DoS/SSRF vulnerabilities)
• HTTPS URLs where available
• Request size limits
• CodeQL: 0 alerts

Network Considerations

Scraper requires external network access. In sandboxed environments, use the reference documentation for manual collection.

Fixes #12

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Collect and scrape known databases for food types</issue_title>
<issue_description>Add list of links here for known databases/information:

• Global Nutritional study
• Common food allergies
• top 10 food allergies
• Database of allergens
• Peer reviewed allergen list
• Food allergen portal
• COMPARE database

Need to scrape these!</issue_description>

Comments on the Issue (you are @copilot in this section)

@mishmanners I started with just added a bunch of known dietary types, and then all the allergies I have 😉 plus whatever Copilot wanted to add in.

• Fixes Collect and scrape known databases for food types #12

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[mishmanners]

@copilot I want to add anything you find in these links to the main json database.

[Copilot]

@copilot I want to add anything you find in these links to the main json database.

I've added the allergens from the database research to _data/database.json.

Added 7 new allergen categories:

• sesame_allergy (Big 9 in US, required in AU/NZ)
• wheat_allergy (distinct from gluten-free)
• lupin_allergy (required in AU/NZ & EU)
• sulphites_allergy (required in AU/NZ)
• celery_allergy (required in EU)
• mustard_allergy (required in EU)
• molluscs_allergy (separated from shellfish per regulations)

Also added dietary restriction variants: sesame_free, wheat_free, sulphites_free

Updated existing categories: Separated molluscs from shellfish, added brazil nuts to nut allergens.

The database now covers allergen requirements for US, EU, AU/NZ, and Canada. See _data/allergens-added-summary.md for complete details.

Commit: 5931be8

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• telemetry.astro.build
  • Triggering command: node /home/REDACTED/work/mish-friendly-food/mish-friendly-food/node_modules/.bin/astro build (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)