Skip to content

Add data ingestion worker for official sources (fogos.pt, PROCIV, NASA FIRMS) #9

@lukesmmr

Description

@lukesmmr

Context

The Disasters API is currently a CRUD store — it receives data via HTTP but does not autonomously ingest from official sources. The mobile app depends on fresh data from:

  • fogos.pt — Portuguese wildfire data
  • PROCIV — Portuguese civil protection alerts
  • NASA FIRMS — Satellite fire detection

Without an ingestion worker, the API data is only as fresh as whatever manually pushes to it.

Proposed Solution

Create a scheduled worker (separate service or cron job) that:

  1. Polls official sources at regular intervals (e.g., every 5 minutes for active sources)
  2. Maps external data to the disasters schema
  3. Upserts into the API using the (source, external_id) unique constraint to prevent duplicates
  4. Updates status of existing records when source data changes (active → contained → resolved)

MVP Scope (Portugal only)

Source Endpoint Frequency Priority
fogos.pt TBD (web scraping or API) 5 min P1
PROCIV TBD (RSS or API) 5 min P1
NASA FIRMS https://firms.modaps.eosdis.nasa.gov/api 15 min P2

Architecture Options

  1. Separate Node.js service with cron (e.g., node-cron)
  2. Supabase Edge Function on a schedule
  3. GitHub Actions scheduled workflow (simplest for MVP)

Future: Global Expansion

Each country/region will need its own source adapters. The worker should be designed with a pluggable adapter pattern:

adapters/
  fogos-pt.ts
  prociv-pt.ts
  nasa-firms.ts
  # Future:
  # copernicus-eu.ts
  # cal-fire-us.ts

Labels

enhancement, infrastructure

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions