Skip to content

[Bug]: NEXRAD ingester hangs when a single item fails to download.  #10

Description

@cwdaniel

Describe the bug
The NEXRAD ingester process hangs permanently after 1-3 days of uptime, causing all 159 station scan keys to expire from Redis. NEXRAD radar data disappears from the map at z8+ with no recovery until the container is manually restarted. The container
reports running=true, restarts=0 so Docker's restart policy never triggers.

Steps to reproduce

  1. Deploy RadrView with all services running
  2. Verify NEXRAD tiles render at z8+ (stations show data)
  3. Wait 1-3 days
  4. Zoom to z8+ — NEXRAD data is gone, only MRMS composite tiles visible
  5. Check Redis: docker exec radrview-redis redis-cli keys 'nexrad:scan:*' | wc -l returns 0
  6. Check ingester: docker logs radrview-ingest-nexrad --tail 3 --timestamps shows last log entry from days ago
  7. Check container: docker inspect radrview-ingest-nexrad --format '{{.State.Status}} {{.State.Running}}' reports running true

Expected behavior
NEXRAD station data should persist indefinitely. The ingester should poll every 60 seconds and never hang.

Screenshots
N/A — the symptom is missing NEXRAD tiles at z8+ (transparent tiles where station coverage should be).

Environment

  • OS: Ubuntu (AMD 4090 server)
  • Docker version: Docker Compose v2
  • RadrView version/commit: main@c17d3c0 through main@6e07f11

Logs
Ingester appears healthy but last output is days old:
$ docker logs radrview-ingest-nexrad --tail 3 --timestamps
2026-04-09T15:48:20Z {"updated":26,"total":159,"msg":"NEXRAD poll cycle complete"}
2026-04-09T15:49:14Z {"updated":18,"total":159,"msg":"NEXRAD poll cycle complete"}
2026-04-09T15:50:29Z {"updated":33,"total":159,"msg":"NEXRAD poll cycle complete"}

Redis monitor shows zero nexrad operations during a full poll cycle:
$ timeout 90 docker exec radrview-redis redis-cli monitor | grep -i nexrad | head -50
(empty)

Additional context
Three root causes identified:

  1. await fileResp.arrayBuffer() hangs forever — AbortSignal.timeout(30000) on fetch() covers the connection and headers, but the body read is a separate operation. If S3 sends headers then stalls the body transfer, arrayBuffer() never resolves.
  2. No Redis command timeout — ioredis defaults to infinite command timeout. A stuck hset or expire call hangs forever.
  3. Promise.allSettled propagates hangs — stations are polled in batches of 10 via Promise.allSettled. If one fetchLatest hangs, the entire batch hangs, blocking the main poll loop permanently.

With 159 stations × 60s cycles (~187K HTTP+Redis operations over 3 days), one stalled operation is statistically inevitable. The Node.js event loop stays alive (container shows running=true) but the main async loop is stuck on an unresolved promise, so
restart: unless-stopped never triggers.

Secondary issue: refreshScanTTL calls redis.expire() on expired keys, which silently returns 0 (no-op). The "data unchanged" code path never rewrites the scan data, so once keys expire they never come back.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions