Problem
Every IODA outage report embeds its chart as a raw SVG string at metadata.rawAPIResponse.image (~330KB; IODA docs average ~440KB vs ~1KB for other reports). Worse, the IODA channel re-scrapes and overwrites that SVG on every poll while an outage is ongoing.
Because the reports/alerts list loads 50–100 reports at once, a single page pulls tens of MB out of Mongo (~43MB measured for the alerts view). As the IODA collection grew, this crossed the 60s API_REQUEST_TIMEOUT, which:
- timed out the alerts list (500s), and
- triggered a double-send crash (
ERR_HTTP_HEADERS_SENT) that took down the API process.
The immediate timeout/crash was patched separately by excluding the SVG field from the list query — but that's a band-aid: documents are still huge, and the alert table / compare views can't show charts from list data.
Root cause
The chart image is stored in the document rather than as a file. The data model treats a large, re-fetched blob as inline report metadata.
High-level fix
Move IODA chart SVGs out of MongoDB into the existing /media file storage, keeping only a small key on the document:
- Persist each chart SVG to disk keyed by the report's
guid (ioda/charts/<hash>.svg). Stable key ⇒ re-fetches overwrite the same file in place (no orphans).
- Store the key (not the SVG) on the report; the API serializes it into a
/media/... URL.
- Frontend renders the chart with
<img src={imageUrl}>.
- One-off, idempotent migration backfills existing IODA reports.
- Remove the list-query band-aid once documents are small, restoring charts in the table/compare views.
Result (dev): alerts list dropped from ~43MB/60s-timeout to ~90KB/0.3s.
Deploy note
The backfill migration is a required deploy step (not optional cleanup): removing the band-aid means any un-migrated report would otherwise reintroduce the timeout. It must run on the host that serves /media, since SVGs are written to that host's local disk.
Problem
Every IODA outage report embeds its chart as a raw SVG string at
metadata.rawAPIResponse.image(~330KB; IODA docs average ~440KB vs ~1KB for other reports). Worse, the IODA channel re-scrapes and overwrites that SVG on every poll while an outage is ongoing.Because the reports/alerts list loads 50–100 reports at once, a single page pulls tens of MB out of Mongo (~43MB measured for the alerts view). As the IODA collection grew, this crossed the 60s
API_REQUEST_TIMEOUT, which:ERR_HTTP_HEADERS_SENT) that took down the API process.The immediate timeout/crash was patched separately by excluding the SVG field from the list query — but that's a band-aid: documents are still huge, and the alert table / compare views can't show charts from list data.
Root cause
The chart image is stored in the document rather than as a file. The data model treats a large, re-fetched blob as inline report metadata.
High-level fix
Move IODA chart SVGs out of MongoDB into the existing
/mediafile storage, keeping only a small key on the document:guid(ioda/charts/<hash>.svg). Stable key ⇒ re-fetches overwrite the same file in place (no orphans)./media/...URL.<img src={imageUrl}>.Result (dev): alerts list dropped from ~43MB/60s-timeout to ~90KB/0.3s.
Deploy note
The backfill migration is a required deploy step (not optional cleanup): removing the band-aid means any un-migrated report would otherwise reintroduce the timeout. It must run on the host that serves
/media, since SVGs are written to that host's local disk.