Skip to content

Fix invalid Prometheus metric names from iostat parse glitches - #4711

Open
sophoah wants to merge 1 commit into
OffchainLabs:masterfrom
sophoah:fix-iostat-invalid-device-metric-name
Open

Fix invalid Prometheus metric names from iostat parse glitches#4711
sophoah wants to merge 1 commit into
OffchainLabs:masterfrom
sophoah:fix-iostat-invalid-device-metric-name

Conversation

@sophoah

@sophoah sophoah commented Jul 17, 2026

Copy link
Copy Markdown

Problem

Prometheus scraping a Nitro node's /debug/metrics/prometheus endpoint failed entirely with:

Error scraping target: invalid metric type "99_await gauge"

The node itself was emitting an illegal metric name, not a Prometheus config problem. The iostat_* metrics come from util/iostat/iostat.go, which spawns iostat -dNxy 1 and parses its stdout to register per-device gauges (iostat_<device>_{await,readspersecond,writespersecond}). A parse glitch caused the literal string "99.99" to get captured as a device name. Real devices on the affected host are things like vg0-root, nvme0n1, loop0, never a bare decimal number. That produced iostat_99.99_await, which is invalid per the Prometheus exposition format since metric names have to match [a-zA-Z_:][a-zA-Z0-9_:]* and dots aren't allowed.

A single invalid metric name aborts the whole scrape, not just that one series. And go-ethereum's metrics registry never unregisters a metric once it's created, so this one bad row permanently broke metrics collection for the node until it was restarted.

There was already some sanitization in place (strings.ReplaceAll(stat.DeviceName, "-", "_"), added in #2491 for LVM device names like vg0-root), but it only handled hyphens, never dots or anything else. Nothing validated that a parsed token actually looked like a plausible device name before it got registered as a metric.

Fix

Two changes in util/iostat/iostat.go:

  1. parseStream now skips a row if its parsed device name parses successfully as a float (e.g. 99.99), since real device names never do. This stops the bad rows from reaching metric registration in the first place.
  2. RegisterAndPopulateMetrics now sanitizes device names with the existing shared metricsutil.CanonicalizeMetricName helper (already used elsewhere in the codebase for this) instead of just replacing hyphens, so any other invalid character gets handled too.

Testing

Added a parseStream test case confirming a row with a numeric device name (99.99) gets dropped.

Added util/metricsutil/metricsutil_test.go, which didn't exist before, confirming CanonicalizeMetricName sanitizes real device names seen in production (vg0-swap, vg0-root, nvme0n1, loop0) the same way the old hyphen-only replacement did. So this change doesn't rename any existing legitimate metric, it only changes behavior for the previously-broken case.

go test, go vet, and gofmt all pass for util/iostat and util/metricsutil.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant