nic-watchdog

A lightweight Go daemon that monitors Ethernet connectivity on Raspberry Pi nodes and automatically recovers from spontaneous link drops caused by known hardware bugs in the Ethernet PHY.

Problem

Raspberry Pi 5 and some Raspberry Pi 4 boards have a known hardware-level bug where the NIC spontaneously drops link or enters a unidirectional TX failure state — the Pi can receive packets but transmitted packets never arrive. The drops occur at random intervals with zero preceding kernel errors.

When the link drops, a simple ip link down/up cycle usually restores connectivity. However, in some cases the PHY TX path is only partially restored — enough for local subnet traffic but not for routed traffic to external IPs.

Upstream issues

• raspberrypi/linux#6420 — RPi5 macb/BCM54213PE, open since Oct 2024
• raspberrypi/firmware#1922 — RPi5 firmware-level, confirms bug occurs before Linux starts
• raspberrypi/linux#3108 — RPi4 genet/BCM54210PE, open since Jul 2019

No upstream fix is available.

How it works

nic-watchdog runs as a long-lived systemd service and checks external connectivity via ICMP ping every 10 seconds. When a failure is detected, it distinguishes between a full link-down (both gateway and external unreachable) and a partial TX failure (gateway reachable but external is not), then escalates recovery:

Step	Condition	Action
1	Gateway up, external down	Flush ARP cache and route cache
2	Still down after step 1	Restart systemd-networkd
3	Still down after step 2	Full interface down/up cycle
—	Both gateway and external down	Full interface down/up cycle immediately

A cooldown (default 10 minutes) prevents repeated full cycles. All state is in-memory — no files in /run.

Usage

nic-watchdog [flags]

Flags:
      --check-interval duration   Check interval (default 10s)
      --cooldown duration         Minimum time between full interface cycles (default 10m0s)
      --gateway string            Gateway IP (auto-detected if empty)
      --iface string              Network interface (auto-detected from default route if empty)
      --metrics-addr string       Address to serve Prometheus /metrics on (e.g. :9101); empty disables
      --ping-target string        External connectivity check target (default "8.8.8.8")
      --soft-max int              Max soft recovery attempts before escalating (default 3)

All flags can also be set via environment variables with the NIC_WATCHDOG_ prefix:

NIC_WATCHDOG_IFACE=eth1 NIC_WATCHDOG_COOLDOWN=5m nic-watchdog

Build

Requires Go 1.26+. Cross-compile for arm64:

make build

The binary is written to bin/nic-watchdog.

Available make targets:

make fmt          # Format code (gofmt + goimports)
make lint         # Run golangci-lint
make license      # Apply SPDX license headers
make check        # fmt + lint + license + go fix
make tidy         # go mod tidy
make build        # check + cross-compile
make clean        # Remove bin/
make deploy       # Build + Ansible deploy to all hosts

Deploy

The deploy target builds the binary and runs the Ansible playbook to deploy to all target nodes:

make deploy

The playbook (deploy/playbook.yml) targets pis, deskpis, and turingpis host groups.

Systemd service

Runs as Type=simple with Restart=always. Requires CAP_NET_ADMIN for interface cycling and ARP/route flush. ICMP probes use the Linux unprivileged datagram socket (IPPROTO_ICMP), gated by net.ipv4.ping_group_range — open by default on Debian / Raspberry Pi OS, so no CAP_NET_RAW is needed.

[Service]
Type=simple
ExecStart=/usr/local/bin/nic-watchdog --ping-target 8.8.8.8 --cooldown 600s --soft-max 3
Restart=always
RestartSec=5
AmbientCapabilities=CAP_NET_ADMIN

Journal output

level=INFO msg="discovered gateway" gateway=192.168.1.1 iface=eth0 routeIface=eth0
level=INFO msg="watchdog started" iface=eth0 routeIface=eth0 target=8.8.8.8 gateway=192.168.1.1 interval=10s
level=INFO msg="external unreachable, gateway up — flushing ARP and routes" gateway=192.168.1.1 attempt=1
level=INFO msg="external connectivity restored" after_soft_attempts=1
level=INFO msg="cycling interface" iface=eth0 gateway=192.168.1.1 target=8.8.8.8
level=INFO msg="connectivity restored after cycle"

Metrics

Prometheus metrics are off by default. Enable with --metrics-addr (or metrics_addr in the playbook):

nic-watchdog --metrics-addr :9101

Exposed series (excerpt — see /metrics for the full set, including free go_* and process_* collectors):

Metric	Type	Labels	Notes
nic_watchdog_external_reachable	gauge	target	Primary alerting signal
nic_watchdog_gateway_reachable	gauge	gateway	Distinguishes link-down from PHY TX failure
nic_watchdog_carrier_up	gauge	iface	Cable / PHY presence
nic_watchdog_check_total	counter	result	ok / external_down / gateway_recovered_on_retry / both_down
nic_watchdog_recovery_total	counter	step	flush / networkd_restart / cycle
nic_watchdog_recovery_failure_total	counter	step	Errors from each recovery step
nic_watchdog_soft_attempts	gauge	—	Current soft counter
nic_watchdog_last_cycle_timestamp_seconds	gauge	—	Unix time of the last full cycle
nic_watchdog_ping_duration_seconds	histogram	target, result	Latency spikes correlate with PHY soft failures

Minimal Prometheus scrape config:

scrape_configs:
  - job_name: nic-watchdog
    static_configs:
      - targets: ['turingpi01:9101', 'turingpi02:9101']

A useful starter alert:

- alert: NICWatchdogExternalDown
  expr: nic_watchdog_external_reachable == 0
  for: 2m

License

Apache 2.0 — see LICENSE.