boot-node: discv5 can wedge silently — no health signal tied to discovery actually working

[iurii-ssv]

Summary

A boot node can stop serving discovery while every external health signal still reports it as healthy. We recently had a boot node in this state for roughly 20 days without detection. Nothing in the boot node reports on whether discv5 is actually working, so the failure is invisible until a node somewhere has to cold-bootstrap and finds no peers.

What happened

The boot node's discv5 UDP socket was wedged: the socket stayed bound, but the application had stopped draining it. Inbound FINDNODE requests queued in the kernel receive buffer and were eventually dropped, so every lookup against it returned nothing and its discv5 routing table stayed permanently empty.

Meanwhile the process was alive and the HTTP server on the TCP port kept answering normally, so from the outside the boot node looked fine.

The cluster it serves kept working the whole time on warm peer tables. When those nodes were eventually restarted for an unrelated reason, they cold-bootstrapped, found no peers, and the cluster partitioned — 20 days after the actual fault.

Why it hid

bootNode.Start() runs the discv5 listener and the HTTP server as independent goroutines (utils/boot_node/node.go). httpServer.ListenAndServe() continuing to serve says nothing about whether discv5's read loop is still draining the UDP socket. Any liveness check pointed at the TCP port therefore passes while discovery is completely dead.

This is also why it was hard to diagnose from outside: TCP reachable, process up, logs clean (the boot node logs nothing after startup), ENR unchanged.

Proposal

Expose a health signal tied to discovery actually functioning, and make it fail closed.

The boot node already has everything needed. The /p2p handler calls listener.AllNodes() and dumps the discv5 routing table. For a boot node specifically, an empty routing table is definitionally broken — its entire job is to know about other nodes.

Concretely:

• Add a health endpoint (or a status code on the existing one) that returns non-200 when the routing table has been empty for longer than some grace period, so a Kubernetes liveness probe restarts the pod automatically. A bounce is a complete fix for this failure mode, so self-healing is achievable rather than just alerting.
• The grace period needs to tolerate genuine cold start, where the table is legitimately empty for a short while.

This would have turned a 20-day silent outage into a self-correcting blip.

As a datapoint on how clearly this discriminates, comparing routing table sizes across our boot nodes at the time:

boot node	AllNodes()
wedged	0
healthy	72
healthy	102

Related

• Separately, there was a reachable path by which the operator node's own discv5 stack could wedge with the same observable signature, via the blocking send to the Unhandled channel in the fork-listener setup. (Fixed in network/discovery: stop undecodable packets from wedging discv5 #2980 — SharedUDPConn now drains Unhandled through an always-receptive goroutine that drops-and-counts on overflow instead of blocking.)

Resolution plan

Two things to separate: the wedge itself, and the lack of detection (this issue's core).

Wedge — operator-node path addressed by #2980. The operator's discv5 wedging via the blocking send to the Unhandled channel (see Related above) is fixed there, so an undecodable-packet flood can no longer stall the socket. The boot node was never exposed to this path — it runs a single discover.ListenV5 with no Unhandled channel — so its wedge had a different cause, which is exactly why detection (not just this one mechanism) is what matters.

Detection — missing on both node types. The operator has the same blind spot as the boot node: p2pNetwork.Healthy() only checks readiness plus a discovery-bootstrap-failed flag, which a runtime wedge never trips (the bootstrap loop keeps running, just yielding nothing). The difference is that the operator already has a self-heal path — the hprobe watchdog restarts the node on persistent unhealth — while the boot node has none.

Decided design — one shared signal, scoped actuation:

• Shared primitive: a read-timestamp wrapper on the discv5 socket. "How long since we last drained a packet" is the direct, cause-agnostic wedge signal, and both node types own the conn they hand to ListenV5.
• Operator: feed staleness into p2pNetwork.Healthy() → the existing hprobe watchdog restarts the node. No routing-table check — a stale-but-populated table would mask a wedge.
• Boot node: a fail-closed health check that returns non-200 when the socket read is stale or the routing table has been empty past a cold-start grace → Kubernetes liveness restarts the pod. Empty-table is the boot node's definitional health (the 0-vs-72/102 datapoint above).
• Starting grace values: ~3 min read-staleness (both node types); ~10 min empty-table cold-start (boot node only).

Work — implemented in #2982:

1. Shared read-timestamp primitive (TimedConn).
2. Operator: wrap the post-fork socket, extend Healthy() → hprobe restart.
3. Boot node: fail-closed health endpoint.