Skip to content

Improve observability: no diagnostic signal when liveness-probe-triggered restarts are caused by silent handshake failures #13

Description

@czprz

Problem

When the wireguard sidecar's liveness probe (wg show wg0 latest-handshakes within 300s) fails and kubelet kills the container, there is no diagnostic signal anywhere (container logs, events, wg show) indicating why the handshake never completed. The container logs only show the normal startup/teardown sequence (wg-quick up/down), and wg show wg0 just shows latest handshake: (none) with no error code or reason.

This makes it impossible to distinguish, from the client side alone, between several very different root causes:

  • Wrong/rotated/revoked peer public key at the VPN provider
  • Provider account expired/suspended
  • Provider's WireGuard service down (while the host still answers ICMP)
  • A network path issue specific to UDP (not caught by ICMP-based reachability checks)

In our case (see context), the container silently retried a failing handshake in a tight ~2.5–5 minute crash loop for 20+ hours before the actual cause (a rotated client key at the VPN provider) was identified — entirely through external network testing, not anything the pod itself surfaced.

Suggested improvements

  1. Log handshake status periodically (e.g. every livenessProbe.periodSeconds) to stdout, even when the probe succeeds, so kubectl logs shows a rolling history of wg show wg0 transfer/handshake state rather than just startup/teardown noise.
  2. On liveness probe failure, have the probe script (or a preStop hook) log the last known wg show wg0 dump output (including sent/received byte counters and handshake epoch) before the container exits — this alone would have made "0 bytes received, ever" visible immediately in kubectl logs --previous, instead of requiring live kubectl exec access mid-crash-loop.
  3. Consider documenting in the README that a handshake epoch of 0 combined with climbing sent bytes and zero received bytes means "the tunnel is not receiving any response from the peer" — a strong signal to check the server/provider side (key rotation, account status), not the local network — since this failure mode is easy to misattribute to local network instability.

Environment

  • Chart: dever-labs/transmission-wireguard
  • Image: ghcr.io/linuxserver/wireguard:1.0.20250521-r1-ls102
  • Provider: Mullvad
  • Symptom: 256+ restarts over 20h, all with Exit Code: 0 / Completed, zero completed handshakes for the container's entire lifetime each cycle.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions