Skip to content

NBRoutingPeer never recovers after a forced re-auth #390

Description

@mvgijssel

NetBird Operator Version

0.7.0

Kubernetes Version

v1.35.0

CNI

Flannel

Describe the bug

Summary

An NBRoutingPeer's netbird daemon authenticates only once, at container start (netbird up with the operator-minted setup key). When the management server later forces a re-authentication — e.g. login-session expiry, or a host/VM suspend/resume — the running daemon drops to Daemon status: SessionExpired / NeedsLogin and never re-auths on its own:

failed to login to Management Service: rpc error: code = PermissionDenied desc = no peer auth method provided, please use a setup key or interactive SSO login

The pod stays Running (no container restart), so nothing recycles it and routing silently dies. In some cases the peer is fully deregistered (peer is not registered).

Why it's not self-healing

  • The NBRoutingPeer CR keeps reporting status.conditions: Ready=True — it reflects Deployment/k8s health, not the daemon's session state, so the dead session is invisible.
  • The generated Deployment has no liveness/readiness probe tied to session health, and the CRD exposes no way to add one (see Missing Probe Configuration in NBRoutingPeer Deployment #85), so kubelet never recycles the wedged pod.
  • Recovery today requires a manual kubectl rollout restart deploy/<routingpeer> (the reusable setup key in the secret re-enrolls the new pod).

The same wedge affects SidecarProfile-injected client sidecars.

Environment

  • operator v0.7.0, netbird client 0.72.4

Suggestions (any one would help)

  1. Surface daemon session state in NBRoutingPeer.status (flip Ready=False on SessionExpired/NeedsLogin) so it's observable/alertable.
  2. Add an opt-in liveness probe on the generated Deployment that fails on a non-connected session, so the pod self-recycles.
  3. Have the peer re-auth from the mounted setup key on session loss instead of only at startup.

Current workaround

An external watchdog CronJob that execs netbird status and rollout restarts the peer on a terminal state: vgijssel/setup#998

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions