longhorn: ship node-drain-policy + replica-replenishment-wait-interval Setting CRs in template tree

[oren-openteams]

Description

NIC already renders one Longhorn Setting CR from its template tree (pkg/argocd/templates/manifests/storage/longhorn-backup/setting-allow-recurring-job-while-volume-detached.yaml). Proposing we add two more that every EKS-hosted consumer of NIC needs to change from Longhorn's defaults:

• node-drain-policy — declared explicitly at Longhorn's own default (block-if-contains-last-replica), so ArgoCD selfHeal reverts drift and operators have a single git-managed file to flip to block-for-eviction-if-contains-last-replica before a nic deploy maintenance window.
• replica-replenishment-wait-interval — set to "0", overriding Longhorn's 600s default.

Both files would live alongside the existing setting under pkg/argocd/templates/manifests/storage/longhorn-backup/, so no new ArgoCD Application, no new tree.

Proposed contents (mirroring the pattern already there):

# setting-node-drain-policy.yaml
apiVersion: longhorn.io/v1beta2
kind: Setting
metadata:
  name: node-drain-policy
  namespace: longhorn-system
  labels:
    app.kubernetes.io/managed-by: nebari-infrastructure-core
value: "block-if-contains-last-replica"

# setting-replica-replenishment-wait-interval.yaml
apiVersion: longhorn.io/v1beta2
kind: Setting
metadata:
  name: replica-replenishment-wait-interval
  namespace: longhorn-system
  labels:
    app.kubernetes.io/managed-by: nebari-infrastructure-core
value: "0"

Use Case

replica-replenishment-wait-interval

Longhorn's default 600s exists for on-prem clusters where a briefly-offline node is expected to come back with its replica intact. On EKS, a "missing" node is almost always a terminated node — its EBS root disk is gone, the replica is not coming back. Waiting 600s just prolongs degraded-state exposure during a rolling NG update.

Longhorn's own EKS-specific doc explicitly recommends replica-replenishment-wait-interval=0 for exactly this reason. It's the only setting the EKS-specific doc names.

node-drain-policy

More nuanced. Longhorn's maintenance doc enumerates five drain policies:

• block-if-contains-last-replica (default) — safe under any cordon, but doesn't help during a planned NG cycle.
• block-for-eviction-if-contains-last-replica — actively evicts last-copy replicas before drain proceeds. But: the docs say "not recommended for normal use — may trigger replica eviction any time a node is cordoned. Only enable it during planned maintenance." An ordinary CA scale-in cordon would kick off pointless data movement.

Codifying the aggressive one permanently is against Longhorn's own guidance. Codifying only the default (which is what the cluster already has) sounds like a no-op — but declaring it explicitly gives operators:

1. A single git-managed file to edit for maintenance ("change value: to the aggressive one, push, ArgoCD reconciles, do the deploy, revert").
2. ArgoCD selfHeal reverting any accidental live drift back to the safe default.
3. A canonical place to document the runbook (as a comment block above value:).

Git becomes the audit trail for maintenance-window policy toggles, rather than a kubectl patch operators have to remember to roll back.

Broader context

Part of the same problem space as #572 (Longhorn PVCs lost when nic deploy cycles storage nodes). Neither of these Setting CRs prevents the failure mode described there — they're complementary hardening. The drain-policy toggle in particular is only useful during a planned maintenance window, and it doesn't protect replicas of detached volumes (Longhorn's drain-policy webhook fires on pod-eviction, and stopped replicas don't have a running pod for it to see). Pre-attach + backups remain necessary.

Known limitation to document

If NIC ships these, the drain-policy Setting file should carry the same maintenance-window runbook comment we're using downstream:

==== MAINTENANCE-WINDOW RUNBOOK ============================================
Before running `nic deploy` or any operation that will cycle EKS nodes:

  1. Edit this file: change `value:` below to
       "block-for-eviction-if-contains-last-replica"
  2. Commit + push to master.
  3. Wait for ArgoCD `longhorn-backup` app to sync (~1min), verify with:
       kubectl -n longhorn-system get settings.longhorn.io \
         node-drain-policy -o jsonpath='{.value}'
  4. Proceed with maintenance.
  5. After nodes settled + replicas rebuilt, revert this file back to
     `block-if-contains-last-replica` and push.

The aggressive policy does NOT protect replicas of DETACHED volumes.
Pre-attach precious detached volumes or ensure backups are current
before starting maintenance.
============================================================================

Without that comment block, an operator finding block-if-contains-last-replica in the file will assume it's a "correct default, do not edit" declaration, and reach for kubectl patch instead of git — losing the audit trail.

Testing

Follows the existing pattern in pkg/argocd/writer_test.go for the current Setting CR — two extra assertions in TestWriteAllToGit_LonghornBackup (or equivalent) that the two new files render at the expected paths with the expected values. No template variables involved, so tests are trivial.