perf(networking): make Cilium native routing (eBPF) the default — resolve the worker-bootstrap deadlock

[rustyrazorblade]

Goal

Make Cilium native routing (eBPF host routing) the default CNI datapath, replacing the current VXLAN tunnel mode, to remove the encapsulation overhead — the whole point of running Cilium/eBPF.

Current state

services/CiliumService.kt installs Cilium 1.19.4 with tunnelProtocol=vxlan + routingMode=tunnel (VXLAN encapsulation). Per the in-code rationale, native routing was avoided because Cilium's eBPF intercepts the K3s agent's outbound TCP to the API server during bootstrap → i/o timeout → worker-node bootstrap deadlock. VXLAN keeps host-to-host traffic outside eBPF interception and sidesteps it, at the cost of datapath performance. The install already points agents directly at the API server (k8sServiceHost/k8sServicePort), which alone did not resolve the deadlock.

Work

• Determine the native-routing config that bootstraps cleanly on K3s (candidates: routingMode=native + autoDirectNodeRoutes, bpf.hostLegacyRouting, kubeProxyReplacement, cni.exclusive=false, install ordering, etc.).
• Verify on a live multi-node cluster (control + ≥1 db-node worker) that the worker agent bootstraps and pod↔pod / pod↔service networking works under native routing.
• Flip the default in CiliumService.kt from VXLAN tunnel to native routing; keep/adjust tests.

Acceptance criteria

• WHEN a cluster is provisioned with the default config, THEN Cilium runs in native routing mode (no VXLAN encapsulation) AND all worker-node K3s agents bootstrap to Ready without the API-server i/o-timeout deadlock.
• WHEN native routing is active, THEN pod-to-pod and pod-to-service traffic works across nodes (verified: a pod on the db node reaches a service/pod on the control node and vice versa).
• Observability (Hubble) and the rest of the stack still function.

Notes

• Highest priority. Ephemeral clusters — no migration concern. Never disable functionality to work around the deadlock; fix the bootstrap ordering/config.

[rustyrazorblade]

Datapath config decisions (recorded after live AWS validation)

Validating this change on a real cross-AZ AWS cluster surfaced two landmines that unit tests can't catch. Recording the reasoning so it doesn't get re-litigated in #819 or any future Cilium work.

The config we landed on (Variant A — "safe native routing")

ipam.mode=eni, eni.enabled=true
routingMode=native, endpointRoutes.enabled=true   # the actual perf win: NO encapsulation, VPC-routable ENI pod IPs
ipv4NativeRoutingCIDR=<VPC CIDR>
kubeProxyReplacement=false                          # K3s keeps its kube-proxy
enableIPv4Masquerade=true
egressMasqueradeInterfaces=ens+                     # iptables masquerade; Nitro primary NIC is ens5, not eth0
bpf.hostLegacyRouting=true                          # keep host networking on the kernel stack
k8sServiceHost=<control private IP>, k8sServicePort=6443
# NO bpf.masquerade, NO tunnelProtocol=vxlan, NO routingMode=tunnel, NO autoDirectNodeRoutes, NO devices pin

Landmine 1 — ENI mode requires a named egress masquerade interface

With enableIPv4Masquerade=true and ipam.mode=eni, the cilium agent panics on startup: "Egress masquerading interfaces cannot be empty when IP masquerading is enabled with IPAM mode other than ClusterPool or Kubernetes." You must name the interface. On Nitro the primary NIC is ens5 (ENIs are ens6/ens7…), so the wildcard egressMasqueradeInterfaces=ens+ is correct. A hardcoded eth0 would be wrong and masquerade nothing. (The original design note that said to omit egress interfaces was wrong — that's what caused the panic.)

Landmine 2 — bpf.masquerade=true blackholes the node's own SSH + API

bpf.masquerade=true clears the panic, BUT per Cilium docs it auto-enables BPF host-routing, which attaches BPF to the primary NIC (ens5) and intercepts inbound SYN/return on the node's own IP — killing host SSH:22 and kube-apiserver:6443 from both the internet and intra-VPC, once the datapath converges. EC2 status checks stay ok/ok (it's an in-node BPF drop). Confirmed by cilium/cilium#46010 (KPR=true kills host networking on 1.19.3/1.19.4; kubeProxyReplacement=false does not trigger it; bpf.hostLegacyRouting=true alone was insufficient under KPR=true). This is our exact Cilium version.

Why kubeProxyReplacement=false (and never "both")

cilium install was effectively enabling kube-proxy replacement while K3s's own kube-proxy was still running (we did not pass --disable-kube-proxy) — undefined Service routing. The two datapaths must never run together. We chose to keep K3s kube-proxy and pin kubeProxyReplacement=false, which also sidesteps #46010 entirely. bpf.hostLegacyRouting=true is belt-and-suspenders to keep the host stack.

What we deliberately did NOT do (rejected: Variant B — "full eBPF")

kubeProxyReplacement=true + K3s --disable-kube-proxy + bpf.masquerade=true + pinned devices={ens5} squeezes more onto the eBPF fast-path, but #46010 shows it still blackholed 1.19.4 even with mitigations — unproven and higher-risk. Deferred until Cilium fixes it; could be a future issue.

The key framing

The performance goal of #805 is removing VXLAN encapsulation — and routingMode=native + ENI IPAM (pods get real VPC IPs, no tunnel) delivers that fully in Variant A. Egress SNAT (iptables) and Services (K3s kube-proxy) staying off the eBPF fast-path is a secondary, deliberate trade-off for reachability/stability, not the overhead this issue set out to remove.

Sources: Cilium masquerading & kube-proxy-free docs; cilium/cilium#46010, #22925, #19427.

[rustyrazorblade]

Live validation update — clean-run results + two more node/infra fixes

The clean up from the baked Variant A code got all the way through Cilium install + K3s agents + most of the observability stack (pods get VPC-routable per-AZ ENI IPs, cross-AZ pod→pod routes with no tunnel, no host blackhole — the core native-routing datapath is proven). But it exposed two further, real, Cilium-ENI-specific defects that gate a fully-successful up. Neither exists on Flannel (Flannel never attaches secondary ENIs).

FIX #1 — OS must not manage Cilium's secondary ENIs (PROVEN)

When Cilium ENI IPAM attaches a second ENI (ens6) to a node — driven by the IPAM pre-allocation watermark, NOT actual IP exhaustion (only a handful of pod IPs are in use) — Ubuntu's systemd-networkd DHCPs ens6 and adds a competing default route (default via … dev ens6). That multi-homes the host → breaks IMDS, egress, and the node's kubelet→apiserver channel → the node goes NotReady → every pod on it (incl. pyroscope-ebpf, which up waits on) is marked not-Ready → GrafanaUpdateConfig times out.

This is a documented, standard Cilium requirement — Cilium AWS ENI docs, "Node Configuration": "any system service trying to manage newly attached network interfaces will interfere… configure your Linux distribution to not manage the newly attached ENI devices." Corroborated by aws/amazon-vpc-cni-k8s#3524. Managed EKS AMIs ship amazon-ec2-net-utils that already does this; our self-managed Ubuntu AMI must add it.

Fix (live-proven): a systemd-networkd/netplan drop-in that leaves secondary ENIs (ens6+, NOT the primary ens5) to Cilium. Proven config on the live cluster recovered the whole chain — single default via ens5, IMDS restored, all 3 nodes Ready, pyroscope-ebpf 1/1 Ready on all nodes, persistent. Implementation notes: must be baked into the AMI (netplan/cloud-init form so cloud-init won't regenerate over it) and must pre-exist before the first secondary ENI attaches (ens6 is attached at runtime by cilium-operator). Tradeoff to settle: [Network] DHCP=no (proven; networkd still brings the link UP, sidesteps the cilium#34590 'ENI stuck DOWN' risk) vs the doc's Unmanaged=yes (hands link-up to Cilium).

FIX #2 — multi-ENI breaks AWS auto-assigned public-IP egress (DECISION PENDING)

AWS's auto-assigned public IPv4 only provides egress NAT for single-ENI instances. The moment Cilium attaches the 2nd ENI, that node loses internet egress — confirmed live: the busy node (control0, 2 ENIs) can't reach the internet even from its primary IP, while the single-ENI db nodes can. Consequence: it can't pull images → its obs pods (grafana/registry/tempo/otel) ImagePullBackOff → up still fails.

This is an architecture decision (owner):

• (b) EIP per node's primary ENI (recommended) — Elastic IPs keep working with multiple ENIs. Cleanly scoped to --cni=cilium only, so Flannel clusters are untouched; preserves the public-IP-per-node SSH model. Cost: EIP region limits + teardown release.
• (a) NAT Gateway — standard, but a shared VPC/subnet re-architecture (private subnets) that changes the Flannel path too and breaks the current public-IP SSH access model.
• (c) pre-pull images before scaling — fragile, rejected.

Design constraint — both CNIs stay selectable

--cni=flannel (default) and --cni=cilium remain fully selectable. All Cilium-ENI infra above is applied only on the --cni=cilium path: the networkd drop-in is inert on Flannel (no secondary ENIs), and the egress fix is gated on the CNI choice. This is a reason to prefer EIP-per-node (cleanly conditional) over NAT-GW (shared change).

Net for #805

Datapath proven; a fully-green up needs BOTH FIX #1 (secondary-ENI networkd drop-in, baked into AMI) AND FIX #2 (multi-ENI egress — EIP or NAT-GW). PR #820 gated on both. This is the accumulating cost of running Cilium ENI native routing on a self-managed AWS AMI — each piece is a known/documented gotcha, not a dead end.

[rustyrazorblade]

Two-CNI acceptance test — run log (in progress)

Testing both --cni=flannel and --cni=cilium end-to-end on the rebuilt AMI (with the secondary-ENI Unmanaged=yes drop-in baked in), every DB kit on each. Plan: test-plans/cni-flannel-cilium-validation.md. Topology per phase: 1 control + 3 db + 3 app, across 3 AZs, i4i.2xlarge.

AMI rebuild

• Base AMI ami-0ae588367df58a80b + Cassandra AMI ami-02ae4ff652cb9f6a2 rebuilt; the packer log confirms configure_cilium_eni_networkd.sh ran (drop-in baked in).
• Caught a collateral break: the Cassandra AMI build 404'd on apache-cassandra-6.0-bin.tar.gz because this branch predated the ci: standardize nightly Cassandra 6.0 artifact to 6.0-HEAD #822 rename to 6.0-HEAD. Fixed by merging main into the branch (cassandra_versions.yaml → 6.0-HEAD).

Phase 1 — Flannel (default): PASS ✅

• up exit 0; all 7 nodes Ready; CNI = Flannel.
• Drop-in confirmed INERT on Flannel — every node single-ENI, no ens6 (the baked 06-cilium-eni-unmanaged.network matches nothing). This is the key regression proof: the AMI change does not affect Flannel clusters.
• Obs stack fully Ready incl. pyroscope-ebpf 1/1 ×7.
• Kits — 8/10 PASS: cassandra (3-node ring + cross-AZ RF=3 read via CONSISTENCY ALL), postgres, clickhouse, ignite3, presto, kafka, flink, sysbench (oltp_read_write vs postgres: 127 tps/2551 qps).
• 2 kit failures, CNI-INDEPENDENT (fail identically on both CNIs; filed P0 separately): bug(tidb kit): install fails — charts.pingcap.org helm repo is dead (NXDOMAIN), no pods created #826 tidb (charts.pingcap.org NXDOMAIN — dead helm repo), bug(trino kit): all workers CrashLoopBackOff — unsupported config properties (web-ui.authentication.type / web-ui.user) #827 trino (workers CrashLoop on rejected web-ui.* config).

Phase 2 — Cilium (--cni=cilium): NOT PASSING YET ❌

• Attempt 1: up failed at EC2 instance creation on an eventual-consistency race — DescribeInstances returned InvalidInstanceID.NotFound (SDK Attempt Count 1, no retry) before K3s/Cilium ran. Not a CNI issue. Finding: the RunInstances→DescribeInstances path needs a resilience4j retry on InvalidInstanceID.NotFound (provisioning-robustness gap, separate from perf(networking): make Cilium native routing (eBPF) the default — resolve the worker-bootstrap deadlock #805).
• Attempt 2 (retry): cleared the flake (transient — confirmed), provisioned instances, installed K3s + Cilium, reached the observability stack. The secondary-ENI drop-in WORKED — pyroscope-ebpf is no longer the failure. But up failed again at GrafanaUpdateConfig, this time on a different pod: s3manager in CrashLoopBackOff. Since s3manager came up fine on Flannel, this is being root-caused as likely Cilium-related (node-partition class, or an S3/service reachability issue via native routing), not orthogonal. Cluster left up for debugging; not patched.

Current status

Cilium up does not yet complete cleanly — s3manager CrashLoop is the open blocker under investigation. #805 is not proven until a clean --cni=cilium up completes with no error and the full kit matrix passes. Flannel side is proven (no regression, drop-in inert).

[rustyrazorblade]

Cilium up-failure root cause: hostNetwork → ClusterIP broken under kubeProxyReplacement=false

Root-caused the s3manager/GrafanaUpdateConfig failure on the clean Cilium run — it's a real Cilium datapath gap tied to design Decision 5 (kubeProxyReplacement pinned OFF).

What works (drop-in validated)

• All 7 nodes Ready (no NotReady node — unlike the pre-drop-in surgeried cluster).
• pyroscope-ebpf 1/1 on all 7 nodes — the exact pod that failed before; the secondary-ENI drop-in FIXED it.
• Baked drop-ins present (05-primary / 06-unmanaged), ens6 marked unmanaged.

The blocker: hostNetwork → ClusterIP returns EPERM

• The obs backends that crashloop — tempo, pyroscope(server), s3manager, grafana — are all hostNetwork pods that resolve DNS via the CoreDNS ClusterIP (dnsPolicy ClusterFirstWithHostNet → resolv.conf 10.43.0.10).
• Tempo crash (verbatim): lookup <bucket>.s3.us-west-2.amazonaws.com on 10.43.0.10:53: write udp ...->10.43.0.10:53: write: operation not permitted — the DNS query to the CoreDNS ClusterIP is blocked.
• Direct repro from the host on EVERY node (single- and multi-ENI): dig @10.43.0.10 ... → refused; >/dev/udp/10.43.0.10/53 → EPERM. So it's not the ens6 anomaly, not node-status, not egress (host→S3 works, aws s3 list-buckets works).
• Chain: tempo/s3manager/pyroscope/grafana can't resolve via ClusterIP DNS → crashloop → up's obs-readiness gate times out → GrafanaUpdateConfig fails. Pods that don't need ClusterIP DNS at startup (victoriametrics, victorialogs, registry, pyroscope-ebpf) are all Running 0 restarts — the discriminator.

Why Cilium-specific

On Flannel, k3s kube-proxy programs iptables DNAT for ClusterIP including host-sourced traffic → these same obs pods were all Running (Phase 1 PASS). Under Cilium with kubeProxyReplacement=false, host-namespace → ClusterIP is not serviced (EPERM). Decision 5 (kpr OFF this cut, to isolate the change) has a real consequence: hostNetwork pods lose ClusterIP access.

Secondary finding (real, but NOT the up-blocker)

The baked drop-in does NOT fully suppress the competing ens6 default route on hot-attach: cloud-init's netplan-hotplug generates /run/systemd/network/10-netplan-ens6.network and DHCPs ens6, which Unmanaged=yes doesn't stop. ens5's lower metric (1024 vs 1026) keeps node-status/egress working (so it doesn't break up), but the drop-in isn't fully doing its job on hot-attach — needs hardening (also disable cloud-init net hotplug, or explicit route suppression).

Decision needed

The kpr=false hostNetwork→ClusterIP gap must be solved for the obs stack. Likely fix: enable kubeProxyReplacement=true (Cilium services ClusterIP for host traffic too) + --disable-kube-proxy on k3s + host-legacy-routing — i.e. the higher-risk 'Variant B' (full eBPF) path we deliberately deferred (reopens cilium#46010 host-blackhole risk, needs careful validation). Cluster left up; not patched. Owner decision pending on keep-pushing vs reassess.