Using Zarf to initialise a multi-node k3s cluster across clean bare metal nodes?

[MarshallEvergreen]

Hi! Been trialling Zarf to bootstrap k3s onto an airgapped bare metal node — works brilliantly for a single-node setup. 🙌

My question: what's the recommended approach for using Zarf to initialise a multi-node k3s cluster across clean bare metal nodes in an airgapped environment?

Specifically:

• Does Zarf have any built-in mechanism for joining additional nodes to the initial single node cluster that it can create?
• If not, is the recommended pattern to pre-initialise the cluster (connect all nodes via k3s first), then run zarf init against the existing cluster?

Thanks in advance!

[brandtkeller]

Hey @MarshallEvergreen !

Multi-node is a tricky/exciting space for which Zarf tends to defer to other tools to orchestrate infrastructure-specific options. Zarf itself doesn't have the awareness of remote filesystems for zarf package deploy - meaning that we can't run a deploy that installs to multiple filesystems through options such as ssh.

In theory you could create a zarf package that only installs the k3s component and then you would run it on each baremetal node with different configurations for server/agent. We've seen this done in some capacity such as this example - but this is still something you need to run imperatively across multiple nodes or automate with the likes of tooling such as ansible.

[MarshallEvergreen]

Hey @brandtkeller! Thanks for the quick response and the example link!

So if I'm interpreting correctly, bootstrapping a multi-node k3s cluster from bare metal with Zarf still requires some upfront manual orchestration (e.g. Ansible) to connect the nodes.

Assuming the cluster is already up, I have a couple of follow-up questions about running zarf init against a pre-existing multi-node cluster.

1. zarf-agent: Since this is a cluster-level mutating webhook registered with the API server, it should intercept image pull requests across all nodes without any per-node setup - is that correct?

2. Registry availability: The default registry is a single-replica deployment with a ReadWriteOnce PVC, so it runs on one node but is reachable by the rest of the cluster via Kubernetes networking. Combined with the webhook behaviour above, this means any node in the cluster should be able to schedule pods that pull images from the Zarf registry?

To summarise my understanding: zarf init with k3s bootstrapping is currently limited to a single node, but running zarf init against a pre-existing multi-node cluster (without the k3s component) should work fine for the reasons above - is that a fair characterisation?

Thanks in advance and for the awesome work on zarf so far!

[brandtkeller]

1. Correct! Nodes can be added without any impact to expected workflows
2. Also correct but with a little added nuance. There are configurations of kube-proxy (using nftables) and other CNI's that may block the nodeport connectivity. We're working to ensure zarf stays agnostic and compatible with all k8s distros. As such there is currently an alternative registry mode proxy for those scenarios while we work on potential changes in kubernetes upstream.