design-proposal: cross-cluster mesh for tenant access to host services#7
Draft
Andrei Kvapil (kvaps) wants to merge 7 commits into
Draft
design-proposal: cross-cluster mesh for tenant access to host services#7Andrei Kvapil (kvaps) wants to merge 7 commits into
Andrei Kvapil (kvaps) wants to merge 7 commits into
Conversation
Propose a controller-driven design that wires Cozystack tenant clusters into a node-to-node WireGuard mesh with the host cluster, using Kilo's mesh-granularity=cross topology. The motivating use case is exposing a Rook-managed Ceph cluster to tenant pods. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Tip 💬 Introducing Slack Agent: The best way for teams to turn conversations into code.Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.
Built for teams:
One agent for your entire SDLC. Right inside Slack. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a design proposal for a cross-cluster mesh using Kilo to allow tenant clusters to access host-cluster services like Ceph. The design utilizes a bipartite node-to-node topology managed by a new operator. The review feedback provides several technical improvements, including addressing MTU overhead for WireGuard, analyzing scalability limits of the N x M mesh, implementing fallback logic for node endpoints, using finalizers for robust resource cleanup, and expanding IP disjointness checks to include Service CIDRs.
|
|
||
| ### Topology | ||
|
|
||
| Both the host cluster and every participating tenant cluster run Kilo with `--mesh-granularity=cross`. In this mode every node is a topology segment of one. Within a single logical location (e.g. all nodes inside one cluster) traffic uses the underlying CNI without WireGuard. Across logical locations every node holds a direct WireGuard tunnel to every node in the other location. |
There was a problem hiding this comment.
The proposal should address MTU configuration for the cross-cluster mesh. Since WireGuard adds encapsulation overhead (typically 60-80 bytes), packets from pods using the default 1500 MTU will exceed the tunnel MTU, leading to fragmentation or packet loss. The design should specify how this will be handled, for example, by configuring the Kilo interface MTU and ensuring MSS clamping is active or by adjusting the CNI MTU in the tenant clusters.
|
|
||
| Both the host cluster and every participating tenant cluster run Kilo with `--mesh-granularity=cross`. In this mode every node is a topology segment of one. Within a single logical location (e.g. all nodes inside one cluster) traffic uses the underlying CNI without WireGuard. Across logical locations every node holds a direct WireGuard tunnel to every node in the other location. | ||
|
|
||
| For the host ↔ tenant pair, the result is a full bipartite mesh: every tenant node has a tunnel to every host node, and vice versa. The number of tunnels is `N × M` where N is the tenant node count and M is the host node count; this is intentional and is what enables the throughput and HA properties described below. |
There was a problem hiding this comment.
The N x M bipartite mesh topology may face scalability challenges as the number of nodes increases. For instance, a 100-node host cluster and a 100-node tenant cluster would result in 10,000 WireGuard peers per node. The proposal should include an analysis of the practical limits for the number of peers the kg-agent and the Linux kernel can manage before performance or control-plane stability is impacted.
| For each `TenantMeshLink`, the operator: | ||
|
|
||
| 1. Validates `spec.podCIDR` against all other `TenantMeshLink` objects and the host cluster's pod-CIDR; any overlap sets `PodCIDRConflict=True` and aborts further reconciliation for that tenant. | ||
| 2. Lists host cluster Nodes; for each node, ensures a `Peer` exists in the tenant cluster with: `publicKey` from the `kilo.squat.ai/wireguard-public-key` annotation, `endpoint` from `kilo.squat.ai/force-endpoint`, and `allowedIPs` containing the node's per-node pod-CIDR. |
There was a problem hiding this comment.
The operator should have a fallback strategy if the kilo.squat.ai/force-endpoint annotation is missing on a host node. Without a defined endpoint, tenant nodes will not be able to initiate the WireGuard handshake. Consider falling back to the Node's ExternalIP or InternalIP, or surfacing a specific error in the TenantMeshLink status.
Suggested change
| 2. Lists host cluster Nodes; for each node, ensures a `Peer` exists in the tenant cluster with: `publicKey` from the `kilo.squat.ai/wireguard-public-key` annotation, `endpoint` from `kilo.squat.ai/force-endpoint`, and `allowedIPs` containing the node's per-node pod-CIDR. | |
| 2. Lists host cluster Nodes; for each node, ensures a Peer exists in the tenant cluster with: publicKey from the kilo.squat.ai/wireguard-public-key annotation, endpoint from kilo.squat.ai/force-endpoint (falling back to Node IP if missing), and allowedIPs containing the node's per-node pod-CIDR. |
| 1. Validates `spec.podCIDR` against all other `TenantMeshLink` objects and the host cluster's pod-CIDR; any overlap sets `PodCIDRConflict=True` and aborts further reconciliation for that tenant. | ||
| 2. Lists host cluster Nodes; for each node, ensures a `Peer` exists in the tenant cluster with: `publicKey` from the `kilo.squat.ai/wireguard-public-key` annotation, `endpoint` from `kilo.squat.ai/force-endpoint`, and `allowedIPs` containing the node's per-node pod-CIDR. | ||
| 3. Lists tenant cluster Nodes; for each node, ensures a `Peer` exists in the host cluster with: `publicKey` from the tenant Node's annotation, `allowedIPs` containing the tenant per-node pod-CIDR, no `endpoint` (the tenant initiates). | ||
| 4. Removes orphaned Peer objects on either side using a label selector tied to the `TenantMeshLink` name. |
There was a problem hiding this comment.
To ensure that Peer objects created in the tenant cluster are reliably cleaned up when a TenantMeshLink is deleted, the operator should use Kubernetes finalizers. Without finalizers, if the operator is unavailable or the tenant API is unreachable during deletion, orphaned Peer resources will remain in the tenant cluster.
Suggested change
| 4. Removes orphaned Peer objects on either side using a label selector tied to the `TenantMeshLink` name. | |
| 4. Uses finalizers to ensure all remote Peer objects are removed from the tenant cluster before the TenantMeshLink is deleted.\n5. Removes orphaned Peer objects on either side using a label selector tied to the TenantMeshLink name. |
|
|
||
| The constraints on pod-CIDRs are: | ||
|
|
||
| - The host pod-CIDR and every tenant pod-CIDR must be pairwise disjoint. |
There was a problem hiding this comment.
The disjointness requirement should be extended to include the Service CIDRs of both clusters. Overlaps between a tenant's pod-CIDR and the host's Service CIDR (or vice versa) can cause routing conflicts, making it impossible for pods to reach internal services or the advertised host services.
Suggested change
| - The host pod-CIDR and every tenant pod-CIDR must be pairwise disjoint. | |
| - The host pod-CIDR, host service-CIDR, and every tenant pod-CIDR must be pairwise disjoint. |
Adjust the proposal to reflect that the controller will be developed as an independent project under the kilo-io organization, per confirmed interest from Kilo maintainer @squat. Generalize the CRD from a tenant-specific TenantMeshLink to a tenant-agnostic ClusterMesh that references peer clusters through a map of kubeconfig Secrets. Move all tenant semantics into a dedicated Cozystack integration section that also accounts for the kubernetes-nodes split (PR cozystack#8) so a single ClusterMesh covers multi-location, multi-backend tenants. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
…oller allowlist + RBAC monopoly Drop the planned admission webhook. Instead, harden the design with two controls owned by the host-cluster operator: - The controller is the only principal with write access to kilo.squat.ai/Peer in any participating cluster. Tenant-provisioning, the dashboard, and cluster admins can author ClusterMesh objects (intent) but never touch Peer directly. - The controller is configured at deploy time with a subnet allowlist (--allowed-cidr). Any ClusterMesh whose allowedNetworks fall outside that list is rejected with a status condition before any Peer is written. The allowlist cannot be widened through the ClusterMesh API. Collapse the per-cluster podCIDR + advertise fields into a single allowedNetworks list, since both are now validated against the same allowlist and can be expressed uniformly. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
…ontainment Make the WG-IP threat model explicit. A tenant root that tampers with a Node's kilo.squat.ai/wireguard-ip annotation must not be able to inject a Peer with attacker-chosen allowedIPs onto the host side. Add: - A second controller-level allowlist, --allowed-wireguard-cidr, that bounds where any kilo0 interface in the mesh may live. spec.clusters carries no WG-CIDR field; the WG address space is host-admin-owned infrastructure, not part of per-mesh data. - Per-Node validation alongside the existing mesh-level checks: WG-IP must be /32 (or /128), in --allowed-wireguard-cidr, and unique within its cluster. PodCIDRs must be in allowedNetworks. Failures skip the offending Node only; the mesh stays Ready. - A primary-boundary statement in Security: the host's exposure to a tenant peer is bounded exclusively by the host-side Peer.allowedIPs, so anything the tenant does to its own kilo0, routes, or kg-agent post-reconcile cannot widen that bound. - Cozystack integration spelled out for both allowlists: pod-pool to --allowed-cidr, WG-pool to --allowed-wireguard-cidr; tenant provisioning allocates from each. WG-IP is now restored to Peer.allowedIPs (standard Kilo Peer shape), since the new allowlist makes that safe and it brings cross-cluster diagnostics back. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
…Networks list Drop the second --allowed-wireguard-cidr allowlist. WG-CIDR is just another entry in the same allowedNetworks list as pod-CIDR and service-CIDR; per-Node WG-IP containment is validated against the cluster's own allowedNetworks rather than against a separate global pool. A tenant root cannot widen its surface to host pod/WG/service-CIDR because those CIDRs live in the host's allowedNetworks (a different spec.clusters entry), and per-Node containment rejects out-of-range annotations. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
|
|
||
| - Pods in any peer cluster can reach selected services in another peer cluster as if they were on the local network. (Cozystack use case: tenant pods reach host Ceph monitors, OSDs and MDS daemons.) | ||
| - Nodes added to or removed from any participating cluster are wired into / detached from the mesh automatically, without per-node manual configuration. | ||
| - A compromise of a peer cluster (up to and including full root on a peer node) cannot affect routing in another peer cluster beyond the network surface that was explicitly granted, and cannot affect unrelated peers. |
There was a problem hiding this comment.
unless it is the cluster running the controller, in which case I guess they do get perms on peer clusters, but that's not new
Member
Author
There was a problem hiding this comment.
Yes, intentional — that's the design. The controller cluster does have perms on remote Peers via the kubeconfigs it holds, and that's exactly the trust direction we want.
| # The controller's own cluster — no kubeconfig needed. | ||
| local: true | ||
| allowedNetworks: | ||
| - 10.4.0.0/16 # WG-CIDR |
There was a problem hiding this comment.
Should any of these be named fields in the struct rather than open fields in allowedNetworks? If the WireGuard mesh CIDR and the Pod CIDR are mandatory then maybe they get special treatment? Alternatively, using the open list can later be easily migrated into the stricter design
There was a problem hiding this comment.
I guess these are all technically optional and it just determines which networks from the Peer resources we want to honor / validate
Member
Author
Member
Author
There was a problem hiding this comment.
Right, exactly. The list is purely about what's permissible / honoured — not categorising CIDRs into kinds. That's why we kept it flat.
|
|
||
| **Mesh-level (halts reconciliation on failure):** | ||
|
|
||
| 1. Every CIDR in every `spec.clusters[*].allowedNetworks` is a subset of the controller's `--allowed-cidr` allowlist; otherwise `NetworksNotAllowed=True`. |
There was a problem hiding this comment.
This is kind of annoying, it means that there is functionally no difference between the cluster admin and the mesh admin. If you want to create a new mesh, then you have to edit the mesh controller deployment to add the allow list. I need to think about this a bit. What are we hoping to defend against here?
Member
Author
There was a problem hiding this comment.
Agreed, and dropped. The --allowed-cidr allowlist is gone in 8830a7c. The current design relies on RBAC on ClusterMesh as the address-surface chokepoint, and cluster-side defense-in-depth is now tracked in Alternatives as future Kilo work (your PeerClass suggestion below).
| namespace: kilo | ||
| spec: | ||
| clusters: | ||
| cluster-a: |
There was a problem hiding this comment.
Maybe in keeping with Kubernetes convention this should become a list of named structs, like how a Pod contains a list of named containers.
Member
Author
There was a problem hiding this comment.
Done in 8830a7c — spec.clusters is now a list of named entries (- name: cluster-a, ...).
| - **`--allowed-cidr` allowlist** bounds what `spec.clusters[*].allowedNetworks` can ever declare. Pod-CIDRs, WG-CIDRs, and service-CIDRs all flow through the same allowlist. A user who can author `ClusterMesh` objects cannot widen the address surface beyond what the host admin pre-approved. | ||
| - **Per-Node containment** validates that every observed annotation (`Node.Spec.PodCIDRs`, `kilo.squat.ai/wireguard-ip`) lies within the cluster's own `allowedNetworks`. A tenant root forging an annotation that points at the host pod-CIDR, host WG-CIDR, or any other CIDR the tenant did not declare itself is rejected — the offending Node is skipped and never appears as a Peer on the host side. | ||
| - **Trust direction by kubeconfig placement.** Whichever cluster holds the controller and the kubeconfig Secrets is the side that drives writes; the side whose kubeconfig is held cannot write back. In Cozystack, only the host holds tenant kubeconfigs — trust flows host → tenant. | ||
| - **Cross-mesh isolation.** Each `ClusterMesh`'s Peers are labelled with the mesh name; the controller never deletes or modifies Peers belonging to a different mesh, and `allowedNetworks` overlap between meshes (not just within a single mesh) is rejected. |
There was a problem hiding this comment.
We should probably also add labels for the source cluster name so if two controllers running on different hosts are managing meshes on the same tenant (some triangle where the two hosts don't know about each other) then the controllers are less likely to compete for ownership of Peers if the mesh object has the same name.
Member
Author
There was a problem hiding this comment.
Added in 8830a7c — every generated Peer now carries kilo-clustermesh.io/mesh: <mesh-name> and kilo-clustermesh.io/source-cluster: <cluster-name>, with the controller's ownership scoped to its own (mesh, source-cluster) pairs. Two controllers can coexist on the same remote cluster as long as they're not declaring the same source cluster.
| 2. **Cluster identifier scope**: should `spec.clusters` keys be free-form strings or follow a stricter schema (e.g. DNS-1123 labels) so they can be reused as label values? Likely the latter; to confirm during implementation. | ||
| 3. **Transitive routing**: with three or more clusters in the same `ClusterMesh`, the controller currently builds a full mesh. Should it support partial topologies (e.g. star)? Out of scope for v1; the CRD shape allows it later. | ||
| 4. **Multi-controller scenarios**: in a deployment where two clusters each run their own controller, how should they coordinate? Likely via a "leader" cluster identified in the CRD; deferred. | ||
| 5. **Per-peer opt-in for received CIDRs**: today `allowedNetworks` is a unilateral declaration on the source side, plus a global allowlist on the controller. Should there additionally be a per-peer `acceptedNetworks` field, so a peer can refuse to accept some of what another peer publishes? Likely unnecessary given the controller-level allowlist, but worth revisiting once there are multi-tenant deployments with heterogeneous policies. |
There was a problem hiding this comment.
The more I read about the controller allowlist, the more o actually started leaning in this direction. Maybe this needs to be a flag on Kilo, actually (or an entirely new PeerClass resource that declares what allowed IPs are permissible for every Peer in a cluster). This would allow individual clusters to guard against peers being created by a rogue cluster mesh controller. It's not blocking: this is orthogonal Kilo work that would be great to upstream to improve the administration of Kilo meshes.
Member
Author
There was a problem hiding this comment.
Agreed — tracked in Alternatives as future Kilo work in 8830a7c, with credit to you. Independent of this ClusterMesh proposal but a clear next direction for hardening Kilo administration.
Rename proposal to ClusterMesh, retitle, and add an explicit Scope section. Move the previously committed Cozystack integration into a deferred / exploratory section that lists potential patterns (routed pod-CIDR mesh, NAT egress, Service-mirror via Outline) without committing to any. Adjust rollout phases and tests to remove Cozystack-specific deliverables that belong to follow-up work. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
|
Two related scalability concerns that I think are missing from the proposal. Aggregate fan-out across tenantsThe scaling analysis covers a single ClusterMesh, but on the host side peers from all tenants accumulate on every node:
WireGuard kernel datapath is fine (O(1) AllowedIPs lookup), but the control plane is a different story:
The proposal needs a scalability section with realistic limits and guidance on when to shard tenants across host clusters. Full bipartite without topology awarenessEvery node in cluster A gets a tunnel to every node in cluster B. In heterogeneous clusters only a subset of nodes run the target workload - tunnels to irrelevant nodes carry zero useful traffic. A spec:
clusters:
- name: host
local: true
allowedNetworks: [...]
nodeSelector:
storage-role: cephIf 10 of 30 host nodes are selected, per-tenant peer count drops by 3x. Across 50 tenants: 750 → 250 peers per host node. Combined effectThese multiply: topology awareness reduces per-mesh fan-out, aggregate analysis tells you how many meshes you can sustain. Without both, there's no way to give operators capacity planning guidance. |
- spec.clusters: change from map keyed by cluster name to a list of named structs, in line with Kubernetes convention. - Drop the in-controller --allowed-cidr allowlist. It functionally collapsed cluster-admin and mesh-admin roles and overlapped with RBAC on ClusterMesh + per-Node containment. Defense story now relies on RBAC at the ClusterMesh creation point. - Add kilo-clustermesh.io/source-cluster label to every generated Peer, enabling co-existence of multiple controllers writing into the same remote cluster without ownership conflicts. - Add an Alternatives entry for a future Kilo-side defense-in-depth mechanism (cluster-level allowlist or PeerClass CRD), as suggested by @squat during review. - Move the flat-vs-typed-fields shape of allowedNetworks to Open Questions, noting the current flat list is intentionally narrow and easy to migrate later. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Arsolitt (Arsolitt)
added a commit
to cozystack/kilo-clustermesh-operator
that referenced
this pull request
May 18, 2026
Generic cluster-to-cluster mesh controller for Kilo that connects 2+ Kubernetes clusters into a flat node-to-node WireGuard mesh using Kilo's mesh-granularity=cross topology. Implements the design proposal from cozystack/community#7. Key components: - ClusterMesh CRD with named CIDR fields (podCIDRs, wireguardCIDR, serviceCIDR, additionalCIDRs) for precise validation - Two-layer validation: mesh-level CIDR overlap detection and per-node annotation validation (PodCIDR containment, WG-IP validity) - Multi-cluster client via controller-runtime pkg/cluster with graceful manager restart on cluster set changes - Peer builder creating per-node and anchor Peer CRDs with label-based ownership isolation between meshes - CRD self-install via embedded YAML applied on startup - Finalizer-based cleanup on ClusterMesh deletion - Helm chart with 38 helm-unittest assertions - Integration tests using dual envtest API servers - CI/CD with GitHub Actions (lint, test, integration, helm, release) Signed-off-by: Arsolitt <arsolitt@gmail.com>
Arsolitt (Arsolitt)
added a commit
to cozystack/kilo-clustermesh-operator
that referenced
this pull request
May 19, 2026
Add docs/known-gaps.md tracking outstanding work, divergences from the upstream proposal (cozystack/community#7), and settled design decisions that should not be re-litigated. Link from README documentation table and project-status note. Captures one blocker gap (no Node watches), one operational risk (silent anchor-peer suppression), and three proposal text corrections (annotation name, prefix rule, flat-vs-typed CRD shape). Signed-off-by: Arsolitt <arsolitt@gmail.com>
Arsolitt (Arsolitt)
added a commit
to cozystack/kilo-clustermesh-operator
that referenced
this pull request
May 20, 2026
Add docs/known-gaps.md tracking outstanding work, divergences from the upstream proposal (cozystack/community#7), and settled design decisions that should not be re-litigated. Link from README documentation table and project-status note. Captures one blocker gap (no Node watches), one operational risk (silent anchor-peer suppression), and three proposal text corrections (annotation name, prefix rule, flat-vs-typed CRD shape). Signed-off-by: Arsolitt <arsolitt@gmail.com>
7 tasks
IvanHunters
pushed a commit
to cozystack/kilo-clustermesh-operator
that referenced
this pull request
May 25, 2026
* fix(generate): align manifests target with repo layout
Restrict CRD generation to ./api/... so manifests no longer produces
config/crd/bases/kilo.squat.ai_peers.yaml for the external Kilo Peer
type. Regenerate deepcopy files so the boilerplate header from
hack/boilerplate.go.txt is committed, matching what controller-gen
produces in CI.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* fix(lint): exclude goconst from test files
Test files contain CIDRs, namespace names and similar fixtures repeated
across cases; promoting them to constants only obscures intent. Add
goconst to the linter exclusion list for _test\.go.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* fix(chart): drop unsupported --namespace and add --metrics-secure flag
The operator binary does not define a --namespace flag; passing it
caused the manager to exit at startup. Remove the argument from the
Deployment and the corresponding unit test.
The chart wired --metrics-bind-address=:8080 while the binary defaults
metrics-secure to true, which started HTTPS on an HTTP port and made
metrics unscrapeable without a TLS setup. Expose metricsSecure in
values.yaml (default false) and pass --metrics-secure explicitly.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* fix(ci): pass --verify=false to helm plugin install
Helm 4.1 requires plugin source verification by default, which the
helm-unittest source does not support. Without the flag the helm CI
job aborts with "plugin source does not support verification" before
helm lint and unit tests can run.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* feat(ci): publish container image to ghcr.io on push to main
Add an image job that builds the multi-arch Containerfile and pushes
to ghcr.io/cozystack/kilo-clustermesh-operator with :main and
:sha-<commit> tags. The job runs only on push to main and waits for
all checks (lint, test, integration, build, helm, generate) so a
broken commit cannot publish an image. Tagged releases are still
handled by .github/workflows/release.yml.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* fix(lint): extract Kilo group name to a constant
golangci-lint v2.12 flagged "kilo.squat.ai" repeated across register.go
and types_test.go via goconst. Define a GroupName constant in the Kilo
v1alpha1 package and use it from both call sites; this also matches
the convention used by upstream Kubernetes API packages.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* fix(lint): extract Kilo group version to a constant
golangci-lint v2.12 picked up another goconst occurrence:
the literal "v1alpha1" repeated across register.go and types_test.go.
Define a GroupVersion constant alongside GroupName and use it from
both call sites.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* ci: run CI and publish image on the dev branch
Add dev to the push/pull_request trigger lists and to the image job's
allow-list. Switch the image tag source from a fixed raw "main" to
type=ref,event=branch so pushes to dev publish :dev and pushes to
main publish :main. VERSION build-arg follows the ref name.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* perf(container): build Go binary on the native runner platform
Add --platform=\$BUILDPLATFORM to the builder stage so the Go
toolchain runs natively on the GitHub runner (amd64) while
cross-compiling for TARGETARCH through GOOS/GOARCH. QEMU emulation
is now only used for the final distroless stage, which is just a
file copy. ARM64 image builds drop from QEMU-emulated compilation
to native compile + cross-link.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* fix(operator): wire startup bootstrap and migrate to events API
The previous cmd/main.go constructed the manager with an unset Registry,
Log and Recorder on ClusterMeshReconciler, which would have caused a
nil pointer panic on the first reconcile. CRD self-install, despite
being available in internal/crd, was never invoked, so the manager
also failed to start with "no matches for kind ClusterMesh".
Startup now:
- runs crd.InstallOrUpdate before manager creation
- reads its namespace from POD_NAMESPACE (downward API)
- lists every ClusterMesh in that namespace, merges cluster entries by
name and builds a multicluster.ClusterRegistry up-front
- registers each cluster.Cluster with the manager so caches start
alongside the leader
- wires the reconciler with the real Registry, slog logger and the
manager's event recorder
- installs a change-watcher that cancels the manager context when
cluster fingerprints drift, triggering a kubelet-driven restart
The reconciler's Recorder field and the integration test suite are
moved off the deprecated record.EventRecorder API onto the events
package, fixing the staticcheck SA1019 warning.
go mod tidy removes seven now-unused indirect dependencies.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* fix(chart): allow SA token automount, inject POD_NAMESPACE and set seccomp RuntimeDefault
Three blocking issues for running the operator on a cluster with
PodSecurity restricted enforced:
- ServiceAccount.automountServiceAccountToken: false stopped the
controller from authenticating to the in-cluster API. Removed; the
operator needs the projected token.
- The container template did not request POD_NAMESPACE, which the new
startup bootstrap reads to scope its ClusterMesh and Secret lookups.
Injected via downward API.
- securityContext.seccompProfile was unset, triggering a PodSecurity
warning under restricted:latest. Set to RuntimeDefault.
helm-unittest suites are updated accordingly. All 41 tests pass.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* fix(operator): scope manager cache to operator namespace
The controller-runtime manager cache defaulted to cluster-scope list/watch
on ClusterMesh and Secret, which required cluster-wide RBAC the chart
deliberately does not grant. The operator only watches namespaced
resources in its own namespace; cluster-scoped resources (Peers, Nodes,
CRDs, Leases) are accessed via the multicluster registry or direct API
calls and are unaffected.
Restrict Cache.DefaultNamespaces to the operator's own namespace so the
existing namespace-scoped Role is sufficient.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* fix(operator): accept cozystack-Kilo wireguard-ip annotation format
Upstream Kilo writes the kilo.squat.ai/wireguard-ip annotation as a host
route ("10.4.0.1/32"); the cozystack-patched Kilo with cross granularity
writes the host IP with the wireguard subnet mask ("100.66.0.3/16"). The
operator previously required /32 (or /128) and reused the raw annotation
value in the Peer AllowedIPs — both of which fail under cozystack-Kilo:
validation skips every node, and even if it did not, every peer would
claim the entire wireguard subnet.
Add a netutil helper that parses an annotation preserving host bits, and
another that emits the host route for a given IP. Validation now checks
only that the host IP falls inside the cluster's wireguardCIDR. The peer
builder normalises AllowedIPs to a /32 (resp. /128) host route so each
peer terminates traffic for exactly one WireGuard IP.
Covered by unit tests for both upstream-style and cozystack-style
annotations.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* fix(chart): grant operator RBAC on events.k8s.io events
The migration to k8s.io/client-go/tools/events.EventRecorder caused
events to be written to the events.k8s.io/v1 API group instead of the
legacy core "events" resource. The Role only granted access to the
latter, so every recorded event was rejected by the apiserver with
events.events.k8s.io is forbidden.
Grant create/patch on events.k8s.io/events alongside the existing rule
for the core group; keep both so the operator stays compatible if any
component falls back to the legacy API.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* docs(readme): document force-endpoint annotation, manual reconcile and roadmap
The operator only sets a Peer endpoint from kilo.squat.ai/force-endpoint;
clusters where nodes are not publicly reachable on that annotation will
end up with peers that have no endpoint and silently drop traffic. The
manager cache also does not watch Node objects, so annotation changes
require a no-op write on the ClusterMesh resource to trigger a reconcile.
Capture both of these in the README so users hit fewer surprises.
Also refresh the surrounding documentation:
- tighten Prerequisites to distinguish apiserver reachability from
per-node UDP reachability;
- replace the broken Remote Cluster Setup snippet with the working
Secret-backed long-lived token procedure that survives 1.24+;
- correct the Architecture section, which previously claimed the
controller watches Node objects;
- add a Possible Improvements list covering a dedicated cross-cluster
endpoint annotation, a Node watcher, and an explicit per-node skip
annotation.
Stop tracking local cluster-specific deployment manifests under deploy/
by adding it to .gitignore; the upstream README now contains the
generic procedure, so the per-cluster files belong in a private branch.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* docs(kilonode): document both accepted wireguard-ip annotation formats
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* fix(restart): guard against nil Cancel in ChangeWatcher.Reconcile
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* test(main): add unit tests for mergeClusterSpecs deduplication
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* fix(containerfile): correct image.source label to cozystack repo
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* test(peer): cover IPv6 and DNS endpoint parsing
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* feat(main): declare version/revision build vars and log at startup
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* fix(chart): default image.repository to cozystack fork
The previous default pointed at the upstream squat image, causing
helm install without overrides to pull the wrong image. Switch the
default to the cozystack fork, add a helm-unittest assertion that
verifies the default, and remove the now-redundant repository override
from the README tag-pinning example.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* fix(validation): normalize wireguard-ip before duplicate detection
Two nodes with "10.4.0.1/16" and "10.4.0.1/32" resolve to the same
WireGuard peer (AllowedIPs = 10.4.0.1/32) and therefore conflict.
The old comparison was a raw-string equality check, so it missed this
case.
Extract the host IP via netutil.ParseHostInCIDR and use that as the
dedup key. Invalid annotations fall back to raw-string keying so that
identical-invalid values still deduplicate without colliding with any
valid IP.
Add three new test cases that cover: same host IP / different prefix
lengths, different host IPs (sanity), and invalid vs valid annotation.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* ci(workflows): include cmd package in unit test job
TestMergeClusterSpecs in cmd/main_test.go was never executed in CI
because the test step listed explicit package paths that omitted
./cmd/.... Add ./cmd/... to the go test invocation so all unit tests
run on every push.
Add internal/citest/workflow_test.go to structurally assert the
presence of ./cmd/... in the ci.yml test step, preventing future
accidental regressions.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* fix(peer): strip brackets from DNS fallback in endpoint parser
buildDNSOrIP strips brackets from the host before calling net.ParseIP,
but the DNS fallback path was returning the unstripped host variable.
A bracketed DNS name like [dns.example.com]:51820 would produce
DNS: "[dns.example.com]", which is an invalid hostname.
Change DNS: host to DNS: cleanHost so the brackets-stripped form is
always used for the DNS field.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* docs(api): correct WireguardCIDR comment about accepted annotation formats
The previous comment stated that wireguard-ip annotations must be a /32
(or /128) host route. This was accurate before the IsHostRoute validation
check was dropped, but has been incorrect since that change. The operator
now accepts any prefix length and validates only the host portion of the
address against WireguardCIDR. Update the Go doc comment and the
generated CRD YAML description to match actual behavior.
Also annotate the existing regression test case to document that it
guards against inadvertent reintroduction of the /32-only requirement.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* test(validation): cover ValidateClusterNetworks and ValidateMeshNetworks
Add table-driven unit tests for both public functions in
internal/validation/mesh.go, which previously had 0% coverage in CI
(integration tests were excluded from the unit test job).
TestValidateClusterNetworks covers:
- empty cluster list (nil error)
- single cluster, multiple disjoint CIDRs (nil error)
- single cluster with serviceCIDR overlapping additionalCIDR (error)
- two clusters with disjoint CIDRs (nil error)
- two clusters with overlapping serviceCIDR (error naming both clusters)
- two clusters with overlapping podCIDR (error naming both clusters)
- invalid CIDR string triggering a parse error
TestValidateMeshNetworks covers:
- empty mesh list (nil error)
- single mesh with valid clusters (nil error)
- two meshes with disjoint network plans (nil error)
- two meshes with an overlapping CIDR (error naming both meshes)
- intra-mesh overlap detected before cross-mesh check
Helpers makeCluster and makeMesh follow the makeNode convention from
node_test.go.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* docs(readme): use non-overlapping CIDRs in Quick Start example
The original example set identical serviceCIDR (10.96.0.0/12) on both
clusters, and both wireguardCIDRs (10.100.x.0/24) fell inside that /12
range — all three overlaps would have caused ValidateClusterNetworks to
return an error and suppress all Peer creation.
Fix by assigning distinct, non-overlapping ranges to every field:
- cluster-a wireguardCIDR: 10.200.0.0/24
- cluster-b wireguardCIDR: 10.200.1.0/24
- cluster-b serviceCIDR: 10.112.0.0/12
Add TestREADMEQuickStartManifestIsValid in internal/validation/mesh_test.go
as a regression guard: it reads README.md, extracts the ClusterMesh YAML
block, unmarshals it, and asserts ValidateClusterNetworks returns nil.
The test fails with the original CIDRs and passes after this fix.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* docs(peer): clarify allowedIPs derivation in BuildPeer comment
The previous comment said "wireguard-ip annotation" without explaining
that only the host IP is extracted and then normalised to a /32 (IPv4)
or /128 (IPv6) host route. The updated comment makes the normalisation
step explicit, which is important context given that the annotation may
carry a wider subnet mask in cozystack-patched Kilo.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* feat(api): add wireguardPort field to ClusterEntry
Introduces an optional WireguardPort field on ClusterEntry, defaulting to
51820. The operator uses this port when synthesising a peer endpoint from
Node.Status.Addresses (i.e. neither clustermesh-endpoint nor force-endpoint
annotation is set on the node).
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* feat(annotations): add clustermesh-endpoint node annotation
Adds AnnotationClustermeshEndpoint constant for an operator-specific
node annotation that conveys the cross-cluster WireGuard endpoint
independently of Kilo's own kilo.squat.ai/force-endpoint. Decoupling
the two prevents the operator's endpoint configuration from affecting
intra-cluster Kilo behaviour (notably the "cross" mesh granularity).
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* test(kilonode): add tests for endpoint resolution chain (Red)
12 tests covering ResolveEndpoint priority order, ExternalIP fallback
(IPv4/IPv6 preference, default port), strict error on malformed annotations,
and ignoring non-ExternalIP address types.
Tests intentionally fail to compile until ResolveEndpoint is implemented.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* feat(kilonode): resolve node endpoint via fallback chain
Implements ResolveEndpoint which determines a node's WireGuard endpoint
via a three-tier priority chain: the operator-specific
kilo.squat.ai/clustermesh-endpoint annotation wins; otherwise the legacy
kilo.squat.ai/force-endpoint annotation; otherwise the first ExternalIP
in Node.Status.Addresses (IPv4 preferred over IPv6) combined with the
fallback port.
A present-but-malformed annotation is a hard error rather than a silent
fall-through, so misconfiguration surfaces immediately instead of
yielding peers without an endpoint.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* refactor(peer): pass ClusterEntry through Build{Peer,AnchorPeer}
Replaces the (meshName, sourceCluster string) and
(meshName, sourceCluster, entry) signatures with a unified
(meshName string, entry *ClusterEntry) shape. The entry carries both
the cluster name and the new WireguardPort field, removing the need to
plumb individual fields through call sites and preparing the builders
to consume kilonode.ResolveEndpoint in the next step.
Existing behaviour is preserved; tests pass without modification beyond
the signature change.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* feat(peer): wire endpoint resolution chain into peer builders
BuildPeer and BuildAnchorPeer now consume kilonode.ResolveEndpoint
instead of reading kilo.squat.ai/force-endpoint directly. The peer
endpoint is therefore picked up from the clustermesh-endpoint
annotation, the legacy force-endpoint annotation, or Node.Status
ExternalIPs in that priority order.
Two behaviour changes flow from this:
- A node with no endpoint source no longer produces a Peer without an
endpoint silently — BuildPeer now returns an error and
BuildAnchorPeer returns nil. Validation should already filter such
nodes; the new behaviour is a defensive surface for missed cases.
- A present-but-malformed endpoint annotation (clustermesh-endpoint or
force-endpoint) is a hard error, not a silent fall-through. The
affected unit test was rewritten accordingly.
baseAnnotations() in builder_test.go now includes a valid force-endpoint
so existing tests succeed by default; tests that exercise specific
chain layers override or delete the relevant key.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* test(validation): add endpoint resolution skip cases (Red)
Adds four ValidateNode test cases that the current implementation does
not satisfy yet: a node missing every endpoint source must be skipped
with ReasonNoEndpoint, a malformed clustermesh-endpoint or
force-endpoint annotation must be skipped with ReasonEndpointInvalid,
and a node whose only endpoint source is an ExternalIP must be accepted.
The pre-existing baseAnnotations() now includes a valid force-endpoint
so that all current "valid node" cases continue to pass once endpoint
validation lands.
Tests fail at compile time because ReasonNoEndpoint and
ReasonEndpointInvalid do not exist yet; the Green commit will add them
alongside the validateEndpoint helper.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* feat(validation): require resolvable endpoint via kilonode.ResolveEndpoint
ValidateNode now exercises the endpoint fallback chain after the
WireGuard IP and public key checks. A node with no source is skipped
with ReasonNoEndpoint; a node whose clustermesh-endpoint or
force-endpoint annotation is present but malformed is skipped with
ReasonEndpointInvalid.
Filtering out such nodes here prevents BuildPeer from later failing
on a per-node basis and surfaces the misconfiguration as a clear
SkippedNodes entry in ClusterMesh status.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* chore(crd): regenerate manifests after wireguardPort addition
Regenerates the ClusterMesh CRD via controller-gen to expose the new
wireguardPort field with default 51820 and bounds [1, 65535], and
mirrors the result into the embedded copy at internal/crd/ which the
operator applies on startup.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* docs(readme): document endpoint resolution chain
Updates the per-node configuration and reconciliation-flow sections to
describe the new three-tier endpoint chain
(kilo.squat.ai/clustermesh-endpoint → kilo.squat.ai/force-endpoint →
first ExternalIP), the strict treatment of malformed annotations
(NodeEndpointInvalid skip reason), and the per-cluster wireguardPort
field used by the ExternalIP fallback.
Removes the now-implemented "dedicated cross-cluster endpoint
annotation" item from the roadmap and updates the prerequisites note
about the WireGuard UDP port to reflect that it is configurable.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* docs: rewrite README and add docs/ tree
Restructure README as a landing page (overview, requirements, quick
start) and split deep content into a flat English-only docs/ tree:
- docs/architecture.md — components, reconciliation flow, anchor
peer, manager cache scoping, CRD bootstrap,
restart watcher
- docs/installation.md — Helm install, embedded CRD bootstrap,
remote-cluster kubeconfig setup, RBAC,
verification, uninstall
- docs/configuration.md — ClusterMesh CRD reference: Spec, Status,
ClusterEntry fields, conditions, CIDR
validation rules
- docs/per-node-setup.md — required Node annotations, three-tier
endpoint resolution chain, strict-invalid
semantics, migration from force-endpoint
- docs/troubleshooting.md — full NodeSkipReason table, mesh-level
validation errors, status conditions,
common pitfalls
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* docs: add known-gaps handoff document
Add docs/known-gaps.md tracking outstanding work, divergences from the
upstream proposal (cozystack/community#7), and settled design decisions
that should not be re-litigated. Link from README documentation table
and project-status note.
Captures one blocker gap (no Node watches), one operational risk
(silent anchor-peer suppression), and three proposal text corrections
(annotation name, prefix rule, flat-vs-typed CRD shape).
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* test(integration): provide resolvable endpoint for nodes
The endpoint resolution chain introduced in f8e0ab2 now rejects nodes that
have neither a clustermesh-endpoint, a force-endpoint, nor an ExternalIP.
The existing fixtures created nodes with no endpoint source at all, so
ValidateNode skipped them with NodeNoEndpoint and the integration tests
that asserted on resulting peer counts started failing in CI.
Attach a default ExternalIP to every node built by makeNode so that the
ExternalIP fallback succeeds, and persist Status.Addresses via the status
subresource (Create does not). The TestHappyPath endpoint assertion is
tightened to target the peer for the node that carries the explicit
force-endpoint, since other peers now legitimately resolve via the
ExternalIP fallback.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* ci(workflows): disable checkout credential persistence
Add persist-credentials: false to every actions/checkout step so the
GITHUB_TOKEN is not retained in the workspace after the action exits.
This is defense-in-depth: jobs that need authenticated access (image
publish) authenticate explicitly via docker/login-action, and the
remaining jobs are read-only consumers of the checked-out tree.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* docs(installation): set yaml language on RBAC fenced block
Address markdownlint MD040 on the ClusterRole snippet.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* docs(installation): strip prompt prefixes from no-output examples
Remove the leading "$ " from single-command shell examples that show
no command output. Addresses markdownlint MD014.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* docs(per-node): set text language on plain fenced blocks
Add text language identifier to the four plain code fences (priority
list, host:port template, IPv6 example, bracketed DNS example).
Addresses markdownlint MD040.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
* docs(readme): merge adjacent blockquote callouts
Replace the blank separator between the CIDR-overlap warning and the
CRD self-install note with a quoted blank line so both callouts live
inside one continuous blockquote. Addresses markdownlint MD028.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
---------
Signed-off-by: Arsolitt <arsolitt@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a design proposal for cross-cluster connectivity between Cozystack-managed tenant clusters and the host cluster.
The motivating use case: a host cluster running Ceph (managed by Rook) that should be reachable from inside tenant clusters as if it were local storage. Standard single-gateway approaches (Submariner, Kilo's default
mesh-granularity=location) bottleneck Ceph traffic; this proposal uses Kilo'smesh-granularity=cross(squat/kilo#328) to build a node-to-node mesh that scales linearly with cluster size and handles Rook-driven failover without controller intervention on the data path.The proposal covers:
cozystack-meshlink-operator) andTenantMeshLinkCRD for managing Peer objects on both sidesLooking for feedback on the open questions, especially the upstream Kilo PR #328 strategy and whether tenant-side Kilo should be a hard requirement.
Test plan
This is a design proposal; no code yet. Implementation testing is scoped in the proposal and will follow in implementation PRs: