|
| 1 | +--- |
| 2 | +date: 2026-04-10 |
| 3 | +title: "Building the Kubernetes Multi-Cluster Plugin for PipeCD — LFX Mentorship" |
| 4 | +linkTitle: "Building the Kubernetes Multi-Cluster Plugin for PipeCD" |
| 5 | +weight: 970 |
| 6 | +author: Mohammed Firdous ([@mohammedfirdouss](https://github.com/mohammedfirdouss)) |
| 7 | +categories: ["Contribution"] |
| 8 | +tags: ["Kubernetes", "Plugin", "LFX Mentorship"] |
| 9 | +--- |
| 10 | + |
| 11 | +If you had told me last year that I would be working with Kubernetes and all things clusters, deployments and service meshes, I would have brushed it off. I am truly grateful for the journey thus far. |
| 12 | + |
| 13 | +Earlier last month, I got accepted as an LFX Mentee for Term 1 of this calendar year. For me it is such a big deal, given my background, and how much effort has been put in behind the scenes to get to this stage. |
| 14 | + |
| 15 | +I'm currently a mentee in the LFX Mentorship program working on [PipeCD](https://pipecd.dev), an open-source GitOps continuous delivery platform. For the past four weeks, I've been building out the `kubernetes_multicluster` plugin specifically implementing the deployment pipeline stages that handle canary, primary and baseline deployments across multiple clusters. |
| 16 | + |
| 17 | +--- |
| 18 | + |
| 19 | +## What is PipeCD and what is this plugin? |
| 20 | + |
| 21 | +PipeCD is an open-source GitOps CD platform that manages deployments across different infrastructure targets like Kubernetes, ECS, Terraform, Lambda and more. Each target type has a plugin that knows how to deploy to it. |
| 22 | + |
| 23 | +The `kubernetes_multicluster` plugin is for teams running the same application across multiple Kubernetes clusters say US, EU and Asia and needing all of them to stay in sync through a single pipeline. Rolling out a new version across clusters one at a time, manually, with no coordination, is error-prone and slow. The plugin lets you define one pipeline that runs across every cluster at the same time, with canary and baseline checks before anything hits production. |
| 24 | + |
| 25 | +## Progressive Delivery and Why These Stages Exist |
| 26 | + |
| 27 | +Before a new version reaches all users, it goes through stages. A canary sends a small slice of traffic to the new version first. A baseline runs the *current* version at the same scale so you have a fair comparison. Primary is the actual promotion. Clean stages remove the temporary resources when you're done. |
| 28 | + |
| 29 | +This pattern is called progressive delivery, because you roll out gradually, check things look good, then commit. If something looks wrong at the canary stage, you stop there. Nothing has touched production yet. |
| 30 | + |
| 31 | +The `kubernetes_multicluster` plugin runs all of this across every cluster at the same time. One pipeline, every cluster, same stages. |
| 32 | + |
| 33 | +A full pipeline looks like this: |
| 34 | + |
| 35 | +```yaml |
| 36 | +stages: |
| 37 | + - name: K8S_CANARY_ROLLOUT |
| 38 | + - name: K8S_BASELINE_ROLLOUT |
| 39 | + - name: K8S_TRAFFIC_ROUTING |
| 40 | + - name: K8S_PRIMARY_ROLLOUT |
| 41 | + - name: K8S_CANARY_CLEAN |
| 42 | + - name: K8S_BASELINE_CLEAN |
| 43 | +``` |
| 44 | +
|
| 45 | +Each of these is a stage I built. The sections below go through what each one does. |
| 46 | +
|
| 47 | +## What I Built |
| 48 | +
|
| 49 | +### K8S_CANARY_ROLLOUT |
| 50 | +
|
| 51 | +The canary stage deploys the new version of your app as a small slice alongside the existing production deployment. If your app normally runs 3 pods, canary might spin up 1 pod (or 20%) of the new version enough to catch problems without affecting most users. |
| 52 | +
|
| 53 | +It loads manifests from Git, creates copies of all workloads with a `-canary` suffix, scales them down to the configured replica count, adds a `pipecd.dev/variant=canary` label, and applies them to every target cluster in parallel. The original deployment is never touched this stage only ever adds resources. |
| 54 | + |
| 55 | + |
| 56 | + |
| 57 | + |
| 58 | + |
| 59 | +--- |
| 60 | + |
| 61 | +### K8S_CANARY_CLEAN |
| 62 | + |
| 63 | +Once the canary window is over, whether you promoted or rolled back, the canary pods are just sitting in every cluster doing nothing. `K8S_CANARY_CLEAN` removes them. |
| 64 | + |
| 65 | +It finds all resources with the label `pipecd.dev/variant=canary` for the application and deletes them in order: Services first, then Deployments, then everything else. The order matters as you don't want to remove the Deployment while the Service is still sending traffic to it. |
| 66 | + |
| 67 | +One thing worth noting: the query is scoped strictly to canary-labelled resources. Even if something goes wrong in the deletion logic, it cannot touch primary resources. |
| 68 | + |
| 69 | + |
| 70 | + |
| 71 | + |
| 72 | + |
| 73 | +--- |
| 74 | + |
| 75 | +### K8S_PRIMARY_ROLLOUT |
| 76 | + |
| 77 | +After the canary looks good, you promote the new version to primary, the workload actually serving all your users. This stage takes the manifests from Git, adds the `pipecd.dev/variant=primary` label, and applies them across all clusters in parallel. |
| 78 | + |
| 79 | +It also has a `prune` option: after applying, it checks what's currently running in the cluster against what was just applied, and deletes anything that's no longer in Git. Useful when you remove a resource from your manifests and want the cluster to reflect that. |
| 80 | + |
| 81 | + |
| 82 | + |
| 83 | + |
| 84 | + |
| 85 | +--- |
| 86 | + |
| 87 | +### K8S_BASELINE_ROLLOUT |
| 88 | + |
| 89 | +This one took me a while to understand and it is the stage I find most interesting to explain as well. |
| 90 | + |
| 91 | +When you're running a canary, the natural thing is to compare it against primary. The issue is that's not a fair comparison primary is handling far more traffic than canary, under different conditions. |
| 92 | + |
| 93 | +Baseline gives you a fairer comparison. You take the *current* version (not the new one) and run it at the same scale as canary. Now your cluster has: |
| 94 | + |
| 95 | +```plaintext |
| 96 | +simple 2/2 ← production, current version |
| 97 | +simple-canary 1/1 ← new version, being tested |
| 98 | +simple-baseline 1/1 ← current version at canary scale |
| 99 | +``` |
| 100 | + |
| 101 | +You compare canary vs baseline, same number of pods, same traffic conditions. If canary is worse, it's obvious. |
| 102 | + |
| 103 | +The key difference from every other rollout stage is one line of code. Canary and primary load manifests from the new Git commit (`TargetDeploymentSource`). Baseline loads from what's currently running (`RunningDeploymentSource`): |
| 104 | + |
| 105 | +```go |
| 106 | +// canary.go — new version |
| 107 | +manifests, err := p.loadManifests(ctx, ..., &input.Request.TargetDeploymentSource, ...) |
| 108 | +
|
| 109 | +// baseline.go — current version |
| 110 | +manifests, err := p.loadManifests(ctx, ..., &input.Request.RunningDeploymentSource, ...) |
| 111 | +``` |
| 112 | + |
| 113 | + |
| 114 | + |
| 115 | + |
| 116 | + |
| 117 | + |
| 118 | + |
| 119 | + |
| 120 | + |
| 121 | +--- |
| 122 | + |
| 123 | +### K8S_BASELINE_CLEAN |
| 124 | + |
| 125 | +Once the analysis is done, baseline resources get cleaned up the same way as canary find everything labelled `pipecd.dev/variant=baseline` and delete it in order. No configuration needed. It doesn't matter whether `createService: true` was set during rollout, it finds whatever is there and removes it. |
| 126 | + |
| 127 | + |
| 128 | + |
| 129 | + |
| 130 | + |
| 131 | + |
| 132 | + |
| 133 | + |
| 134 | + |
| 135 | + |
| 136 | + |
| 137 | +--- |
| 138 | + |
| 139 | +### K8S_TRAFFIC_ROUTING |
| 140 | + |
| 141 | +Canary and baseline pods exist in the cluster but get no traffic until this stage runs. Without it, you're analysing pods that nobody is actually hitting. This stage is what sends real user traffic to them. |
| 142 | + |
| 143 | +Two methods are supported: |
| 144 | + |
| 145 | +**PodSelector** (no service mesh needed): changes the Kubernetes Service selector to point at one variant. All-or-nothing 100% to canary or 100% back to primary. |
| 146 | + |
| 147 | + |
| 148 | + |
| 149 | + |
| 150 | + |
| 151 | + |
| 152 | + |
| 153 | + |
| 154 | + |
| 155 | +**Istio**: updates VirtualService route weights to split traffic across all three variants at once for example, primary 80%, canary 10%, baseline 10%. Also supports `editableRoutes` to limit which named routes the stage is allowed to modify. |
| 156 | + |
| 157 | +One small thing I added on top of the traffic routing stage: per-route logging. When the stage runs, it now logs each route it processes whether it was skipped (because it's not in `editableRoutes`) or updated with new weights. Before this, the log just said "Successfully updated traffic routing" with no detail. Now you can see exactly which routes changed and to what percentages, which is useful when debugging a misconfigured VirtualService. |
| 158 | + |
| 159 | + |
| 160 | + |
| 161 | + |
| 162 | + |
| 163 | + |
| 164 | + |
| 165 | + |
| 166 | + |
| 167 | + |
| 168 | + |
| 169 | +--- |
| 170 | + |
| 171 | +## Something I Found Interesting |
| 172 | + |
| 173 | +The thing that surprised me was how `errgroup` handles running across multiple clusters without much extra code. |
| 174 | + |
| 175 | +Every stage needs to run against N clusters, not one. A simple for-loop would run them one at a time slow, and if cluster 2 fails you don't find out until cluster 1 is already done. |
| 176 | + |
| 177 | +`errgroup` runs all clusters at the same time and returns the first error: |
| 178 | + |
| 179 | +```go |
| 180 | +eg, ctx := errgroup.WithContext(ctx) |
| 181 | +for _, tc := range targetClusters { |
| 182 | + tc := tc |
| 183 | + eg.Go(func() error { |
| 184 | + return canaryRollout(ctx, tc.deployTarget, ...) |
| 185 | + }) |
| 186 | +} |
| 187 | +return eg.Wait() |
| 188 | +``` |
| 189 | + |
| 190 | +All clusters run in parallel. If any one fails, the stage fails immediately. The same pattern is used across every stage, so adding a new stage is mostly just writing the per-cluster logic the concurrency part is already solved. |
| 191 | + |
| 192 | +## What's Next |
| 193 | + |
| 194 | +The next piece is `DetermineStrategy`, that is the logic that decides what kind of deployment to trigger based on what changed in Git. After that, livestate drift detection so PipeCD can flag when a cluster has drifted from what Git says it should be. |
| 195 | + |
| 196 | +To get involved, check out the PipeCD project and come join us on Slack. |
| 197 | + |
| 198 | +## Links |
| 199 | + |
| 200 | +- [PipeCD repository](https://github.com/pipe-cd/pipecd) |
| 201 | +- [LFX Mentorship Program](https://mentorship.lfx.linuxfoundation.org) |
| 202 | +- [Issue #6446, kubernetes_multicluster plugin](https://github.com/pipe-cd/pipecd/issues/6446) |
| 203 | +- [PR #6629 K8S_TRAFFIC_ROUTING](https://github.com/pipe-cd/pipecd/pull/6629) |
| 204 | +- [PR #6648 Per-route logging in K8S_TRAFFIC_ROUTING](https://github.com/pipe-cd/pipecd/pull/6648) |
| 205 | +- [Slack #PipeCD](https://app.slack.com/client/T08PSQ7BQ/C01B27F9T0X) |
0 commit comments