Skip to content

CFP-45105 split cilium agent daemonset #91

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mrkiani98 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

CFP-45105 split cilium agent daemonset #91

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions cilium/CFP-45105-split-cilium-agent-daemonset.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# CFP-003: Template

**SIG: SIG-NAME** ([View all current SIGs](https://docs.cilium.io/en/stable/community/community/#all-sigs))

**Begin Design Discussion:** 2026-04-07

**Cilium Release:** 1.19.2

**Authors:** Mohammadreza Kiani (mohammadrezakiani.shojaee@gmail.com)

**Status:** Implementable

## Summary

Ability to split cilium agent daemonset into multiple daemonsets per different node pools.
* NOTE: We can work on implementing this feature on the helm chart by opening respective PRs, but first, we want to know the maintainers' and also the community's opinion about the possible different options which we will explain here.

## Motivation

For better resource optimization in case of different usage pattern in different node pools based on nodes' sizes and workloads.

@HadrienPatte HadrienPatte Apr 16, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the target objective here is essentially to be able to have different resource requests/limits based on node size, this is a problem we've faced as well and we have solved it with a different solution from the two presented in this CFP (we use a patched version of the cilium chart). Sharing our implementation in case it can be of interest.

Our solution is essentially a single cilium chart with multiple dynamic agent daemonsets.

In our values we define multiple different resources for daemonset "profiles":

 agent:
    profiles:
      - name: default
        resources:
          limits:   { cpu: 90m, memory: 2Gi }
          requests: { cpu: 90m, memory: 2Gi }
      - name: small
        resources:
          limits:   { cpu: 90m, memory: 512Mi }
          requests: { cpu: 90m, memory: 512Mi }
      - name: medium
        resources:
          limits:   { cpu: 90m, memory: 1.5Gi }
          requests: { cpu: 90m, memory: 1.5Gi }
      - name: large
        resources:
          limits:   { cpu: 90m, memory: 2Gi }
          requests: { cpu: 90m, memory: 2Gi }

Then in the daemonset template, we loop over those profiles with range:

{{ range $profile := .Values.agent.profiles }}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  # "default" profile keeps the original name for backward compatibility
  name: cilium-agent{{ if ne $profile.name "default" }}-{{ $profile.name }}{{ end }}
spec:
  ...

And we update the daemonset node affinity to make sure no node ends up with two agents and every node does get an agent:

 affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              {{ if eq $profile.name "default" }}
              # Catch-all: matches nodes that have no profile label at all
              - key: "cilium.io/agent-profile"
                operator: DoesNotExist
              {{ else }}
              - key: "cilium.io/agent-profile"
                operator: In
                values:
                  - {{ $profile.name }}
              {{ end }}

And set the resources according to the profile:

resources: {{ toJson $profile.resources }}

The main advantage of this design is that it's highly flexible:

  • configure nodes to be labeled at provisioning with the appropriate label to opt-in to a sized profile (for example cilium.io/agent-profile: small)
  • you can configure as many different profiles as you want and need
  • if you don't set any custom profile, you just end up with the default one and everything renders as it does today, so no change for people not needing this advanced setup
  • single chart solution: no duplicate chart, no duplicate RBAC resources

@mrkiani98 mrkiani98 Apr 21, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the details. Actually for now, we also have a kind of similar approach with some custom in place implementation to have the multiple agent daemonsets with different resources set. But in general, the goal of this feature request is to make the chart dynamic enough to support this implementation without any custom patched setup.


## Goals

The goal here is to make the chart flexible to let us deploy one operator alongside with multiple agent daemonsets per nodepool.

## Proposal

### Overview

While using the community cilium helm chart to deploy the operator and agent, we faced some issues as we cannot split the agent daemonset into multiple daemonsets via separate releases in the same cluster because of some naming conflicts of some RBAC resources defined for the agent.

@squeed squeed Apr 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not entirely sure why this is desirable, but would separate namespaces be the solution?

@mrkiani98 mrkiani98 Apr 15, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Separate namespaces still cannot be the solution as the name of the cluster role and cluster role binding is hardcoded and the same issue will happen again.


The list of the problematic resources is as following:

- ClusterRole/cilium
- ClusterRoleBinding/cilium
- Role/cilium-config-agent
- RoleBinding/cilium-config-agent

There are two options as following which both of them are possible to implement and there is no clear trade off to decide which one is better and that's why we will put both options here to first decide which one is a better go.

### Option 1:

Add the ability to enable/disable all related RBAC resources, so that we can have one release which will contain operator and all RBAC resources which will be shared via other multiple agent daemonset releases (in agent releases, the RBAC resources would be disabled).

#### Pros

- No duplicate RBAC resources.
- Easier management of the resources.
- Easier implementation and maintenance in the chart.

#### Cons

- No separation and isolation for the separated agent daemonsets RBAC resources.
- Having a kind of dependency of the shared RBAC resources in all agent daemonset releases on the operator release.

### Option 2:

Make the RBAC resources naming dynamic (generated via the release name via a helper function), so that each daemonset agent will have its own RBAC resources.

#### Pros

- Better separation and isolation for separated agent daemonsets RBAC resources.

#### Cons

- Duplicate RBAC resources.
- More complex management and maintenance.
- More difficult implementation in the chart.