Ownership conflict errors with cert-manager Secrets in Cryostat operator 3.0.0 using custom Helm deployment

[Prasunamadasu]

Current Behavior

We have deployed Cryostat operator version 3.0.0 using a custom Helm chart. When enabling cert-manager integration (enableCertManager: true) in the Cryostat resource, the operator logs repeatedly show errors like:

ERROR controllers.Cryostat Failed to set up TLS for Cryostat {"Request.Namespace": "example-ref", "Request.Name": "backend-cryostat", "error": "Object example-ref/cryostat-ca-fc1f509aeecebf908ff0acdd562c586b0c59d9791075dd1dc53b2073c0f7f537 is already owned by another Certificate controller backend-cryostat-ca"}

We have verified that RBAC permissions are correctly configured and cert-manager is installed properly. This issue only occurs when cert-manager integration is enabled.

We request investigation on whether this is a known bug or expected behavior in Cryostat operator 3.0.0, and guidance on how to resolve these ownership conflict errors when using Cryostat with cert-manager.

Thank you!

Expected Behavior

No response

Steps To Reproduce

No response

Environment

- OS: 
- Environment: 
- Version:

Anything else?

No response

[andrewazores]

Do you see this kind of error when deploying using the official Cryostat Operator or Helm Chart, or does this only happen with your custom Operator-via-Chart method? If it's only with your custom chart, would you be able to share that chart?

[Prasunamadasu]

Thank you for your response.

We have been deploying the Cryostat operator using a custom Helm chart for about a year without enabling cert-manager integration, and it has been working well.

Recently, we installed cert-manager cluster-wide for other purposes. After enabling cert-manager integration (enableCertManager: true) in the Cryostat resource, we began to encounter ownership conflict errors in the operator logs.

We have not tested the official Cryostat operator Helm chart or manifests with cert-manager integration enabled to determine if the issue also happens there.

If it would be helpful, I can share the full custom Helm chart, but for now here are the most relevant portions:

deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "chart.fullname" . }}
  labels:
    {{- include "chart.labels" . | nindent 4 }}
spec:
  replicas: 2
  selector:
    matchLabels:
      {{- include "chart.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "chart.selectorLabels" . | nindent 8 }}
    spec:
      serviceAccountName: {{ include "chart.serviceAccountName" . }}
      containers:
      - name: manager
        image: {{ .Values.operator.image }}
        imagePullPolicy: {{ .Values.operator.pullPolicy }}
        command:
        - /manager
        args:
        - "--leader-elect"
        - "--health-probe-bind-address=:8081"
        - "--metrics-bind-address=127.0.0.1:8080"
        securityContext:
          allowPrivilegeEscalation: false
          privileged: false
          runAsNonRoot: true
          capabilities:
            drop:
              - ALL
        env:
        - name: RELATED_IMAGE_OAUTH2_PROXY
          value: {{ .Values.relatedImages.oauth2Proxy }}
        - name: RELATED_IMAGE_OPENSHIFT_OAUTH_PROXY
          value: {{ .Values.relatedImages.openshiftOauthProxy }}
        - name: RELATED_IMAGE_CORE
          value: {{ .Values.relatedImages.core }}
        - name: RELATED_IMAGE_DATASOURCE
          value: {{ .Values.relatedImages.datasource }}
        - name: RELATED_IMAGE_GRAFANA
          value: {{ .Values.relatedImages.grafana }}
        - name: RELATED_IMAGE_REPORTS
          value: {{ .Values.relatedImages.reports }}
        - name: RELATED_IMAGE_STORAGE
          value: {{ .Values.relatedImages.storage }}
        - name: RELATED_IMAGE_DATABASE
          value: {{ .Values.relatedImages.database }}
        - name: WATCH_NAMESPACE
          value: ""
        - name: OPERATOR_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        resources:
          {{- toYaml .Values.operator.resources | nindent 12 }}
        ports:
        - containerPort: {{ .Values.operator.port }}
          name: webhook-server
          protocol: TCP
        volumeMounts:
        - mountPath: /tmp/k8s-webhook-server/serving-certs
          name: webhook-cert
          readOnly: true
      volumes:
      - name: webhook-cert
        secret:
          secretName: {{ include "chart.fullname" . }}-service-cert

rbac.yaml:

- apiGroups:
  - cert-manager.io
  resources:
  - certificates
  - issuers
  verbs:
  - get
  - list
  - watch
  - create
  - delete
  - update

cryostat.yaml:

apiVersion: operator.cryostat.io/v1beta2
kind: Cryostat
metadata:
  name: backend-cryostat
  namespace: example-ref
spec:
  enableCertManager: true
  targetNamespaces:
  - example-ref
  storageOptions:
    pvc:
      spec:
        resources:
          requests:
            storage: 500Mi
        storageClassName: aws-ebs-ephemeral

If you need any more sections of the chart, just let me know! I hope this helps clarify our setup and assists in troubleshooting the integration issue.

Thank you!

[andrewazores]

@ebaron what do you make of this?

[andrewazores]

I guess this is probably related to #901 .

[ebaron]

This could be caused by having the --enable-certificate-owner-ref set on cert-manager. I haven't tested this configuration. If this is the case, then we should attempt to detect this and work with that option. In this case, both our operator and cert-manager will try to set owner references on the certificate secret for GC purposes.

https://cert-manager.io/docs/usage/certificate/#cleaning-up-secrets-when-certificates-are-deleted

[ebaron]

@Prasunamadasu do you know if you have the --enable-certificate-owner-ref option set on your global cert-manager instance? That would help confirm my suspicion.

[andrewazores]

If it would be helpful, I can share the full custom Helm chart, but for now here are the most relevant portions:

@Prasunamadasu since you have this written and working already, if you're able to share that full chart I think it would be really neat to see. That was also the original request in #901, and the discussion there did make it sound like this would be a useful thing to have available for the community.

[Prasunamadasu]

@andrewazores

Since there seems to be interest in having a Helm chart for deploying the Cryostat operator, could you please advise where the best place to share this Helm chart would be?

Thank you!

[Prasunamadasu]

@Prasunamadasu do you know if you have the --enable-certificate-owner-ref option set on your global cert-manager instance? That would help confirm my suspicion.

@ebaron I have checked the cert-manager instance in our cluster and can confirm that the flag --enable-certificate-owner-ref=true is indeed set in the cert-manager container arguments.

[ebaron]

@Prasunamadasu do you know if you have the --enable-certificate-owner-ref option set on your global cert-manager instance? That would help confirm my suspicion.

@ebaron I have checked the cert-manager instance in our cluster and can confirm that the flag --enable-certificate-owner-ref=true is indeed set in the cert-manager container arguments.

Thank you, that's very helpful information!

[andrewazores]

@andrewazores

Since there seems to be interest in having a Helm chart for deploying the Cryostat operator, could you please advise where the best place to share this Helm chart would be?

Thank you!

Perhaps it could live in a directory within this repository, like cryostat-operator/helm/? You could open a PR to contribute it that way. I don't think our core team will be able to devote much time to maintaining it, but if it's a simple enough chart then hopefully it should not need too much maintenance work. Maybe someone in the community (even yourself) can help with whatever maintenance might be required, too.