NebulaCluster container mount-path-permissions-init chown: /usr/local/nebula/data: Operation not permitted

[aleksrosz]

Describe the bug (required)
I try to run NebulaCluster using Nebula Operator Cherry-Pick Support Non-root Image (#546) (#547)

apiVersion: apps.nebula-graph.io/v1alpha1
kind: NebulaCluster
metadata:
  name: nebula
spec:
  alpineImage: "vesoft/nebula-alpine:latest"
  graphd:
    resources:
      requests:
        cpu: "500m"
        memory: "500Mi"
      limits:
        cpu: "2"
        memory: "2Gi"
    replicas: 1
    image: vesoft/nebula-graphd
    version: v3.6.0
    service:
      type: NodePort
      externalTrafficPolicy: Local
    logVolumeClaim:
      resources:
        requests:
          storage: 1Gi
      storageClassName: storage1
  metad:
    resources:
      requests:
        cpu: "500m"
        memory: "500Mi"
      limits:
        cpu: "1"
        memory: "1Gi"
    replicas: 1
    image: vesoft/nebula-metad
    version: v3.6.0
    dataVolumeClaim:
      resources:
        requests:
          storage: 5Gi
      storageClassName: storage1
    logVolumeClaim:
      resources:
        requests:
          storage: 1Gi
      storageClassName: storage1
  storaged:
    resources:
      requests:
        cpu: "500m"
        memory: "500Mi"
      limits:
        cpu: "2"
        memory: "2Gi"
    replicas: 3
    image: vesoft/nebula-storaged
    version: v3.6.0
    dataVolumeClaims:
    - resources:
        requests:
          storage: 10Gi
      storageClassName: storage1
    logVolumeClaim:
      resources:
        requests:
          storage: 1Gi
      storageClassName: storage1
  exporter:
    image: vesoft/nebula-stats-exporter
    version: v3.3.0
    replicas: 1
    maxRequests: 20
  agent:
    image: vesoft/nebula-agent
    version: latest
    resources:
      requests:
        cpu: "100m"
        memory: "128Mi"
      limits:
        cpu: "200m"
        memory: "256Mi"
  reference:
    name: statefulsets.apps
    version: v1
  schedulerName: default-scheduler
#  enablePVReclaim: false
#  enableBR: false
  logRotate:
    rotate: 5
    size: "100M"
  imagePullPolicy: Always
  topologySpreadConstraints:
  - topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway

Your Environments (required)

OpenShift version 4.16.4
Kubernetes version v1.29.6+aba1e8d
Channel stable-4.16

How To Reproduce(required)

Steps to reproduce the behavior:

1. Step 1
   oc create -f <nebula-cluster>.yaml
2. Step 2
   Check logs of container oc logs nebula-metad-0 -c mount-path-permissions-init

Fixing permissions for: /usr/local/nebula/data
chown: /usr/local/nebula/data: Operation not permitted
chown: /usr/local/nebula/data: Operation not permitted
Fixing permissions for: /usr/local/nebula/logs
chown: /usr/local/nebula/logs: Operation not permitted
chown: /usr/local/nebula/logs: Operation not permitted

Expected behavior
Proper container permissions and possibility to run application with pod without root permissions.

Additional context
This same error is in pod nebula-storaged

[aleksrosz]

More info because I forgot to add info about security context:

oc get pod nebula-metad-0 -o yaml
apiVersion: v1
kind: Pod
metadata:
    nebula-graph.io/cm-hash: 358ef367cff6e559
    openshift.io/scc: restricted-v2
    seccomp.security.alpha.kubernetes.io/pod: runtime/default
  generateName: nebula-metad-
  labels:
    app.kubernetes.io/cluster: nebula
    app.kubernetes.io/component: metad
    app.kubernetes.io/managed-by: nebula-operator
    app.kubernetes.io/name: nebula-graph
    apps.kubernetes.io/pod-index: "0"
    controller-revision-hash: nebula-metad-76dfc79dd9
    statefulset.kubernetes.io/pod-name: nebula-metad-0
  name: nebula-metad-0
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: nebula-metad
    uid: 9cf881fc-ce81-4655-9168-0a25c0f97c85
  resourceVersion: "166017737"
  uid: 216aaad2-d40f-40b7-b671-57e7464006b0
spec:
  containers:
  - command:
    - /bin/sh
    - -ecx
    - exec /usr/local/nebula/bin/nebula-metad --flagfile=/usr/local/nebula/etc/nebula-metad.conf
      --meta_server_addrs=nebula-metad-0.nebula-metad-headless.dev-live-tv.svc.cluster.local:9559
      --local_ip=$(hostname).nebula-metad-headless.dev-live-tv.svc.cluster.local --daemonize=false
    env:
    - name: MY_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: HTTP_PORT
      value: "19559"
    - name: SCRIPT
      value: |2

        set -x

        if [ ! -s "metadata/flags.json" ]; then
         echo "flags.json is empty"
         exit 0
        fi
        while :
        do
          curl -i -X PUT -H "Content-Type: application/json" -d @/metadata/flags.json -s "http://${MY_IP}:${HTTP_PORT}/flags"
          if [ $? -eq 0 ]
          then
            break
          fi
          sleep 1
        done
    image: vesoft/nebula-metad:v3.6.0
    imagePullPolicy: Always
    lifecycle:
      postStart:
        exec:
          command:
          - /bin/sh
          - -c
          - echo "$SCRIPT" > /tmp/post-start-script && sh /tmp/post-start-script
    name: metad
    ports:
    - containerPort: 9559
      name: thrift
      protocol: TCP
    - containerPort: 19559
      name: http
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /status
        port: 19559
        scheme: HTTP
      initialDelaySeconds: 10
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: 500m
        memory: 500Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      runAsNonRoot: true
      runAsUser: 7470
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /usr/local/nebula/data
      name: metad-data
      subPath: data
    - mountPath: /usr/local/nebula/logs
      name: metad-log
      subPath: logs
    - mountPath: /usr/local/nebula/etc/nebula-metad.conf
      name: nebula-metad
      subPath: nebula-metad.conf
    - mountPath: /metadata
      name: flags
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-llr5t
      readOnly: true
  - command:
    - /bin/sh
    - -ecx
    - sh /logrotate.sh; crond -f -l 2
    env:
    - name: LOGROTATE_ROTATE
      value: "5"
    - name: LOGROTATE_SIZE
      value: 100M
    image: vesoft/nebula-alpine:latest
    imagePullPolicy: Always
    name: ng-logrotate
    resources: {}
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      runAsNonRoot: true
      runAsUser: 7470
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /usr/local/nebula/logs
      name: metad-log
      subPath: logs
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-llr5t
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: nebula-metad-0
  initContainers:
  - args:
    - echo "$SCRIPT" > /tmp/modify-permissions-script && bash /tmp/modify-permissions-script
    command:
    - /bin/sh
    - -c
    env:
    - name: SCRIPT
      value: |2

        #!/bin/bash

        # Convert comma-separated list into an array
        IFS=',' read -ra MOUNT_PATHS <<< "${MOUNT_PATHS_STR}"

        # Loop through each data mount point
        for MOUNT_PATH in "${MOUNT_PATHS[@]}"; do
            if [ -d "${MOUNT_PATH}" ]; then
                echo "Fixing permissions for: ${MOUNT_PATH}"
                chown -R ${USER_ID}:${GROUP_ID} "${MOUNT_PATH}"
            else
                echo "Warning: ${MOUNT_PATH} does not exist, skipping..."
            fi
        done
    - name: USER_ID
      value: "999"
    - name: GROUP_ID
      value: "998"
    - name: MOUNT_PATHS_STR
      value: /usr/local/nebula/data,/usr/local/nebula/logs
    image: vesoft/nebula-alpine:latest
    imagePullPolicy: Always
    name: mount-path-permissions-init
    resources:
      requests:
        cpu: 30m
        memory: 30Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      runAsNonRoot: true
      runAsUser: 7470
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /usr/local/nebula/data
      name: metad-data
      subPath: data
    - mountPath: /usr/local/nebula/logs
      name: metad-log
      subPath: logs
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-llr5t
      readOnly: true
  - args:
    - echo "$SCRIPT" > /tmp/dynamic-flags-script && sh /tmp/dynamic-flags-script
    command:
    - /bin/sh
    - -c
    env:
    - name: NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: PARENT_NAME
      value: nebula-metad
    - name: APISERVER
      value: https://kubernetes.default.svc
    - name: SERVICEACCOUNT
      value: /var/run/secrets/kubernetes.io/serviceaccount
    - name: SCRIPT
      value: "\nset -exo pipefail\n\nTOKEN=$(cat ${SERVICEACCOUNT}/token)\nCACERT=${SERVICEACCOUNT}/ca.crt\n
        \           \ncurl -s --cacert ${CACERT} --header \"Authorization: Bearer
        ${TOKEN}\" -X GET ${APISERVER}/apis/apps/v1/namespaces/${NAMESPACE}/statefulsets/${PARENT_NAME}
        | jq .metadata.annotations > /metadata/annotations.json\njq '.\"nebula-graph.io/last-applied-dynamic-flags\"
        | fromjson' /metadata/annotations.json > /metadata/flags.json\ncat /metadata/flags.json\n"
    image: vesoft/nebula-alpine:latest
    imagePullPolicy: Always
    name: dynamic-flags
    resources: {}
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      runAsNonRoot: true
      runAsUser: 7470
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /metadata
      name: flags
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-x5t
      readOnly: true
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 7470
    seLinuxOptions:
      level: s0:c31,c5
    seccompProfile:
      type: RuntimeDefault
  serviceAccount: nebula-sa
  serviceAccountName: nebula-sa
  subdomain: nebula-metad-headless
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  - effect: NoSchedule
    key: node.kubernetes.io/memory-pressure
    operator: Exists
  topologySpreadConstraints:
  - labelSelector:
      matchLabels:
        app.kubernetes.io/cluster: nebula
        app.kubernetes.io/component: metad
        app.kubernetes.io/managed-by: nebula-operator
        app.kubernetes.io/name: nebula-graph
    maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
  volumes:
  - name: metad-data
    persistentVolumeClaim:
      claimName: metad-data-nebula-metad-0
  - name: metad-log
    persistentVolumeClaim:
      claimName: metad-log-nebula-metad-0
  - configMap:
      defaultMode: 420
      items:
      - key: nebula-metad.conf
        path: nebula-metad.conf
      name: nebula-metad
    name: nebula-metad
  - emptyDir:
      medium: Memory
    name: flags
  - name: kube-api-access-llr5t
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
      - configMap:
          items:
          - key: service-ca.crt
            path: service-ca.crt
          name: openshift-service-ca.crt

[aleksrosz]

Hi
FYI. I’m working on this issue and compatibility of nebula operator and nebula images with OpenShift. Probably I will finish it till the end of next week.