Loop of VM template delete and recreate

[balazsbme]

In the Sylva validation platform I have noticed the following infinite loop with VM template delete and recreation.

See oned logs when the issue was happening and then I locked the template as a tmp workaround to stop this: see attached.

See the templates being recreated by the increasing index:

oneadmin@fe1:~$ onetemplate list
   ID USER     GROUP    NAME                                                                                                       REGTIME
33613 oneadmin oneadmin validation-capone-rke2-wkld-worker-any-vf                                                           11/12 17:36:52
33612 oneadmin oneadmin validation-capone-rke2-wkld-master                                                                  11/12 17:36:52
 2989 oneadmin oneadmin validation-capone-rke2-harbor-opennebula-worker                                                     11/05 14:34:54
 2988 oneadmin oneadmin validation-capone-rke2-harbor-opennebula-master                                                     11/05 14:34:54
 2987 oneadmin oneadmin validation-capone-rke2-mgmt-worker                                                                  11/05 14:21:17
 2986 oneadmin oneadmin validation-capone-rke2-mgmt-master                                                                  11/05 14:21:17

CAPONE pod in the management cluster keeps logging this: "capone-system/capone-controller-manager-66845bc945-b9nhn:manager"

2025-11-13T11:20:03Z    ERROR    Reconciler error    {"controller": "onecluster", "controllerGroup": "infrastructure.cluster.x-k8s.io", "controllerKind": "ONECluster", "ONECluster": {"name":"rke2-capone-workload-cell-site","namespace":"rke2-capone-workload-cell-site"}, "namespa
ce": "rke2-capone-workload-cell-site", "name": "rke2-capone-workload-cell-site", "reconcileID": "1de4a281-0fe7-47d0-bd8f-b86275112628", "error": "failed to create templates: Failed to delete existing VM template: OpenNebula error [AUTHORIZATION]: [one.template.delete] User [0] 
: TEMPLATE is locked. [33612].", "errorVerbose": "Failed to delete existing VM template: OpenNebula error [AUTHORIZATION]: [one.template.delete] User [0] : TEMPLATE is locked. [33612].\nfailed to create templates\ngithub.com/OpenNebula/cluster-api-provider-opennebula/internal/c
ontroller.(*ONEClusterReconciler).reconcileNormal\n\t/workspace/internal/controller/onecluster_controller.go:191\ngithub.com/OpenNebula/cluster-api-provider-opennebula/internal/controller.(*ONEClusterReconciler).Reconcile\n\t/workspace/internal/controller/onecluster_controller.
go:157\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandle
r\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/control
ler.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}                   
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler                                                                                                                                                                                            
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:316                                                                                                                                                                                      
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem                                                                                                                                                                                         
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:263                                                                                                                                                                                      
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2                                                                                                                                                                                               
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.6/pkg/internal/controller/controller.go:224    

These are the template definiions:

  capone:
    public_network: Service

    master_template: validation-capone-rke2-wkld-master
    worker_template: validation-capone-rke2-wkld-worker-any-vf

    images:
      - imageName: validation-capone-rke2-wkld-node
        imageContent: |
          PATH = "https://d24fmfybwxpuhu.cloudfront.net/ubuntu2204-7.0.0-0-20250528.qcow2"
          DEV_PREFIX = "vd"

      - imageName: ubuntu-22-server-cloudimg
        imageContent: |
          PATH = "https://cloud-images.ubuntu.com/releases/jammy/release-20251017/ubuntu-22.04-server-cloudimg-amd64.img"
          DEV_PREFIX = "vd"

    templates:
      - templateName: validation-capone-rke2-wkld-master
        templateContent: |
          CONTEXT = [
            NETWORK = "YES",
            SET_HOSTNAME = "$NAME",
            # full content skipped
      - templateName: validation-capone-rke2-wkld-worker-any-vf
        templateContent: |
          CONTEXT = [
            NETWORK = "YES",
            SET_HOSTNAME = "$NAME",
            SSH_PUBLIC_KEY = ""
            # full content skipped

  control_plane_replicas: 1

And in another workload cluster I am reusing these templates:

  capone:
    public_network: Service

    master_template: validation-capone-rke2-wkld-master
    worker_template: validation-capone-rke2-wkld-worker-any-vf

  control_plane_replicas: 3

This might be causing the issue, as I assume this haven't been tested. I have been working with this reuse for a while, and I only noticed now this problem, might have been around for a while.