Repetitive update of the reconciled VaultSecret status

[sadovnikov]

While preparing to test #37, another problem was discovered. It can be reproduced with both 1.1.2 and 1.2.1.

The scenario is:

• the Vault operator runs in argocd namespace and watches on its own namespace. It runs using default SA
• the git-github VaultSecret is being created in the namespace. It should be synced with the git-github secret

Everything works smoothly, but the following gets logged by the Vault operator

2022-03-11T15:00:56.732Z	ERROR	controller	Reconciler error	{"reconcilerGroup": "maupu.org", "reconcilerKind": "VaultSecret", "controller": "vaultsecret", "name": "git-github", "namespace": "argocd", "error": "Operation cannot be fulfilled on vaultsecrets.maupu.org \"git-github\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/internal/controller/controller.go:237
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/internal/controller/controller.go:209
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/internal/controller/controller.go:188
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
	/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
	/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
	/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90

Seems the operator attempts to reconcile the secret three times. The first two attempts start at nearly the same time and one of the threads gets into the conflict trying to update the status of the VaultSecret.

This is confirmed by the logs of the API server. it gets two requests to update /apis/maupu.org/v1beta1/namespaces/argocd/vaultsecrets/git-github/status from system:serviceaccount:argocd:default. The second update is rejected with "Conflict" 409.

❯ kubectl -n argocd logs vault-secret-7978575d49-88b77 | grep "Reconciling VaultSecret"
2022-03-11T15:00:41.050Z	INFO	vaultsecret-operator	Reconciling VaultSecret	{"Request.Namespace": "argocd", "Request.Name": "git"}
2022-03-11T15:00:41.244Z	INFO	vaultsecret-operator	Reconciling VaultSecret	{"Request.Namespace": "argocd", "Request.Name": "git-github-internal"}
2022-03-11T15:00:41.322Z	INFO	vaultsecret-operator	Reconciling VaultSecret	{"Request.Namespace": "argocd", "Request.Name": "argocd-secret"}
2022-03-11T15:00:41.387Z	INFO	vaultsecret-operator	Reconciling VaultSecret	{"Request.Namespace": "argocd", "Request.Name": "git"}
2022-03-11T15:00:56.574Z	INFO	vaultsecret-operator	Reconciling VaultSecret	{"Request.Namespace": "argocd", "Request.Name": "git-github"}
2022-03-11T15:00:56.668Z	INFO	vaultsecret-operator	Reconciling VaultSecret	{"Request.Namespace": "argocd", "Request.Name": "git-github"}
2022-03-11T15:00:57.732Z	INFO	vaultsecret-operator	Reconciling VaultSecret	{"Request.Namespace": "argocd", "Request.Name": "git-github"}