ApisixConsumer reconciler retries forever after a successful delete (`failed to get ApisixConsumer … not found`)

[circleyu]

Description

Deleting an ApisixConsumer is processed correctly by the provider (the consumer is removed from APISIX), but ApisixConsumerReconciler still returns the original NotFound error. controller-runtime then rate-limits and retries the same request indefinitely, flooding the logs with:

ERROR  controllers.ApisixConsumer  failed to get ApisixConsumer  {"error": "ApisixConsumer.apisix.apache.org \"<name>\" not found"}
ERROR  controller-runtime          Reconciler error
INFO   provider.client             syncing all resources

This is still present on master:

https://github.com/apache/apisix-ingress-controller/blob/master/internal/controller/apisixconsumer_controller.go

Steps to reproduce

1. Create a namespaced ApisixConsumer that AIC watches (any namespace the controller is configured for), for example key-auth with a secretRef.

2. Wait until it is accepted / synced (service_number increases, ingest with the key succeeds).

3. Delete the CR:

   kubectl -n <ns> delete apisixconsumer <name>

4. Watch AIC logs.

Expected

After Get returns NotFound and Provider.Delete succeeds, reconcile should return ctrl.Result{}, nil and stop. No further retries for that object.

This is already how ApisixTlsReconciler behaves:

if err := r.Get(ctx, req.NamespacedName, &tls); err != nil {
    if client.IgnoreNotFound(err) == nil {
        // ...
        if err := r.Provider.Delete(ctx, &tls); err != nil {
            return ctrl.Result{}, err
        }
        r.Log.Info("deleted apisix tls", "tls", tls.Name)
        return ctrl.Result{}, nil
    }
    return ctrl.Result{}, err
}

https://github.com/apache/apisix-ingress-controller/blob/master/internal/controller/apisixtls_controller.go

Actual

ApisixConsumerReconciler.Reconcile (current master):

ac := &apiv2.ApisixConsumer{}
if err := r.Get(ctx, req.NamespacedName, ac); err != nil {
    if k8serrors.IsNotFound(err) {
        ac.Namespace = req.Namespace
        ac.Name = req.Name
        ac.TypeMeta = metav1.TypeMeta{
            Kind:       KindApisixConsumer,
            APIVersion: apiv2.GroupVersion.String(),
        }
        if err := r.Provider.Delete(ctx, ac); err != nil {
            r.Log.Error(err, "failed to delete provider", "ApisixConsumer", utils.NamespacedName(ac))
            return ctrl.Result{}, err
        }
        // missing: return ctrl.Result{}, nil
    }
    r.Log.Error(err, "failed to get ApisixConsumer", "request", req.NamespacedName)
    return ctrl.Result{}, err
}

When Get is NotFound and Provider.Delete succeeds, execution falls through and still returns the NotFound error. controller-runtime treats that as a failed reconcile and retries with exponential backoff (observed ~20s, ~41s, then continuing until max delay).

The provider path itself appears fine: each retry is followed by syncing all resources, so the delete is applied (and then re-applied). The object is gone from the API server.

Sample logs

INFO   controllers.ApisixConsumer  reconcile
       {"request": {"name":"cust-8fa274728c24","namespace":"signalgate-consumers"}}
ERROR  controllers.ApisixConsumer  failed to get ApisixConsumer
       {"request": {"name":"cust-8fa274728c24","namespace":"signalgate-consumers"},
        "error": "ApisixConsumer.apisix.apache.org \"cust-8fa274728c24\" not found"}
ERROR  controller-runtime          Reconciler error
       {"controller": "apisixconsumer", "controllerKind": "ApisixConsumer",
        "namespace": "signalgate-consumers", "name": "cust-8fa274728c24",
        "error": "ApisixConsumer.apisix.apache.org \"cust-8fa274728c24\" not found"}
INFO   provider.client             syncing all resources
INFO   provider.client             syncing resources for config  {"service_number": 27}

Line numbers in a 2.x build match controller/apisixconsumer_controller.go:61 (reconcile) and :77 (failed to get ApisixConsumer).

Restarting the AIC pod clears the workqueue and stops the loop until the next delete.

Suggested fix

Return success after a successful provider delete on NotFound, and only log/return Get errors that are not NotFound:

if err := r.Get(ctx, req.NamespacedName, ac); err != nil {
    if k8serrors.IsNotFound(err) {
        ac.Namespace = req.Namespace
        ac.Name = req.Name
        ac.TypeMeta = metav1.TypeMeta{
            Kind:       KindApisixConsumer,
            APIVersion: apiv2.GroupVersion.String(),
        }
        if err := r.Provider.Delete(ctx, ac); err != nil {
            r.Log.Error(err, "failed to delete provider", "ApisixConsumer", utils.NamespacedName(ac))
            return ctrl.Result{}, err
        }
        return ctrl.Result{}, nil
    }
    r.Log.Error(err, "failed to get ApisixConsumer", "request", req.NamespacedName)
    return ctrl.Result{}, err
}

Please also check other 2.x reconcilers for the same fall-through. ApisixTlsReconciler already returns nil after delete; ApisixConsumer does not.

Environment

• APISIX Ingress Controller: 2.x (controller-runtime ApisixConsumer reconciler; controller/apisixconsumer_controller.go)
• Trigger: kubectl delete of ApisixConsumer (or any client that deletes the CR)
• Confirmed against current master source

[alliasgher]

Opened #2850 for this.

On your question about other controllers: there are 15 Provider.Delete call sites across 13 files in internal/controller, and all of them already return after a successful delete. ApisixConsumer is the only one missing it.