Kong Ingress Controller (DB-less) not converging state across pods – one instance missing routes leading to intermittent 404s

[akshaysharama]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

🔎 Summary

In a DB-less Kong deployment with multiple replicas, one Kong instance intermittently serves 404 responses due to incomplete route configuration, while other instances function correctly.

This results in inconsistent routing behavior depending on which pod receives the request.

---

Controller Resources:

requests:
  cpu: 25m
  memory: 100Mi
limits:
  cpu: 100m
  memory: 250Mi

---

🚨 Observed Behavior

• Requests via LoadBalancer intermittently return:

  • ✅ 200 OK (healthy pods)
  • ❌ 404 Not Found (faulty pod)

• Kong Admin API comparison across pods:

  curl -sk https://localhost:8001/routes | jq '.data | length'

  • Healthy pods → full route set
  • Faulty pod → partial routes (e.g., only TCPIngress)

• Faulty pod:

  • Remains in partial state indefinitely
  • Does not self-recover
  • Appears Ready from Kubernetes perspective

---

🔍 Additional Observations

• No critical errors in logs (log level = warn)
• Occasional network timeout logs (unclear if related)
• Only partial configuration present on affected pod (TCPIngress present, HTTP Ingress missing)
• https://artifacthub.io/packages/helm/kong/kong/2.27.0 (helm chart we are using to deploy)

---

💥 Impact

• Intermittent production 404 errors
• Non-deterministic routing behavior
• Difficult to detect via health checks (pods remain Ready)

---

🔧 Workarounds

• Restarting affected pod → resolves issue temporarily

---

🙏 Any guidance or pointers would be appreciated.

Expected Behavior

📊 Expected Behavior

All Kong instances should independently converge to the same configuration derived from Kubernetes resources, resulting in consistent routing across all pods.

---

Steps To Reproduce

### 🧪 Steps to Reproduce (approximate)

1. Deploy Kong (DB-less) with KIC v2.11 and multiple replicas
2. Apply multiple Ingress and TCPIngress resources
3. Send traffic via LoadBalancer
4. Observe intermittent 404 responses
5. Compare `/routes` across pods → one pod shows missing routes

---

Kong Ingress Controller version

### 📦 Environment

* **Kong Version:** 3.3
* **Kong Ingress Controller Version:** 2.11
* **Kubernetes Distribution:** k3s
* **Deployment Mode:** DB-less (`KONG_DATABASE=off`)
* **Replicas:** Multiple (behind LoadBalancer)

Kubernetes version

Client Version: v1.30.7+k3s1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.7+k3s1

Anything else?

No response

[pmalek]

Hi @akshaysharama,

Thanks for you report. KIC 2.11 (similarly to Kong 3.3) has already reached EOL as per https://developer.konghq.com/kubernetes-ingress-controller/support/#kong-ingress-controller-versions.

Do you mind verifying this issue with supported versions and reporting back?

[akshaysharama]

@pmalek,

Sorry for the delayed response—I was tied up with release activities.

We’ve upgraded the version, and at the moment we’re unable to reproduce the issue. It appears to be intermittent.

During debugging, I noticed that we are running in DB-less mode. In this mode, leader election is typically disabled, meaning all ingress controller instances push configuration independently. If that’s the case, this might be a race condition causing the issue we observed.

Additionally, we have explicitly enabled leader election with the following configuration (KIC version 3.5). However, we’re still not seeing any lease objects (kubectl get leases -A) or related logs in the ingress controller, which is unexpected.

 Image:           kong/kubernetes-ingress-controller:3.5
    Image ID:     kong/kubernetes-ingress-controller@sha256:abfe1735d4901c4965c50bdb55f3de086d6a04f4c596359e7e92180f58fa850c
    Ports:           10255/TCP, 10254/TCP
    Host Ports:      0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    State:           Running
      Started:       Fri, 17 Jul 2026 05:28:25 +0000
    Ready:           True
    Restart Count:   0
    Limits:
      cpu:     100m
      memory:  250M
    Requests:
      cpu:      25m
      memory:   100M
    Liveness:   http-get http://:10254/healthz delay=5s timeout=5s period=10s #success=1 #failure=3
    Readiness:  http-get http://:10254/readyz delay=5s timeout=5s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:                               apigw-kong-7dbbd677fd-l8xdl (v1:metadata.name)
      POD_NAMESPACE:                          pod-services (v1:metadata.namespace)
      CONTROLLER_ELECTION_ID:                 kong-ingress-controller-leader-kong
      CONTROLLER_INGRESS_CLASS:               kong
      CONTROLLER_LEADER_ELECTION:             true
      CONTROLLER_KONG_ADMIN_TLS_SKIP_VERIFY:  true
      CONTROLLER_KONG_ADMIN_URL:              https://localhost:8444
      CONTROLLER_LOG_LEVEL:                   debug
      CONTROLLER_PUBLISH_SERVICE:             services/apigw-kong-proxy

@pmalek , also, we have an architecture, where we have 3 instance, each instance(pod), has one KIC and one proxy, so I guess each KIC will push config to its local proxy as per the above config, if that's so then there is no way of race condition, so what could have happen for this incomplete config,? as it is seeing very frequently now.

[pmalek]

Regarding the leader election:

You've defined CONTROLLER_LEADER_ELECTION and that flag doesn't exist. Leader election is detected based on other settings, here's the logic for dbless:

kubernetes-ingress-controller/internal/manager/setup.go

Lines 139 to 146 in d8eb9a0

	if dbmode.IsDBLessMode() {
	if c.KongAdminSvc.IsPresent() {
	logger.Info("DB-less mode detected with service detection, enabling leader election")
	return true
	}
	logger.Info("DB-less mode detected, disabling leader election")
	return false
	}

. You can force leader election via CONTROLLER_FORCE_LEADER_ELECTION flag.

---

Regarding the 404 issue:

would you be able to gather KIC logs together with logs showing how you're trying to reach the proxy and where it's failing? This would help us correlated the timestamps and do some troubleshooting.

You also wrote:

Remains in partial state indefinitely

Does that mean that when this happens, it's never able to recover?

[akshaysharama]

Hi @pmalek,

I think leader election won't really help here in our design, or the way we are deploying Kong.
We have 3 instances, and each instance (pod) contains one KIC and one proxy. Based on the current setup, each KIC should push configuration only to its corresponding local proxy (CONTROLLER_KONG_ADMIN_URL: https://localhost:8444). If that is the case, there shouldn’t really be a chance for a race condition.
Is my understanding correct @pmalek?

if yes, given this, what could be causing the incomplete configuration issue? We’re also observing this much more frequently now.

issue:
After the greenfield deployment, everything works fine initially. However, after a few days, we start seeing intermittent issues. Specifically, when Kong or the node restarts, one of the Kong instances sometimes comes up with a partial configuration.

Because of this, if a request hits that particular instance via ingress, it results in a 404, as the required ingress configuration or metadata is missing on that instance.

What’s unclear is why this is happening only to one instance at a time. Since each pod has its own local KIC syncing configuration to its local proxy, a race condition or sync issue doesn’t seem like a strong possibility here.

it will be stuck in same state indefinitely until you restart. @pmalek

[pmalek]

Hi @pmalek,

I think leader election won't really help here in our design, or the way we are deploying Kong. We have 3 instances, and each instance (pod) contains one KIC and one proxy. Based on the current setup, each KIC should push configuration only to its corresponding local proxy (CONTROLLER_KONG_ADMIN_URL: https://localhost:8444). If that is the case, there shouldn’t really be a chance for a race condition. Is my understanding correct @pmalek?

if yes, given this, what could be causing the incomplete configuration issue? We’re also observing this much more frequently now.

Yes, in that case leader election shouldn't be relevant (I just merely commented to show you how leader election is enabled in KIC).

issue: After the greenfield deployment, everything works fine initially. However, after a few days, we start seeing intermittent issues. Specifically, when Kong or the node restarts, one of the Kong instances sometimes comes up with a partial configuration.

Because of this, if a request hits that particular instance via ingress, it results in a 404, as the required ingress configuration or metadata is missing on that instance.

What’s unclear is why this is happening only to one instance at a time. Since each pod has its own local KIC syncing configuration to its local proxy, a race condition or sync issue doesn’t seem like a strong possibility here.

it will be stuck in same state indefinitely until you restart.

This does seem like a race condition of some sorts. If you could get us some logs:

• from KIC (debug enabled)
• from something hitting the ingress ( to correlate when the connectivity was lost with KIC events )

We could then take a closer look at this.