request help: Apisix ingress controller not able to sync the config for either large num of apis and admin config or with large number of paths in single route

[taherbohri]

Issue description

Hi Team ,
We are using apisix ingress controller and apisix gateway , where we are using apisix crd with apisixupstream , apisixroute , and pluginconfig

we were deploying mutiple apis where we are creating one apisixroute per api with mutiple paths init , and we are having apis arround 700 .

In one segment the with One apisixroute with mutiple paths the deployment and validation were successfull since it was having only 100 apis .

But in this another segment we are having 700 apis and in few apis the paths are arround 150 and being added in one apisixroute so here we were getting the error as -

2026-06-05T15:35:48.086Z  ERROR  status.updater  status/updater.go:94  unable to update status  {"name": "insightvm-apis", "namespace": "apis-ns", "error": "ApisixRoute.apisix.apache.org "insightvm-apis" is invalid: status.conditions[0].message: Too long: may not be more than 32768 bytes"}

SO we made the change to apisixrote as for each path we will create a separate apisixroute , and it fixed the above issue , but then ingress controller was only able to sync the apis till 680 apis after that it failed with error -

ERROR provider apisix/provider.go:282 failed to sync {"error": "failed to sync 1 configs: GatewayProxy/hawkeye-ns/hke-int-apisix-ingress-controller-config"}

and apisixgateway throws error as - 192.168.24.247 - - [09/Jun/2026:11:55:37 +0000] 192.168.112.152:9180 "PUT /apisix/admin/configs HTTP/1.1" 500 58 0.066 "-" "axios/1.13.2" - - - "http://192.168.112.152:9180"

Environment

apisix ingress controller cersion - 2.0.1
Adc version - 0.24.2
Apisix gateway version - 3.14

Standalone traditional mode without etcd

[Baoyuantop]

Thanks for the detailed report @taherbohri. There seem to be two separate issues here.

The first error:

status.conditions[0].message: Too long: may not be more than 32768 bytes

is a Kubernetes status update failure, not APISIX Gateway rejecting the route config. When sync fails, the ingress controller writes the failure details into the CRD condition message. With a large ApisixRoute / many paths, the aggregated message can exceed Kubernetes' limit for a condition message, and the status update is rejected. The controller should cap or summarize this message before writing it back to status.

The later error:

failed to sync config
PUT /apisix/admin/configs HTTP/1.1 500

is a different failure in the ADC / APISIX standalone Admin API sync path. The access log only shows that APISIX returned 500; it is not enough to identify the root cause. In standalone mode, APISIX stores the full config in the standalone-config shared dict, whose default size is 10m, so a large generated config may hit a shared-dict capacity issue or another APISIX-side error. We need the APISIX error log or the ADC response body to confirm.

Could you please provide:

1. APISIX Gateway error.log around the same timestamp, not only access log;
2. Full ingress-controller logs around failed to sync config;

[taherbohri]

sure sir,
one thing to confirm that 32 kb issue only comes when we are deploying more than 50 paths in single apisixroute , so we have now switched to 1 apisixroute for each path(this will create very large number of routes as each api may have arroung 10 to 12 paths and some having more than 100 paths) which had resolved 32 kb issue , but now with this new approach aswell I was able to sync arround 680 apis untill this everything was working fine then after adding few more apis I started seeing this error in ingress controller -

code 500"} 2026-06-10T06:33:49.189Z ERROR provider.client client/client.go:269 failed to execute adc command {"config": {"name":"GatewayProxy/apis-ns/hke-int-apisix-ingress-controller-config","serverAddrs":["http://192.168.24.245:9180","http://192.168.92.19:9180","http://192.168.118.56:9180","http://192.168.108.149:9180","http://192.168.112.152:9180"],"tlsVerify":false}, "error": "ADC execution error for GatewayProxy/apis-ns/hke-int-apisix-ingress-controller-config: [ServerAddr: http://192.168.24.245:9180,http://192.168.92.19:9180,http://192.168.118.56:9180,http://192.168.108.149:9180,http://192.168.112.152:9180, Err: http://192.168.112.152:9180: Request failed with status code 500; http://192.168.118.56:9180: Request failed with status code 500; http://192.168.108.149:9180: Request failed with status code 500; http://192.168.24.245:9180: Request failed with status code 500; http://192.168.92.19:9180: Request failed with status code 500]"} 2026-06-10T06:33:49.192Z ERROR provider.client client/client.go:210 failed to sync resources {"name": "GatewayProxy/apis-ns/hke-int-apisix-ingress-controller-config", "error": "ADC execution errors: [ADC execution error for GatewayProxy/apis-ns/hke-int-apisix-ingress-controller-config: [ServerAddr: http://192.168.24.245:9180,http://192.168.92.19:9180,http://192.168.118.56:9180,http://192.168.108.149:9180,http://192.168.112.152:9180, Err: http://192.168.112.152:9180: Request failed with status code 500; http://192.168.118.56:9180: Request failed with status code 500; http://192.168.108.149:9180: Request failed with status code 500; http://192.168.24.245:9180: Request failed with status code 500; http://192.168.92.19:9180: Request failed with status code 500]]"}

and in apisixgateway logs -

ERROR provider apisix/provider.go:282 failed to sync {"error": "failed to sync 1 configs: GatewayProxy/apis-ns/hke-int-apisix-ingress-controller-config"}

[mayankmurari]

Hi @Baoyuantop
Thanks for looking into this.

To provide additional context on our deployment and the conditions under which we are seeing the issue:

Current deployment model

• We create one ApisixRoute per path. Total count: 1400+ ApisixRoutes.
• We create one ApisixUpstream per application. Total count: 100+ ApisixUpstreams.
• We create one ApisixPluginConfig per application. Total count: 100+ ApisixPluginConfigs.
• Overall, we are managing 600+ APIs containing 1400+ paths.

Observed behavior

• When all resources are deployed together, synchronization fails with ADC execution errors and the corresponding GatewayProxy sync failure:

  • ADC execution error ... Request failed with status code 500
  • failed to sync 1 configs: GatewayProxy/...

• If we split the deployment into two batches, both deployments complete successfully.

• We captured the Admin API request size during a successful deployment of approximately 500 APIs, and the generated configuration payload sent to APISIX was approximately 9.56 MB.

• We also attempted to increase luaSharedDicts beyond the default value as part of our investigation, but were not able to increase it in our deployment.

Based on this behavior, we are trying to better understand the scalability characteristics and recommended deployment pattern for APISIX Ingress Controller at this scale.

Questions

1. Is there a documented or known size/complexity limit for a single GatewayProxy configuration or ADC config push?

2. Are these failures typically driven by:

   • total number of APIs,
   • total number of routes/paths,
   • generated configuration size,
   • shared dictionary memory limits, or
   • some other scalability constraint?

3. Does a configuration payload size of approximately 9.56 MB for ~500 APIs indicate that we may be approaching a known APISIX, ADC, or GatewayProxy limit? What is the config parameter to increase this size? We tried increaseing value for luaSharedDicts which was not accepted config.

4. In our current model, we use one ApisixRoute per path. Is this the recommended pattern for large-scale deployments, or is there a better approach?

5. Previously, we grouped multiple paths within a single ApisixRoute, but eventually encountered:

   • status.conditions[0].message: Too long: may not be more than 32768 bytes
     To avoid that limit, we moved to one-route-per-path. Is this the recommended workaround, or is there a different scaling strategy that should be used?

6. What is the recommended design pattern for managing 1000+ APIs / 10000+ paths with APISIX Ingress Controller with API Driven Standalone Mode while avoiding both:

   • Kubernetes status size limitations,
   • ADC/GatewayProxy synchronization failures, and
   • potential memory/config-size bottlenecks?

Any guidance on supported scale limits, recommended resource organization, memory tuning recommendations, or known best practices for deployments of this size would be greatly appreciated.

[taherbohri]

Hi @Baoyuantop
We understood the issue is related to the config size , since you mentioned the config size by default supports only 10 mb and since our config is exceeding that 10mb limit we are facing the config sync issue ,
so We tried updating the standalone-config as -

gateway values file - 
    metaLuaSharedDicts:
      - name: standalone-config
        size: 50m

and in gateway config template -

inside nginx_config -

 meta:
    lua_shared_dict:
    {{- range $dict := .Values.apisix.nginx.metaLuaSharedDicts }}
      {{ $dict.name }}: {{ $dict.size }}
    {{- end }}
  {{- end }}

It worked for us , I was able to see the config of 11.8 mb getting synced.

But then I noticed the out of 5 pods of gateway one by one pods were not getting config again , like some were getting then some were not and then again all were not , like this the pods were coming up and down.

Like the config size was getting set and reset from 50 to 10 mb during the run time and similar in ingress controller , sometime it was able to sync the config to the admin api and sometimes not.

@Baoyuantop Could you please recommend the solution for above as we will have a large config in and it may also increase in future .

Note - we are on 3.14 and I have manually updated the gateway config template to sync the config size.