bug: standalone/ADC - an out-of-band `secrets` resource deadlocks the sync loop (`secrets_conf_version` 400), blocking all reconciliation

[jens-skribble]

Current Behavior

We run APISIX in API-driven standalone mode driven by the ingress controller
(ADC, provider.type: apisix-standalone). We wanted a route's plugin to reference a
Vault secret via APISIX's native $secret://vault/.... There is no CRD or
GatewayProxy field we could find to declare the backing secrets secret-manager, so
the only way to get one onto the data plane is to register it directly on APISIX's
standalone Admin API (out of band).

Doing that deadlocks the controller. Once the out-of-band manager bumps APISIX's
secrets_conf_version from 0 to 1, every subsequent controller/ADC full-config
push is rejected:

PUT /apisix/admin/configs → 400 Bad Request:
secrets_conf_version must be greater than or equal to (1)

ADC has no secrets resource type, so its push carries secrets_conf_version: 0
while APISIX is now at 1; APISIX's monotonic version check rejects the whole
push. Because each standalone update replaces the full configuration atomically,
nothing reconciles after that: the out-of-band manager is not wiped, the controller
is simply stuck, and even unrelated routes stop applying (they 404 at the gateway).

This looks like a robustness bug independent of the secret use case: once
secrets_conf_version is > 0 by any means (an out-of-band manager, a leftover
from a prior setup), ADC - which always sends 0 - can never sync again, and it takes
all other resources down with it.

For reference, APISIX itself handles $secret:// fine in this mode - we confirmed it
end-to-end with two standalone docker/podman baselines (in the attached bundle), so the
problem is on the controller/ADC side, not APISIX:

• compose/api-driven/ - in API-driven standalone, APISIX resolves $secret://vault/...
  when a secrets manager is present in the /configs payload.
• compose/file-driven/ - the documented apisix.yaml secrets: block resolves, but
  that mode disables the Admin API the controller drives (so it can't be used together
  with the controller).

Expected Behavior

• The controller/ADC should not deadlock on a resource type it doesn't manage: it
  should send a valid (or otherwise tolerated) secrets_conf_version so a stray or
  out-of-band secrets resource can't block reconciliation of everything else.
• Ideally there would be a supported way to declare an APISIX secrets secret-manager
  through the controller (e.g. a secrets field on GatewayProxy, or a dedicated
  CRD), so $secret://vault/... references can be used. Today we can't find one.
  (secretRef reads a Kubernetes Secret and injects plaintext - a different mechanism,
  not APISIX's native $secret:// manager.)

Error Logs

Controller manager container - this block repeats on every sync attempt after the
out-of-band secrets resource exists:

2026-07-22T09:14:48.222Z    INFO    provider.client    client/client.go:214    syncing all resources
2026-07-22T09:14:48.239Z    ERROR    provider.executor    client/executor.go:420    ADC Server sync failed    {"result": {"status":"all_failed","total_resources":0,"success_count":0,"failed_count":1,"success":[],"failed":[],"endpoint_status":[{"server":"http://10.244.0.3:9180","success":false,"reason":"PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}]}, "error": "ADC Server sync failed (standalone mode): http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}
2026-07-22T09:14:48.240Z    ERROR    provider.executor    client/executor.go:153    failed to run http sync for server    {"server": "http://10.244.0.3:9180", "error": "ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}
2026-07-22T09:14:48.240Z    ERROR    provider.client    client/client.go:306    failed to execute adc command    {"config": {"name":"GatewayProxy/ingress-apisix/apisix-config","serverAddrs":["http://10.244.0.3:9180"],"tlsVerify":false}, "error": "ADC execution error for GatewayProxy/ingress-apisix/apisix-config: [ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)]"}
2026-07-22T09:14:48.240Z    ERROR    provider.client    client/client.go:247    failed to sync resources    {"name": "GatewayProxy/ingress-apisix/apisix-config", "error": "ADC execution errors: [ADC execution error for GatewayProxy/ingress-apisix/apisix-config: [ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)]]"}
2026-07-22T09:14:48.240Z    ERROR    provider    apisix/provider.go:282    failed to sync    {"error": "failed to sync 1 configs: GatewayProxy/ingress-apisix/apisix-config"}
2026-07-22T09:14:56.241Z    INFO    provider.client    client/client.go:214    syncing all resources
2026-07-22T09:14:56.254Z    ERROR    provider.executor    client/executor.go:420    ADC Server sync failed    {"result": {"status":"all_failed","total_resources":0,"success_count":0,"failed_count":1,"success":[],"failed":[],"endpoint_status":[{"server":"http://10.244.0.3:9180","success":false,"reason":"PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}]}, "error": "ADC Server sync failed (standalone mode): http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}
2026-07-22T09:14:56.254Z    ERROR    provider.executor    client/executor.go:153    failed to run http sync for server    {"server": "http://10.244.0.3:9180", "error": "ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}
2026-07-22T09:14:56.254Z    ERROR    provider.client    client/client.go:306    failed to execute adc command    {"config": {"name":"GatewayProxy/ingress-apisix/apisix-config","serverAddrs":["http://10.244.0.3:9180"],"tlsVerify":false}, "error": "ADC execution error for GatewayProxy/ingress-apisix/apisix-config: [ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)]"}
2026-07-22T09:14:56.255Z    ERROR    provider.client    client/client.go:247    failed to sync resources    {"name": "GatewayProxy/ingress-apisix/apisix-config", "error": "ADC execution errors: [ADC execution error for GatewayProxy/ingress-apisix/apisix-config: [ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)]]"}
2026-07-22T09:14:56.255Z    ERROR    provider    apisix/provider.go:282    failed to sync    {"error": "failed to sync 1 configs: GatewayProxy/ingress-apisix/apisix-config"}

Steps to Reproduce

A self-contained reproduction is attached as apisix-secrets-repro.zip:

• k8s/ - a kustomization that stands everything up + a Job that automates steps 3–6
  below and prints the controller's own rejection log.
• compose/api-driven/ - docker/podman baseline: APISIX resolves $secret://vault/...
  in API-driven standalone (shows the gap is controller-side, not APISIX).
• compose/file-driven/ - docker/podman baseline: the documented apisix.yaml
secrets: block resolves, but that mode has no Admin API for the controller to drive.

To reproduce the deadlock manually against the controller:

1. Deploy APISIX in API-driven standalone mode (deployment.role: traditional,
   role_traditional.config_provider: yaml) + APISIX Ingress Controller 2.1.0 — e.g.
   the apisix Helm chart 2.16.0 with ingress-controller.enabled: true,
   ingress-controller.config.provider.type: apisix-standalone, and
   ingress-controller.gatewayProxy.createDefault: true.
2. Apply any ApisixRoute; confirm the controller syncs it —
   GET /apisix/admin/configs shows the route and .secrets is empty (secrets = 0).
3. Register a secrets manager out of band via the standalone Admin API (echo the
   existing *_conf_version values back, omit secrets_conf_version, add the manager):

   PUT /apisix/admin/configs
   { ...existing *_conf_version..., "secrets": [
       { "id": "vault/oidc", "uri": "http://vault.example:8200", "prefix": "kv/apisix", "token": "root" } ] }
   

   Confirm .secrets now has 1 entry (secrets = 1); APISIX's secrets_conf_version
   is now 1.
4. Trigger a controller re-sync (e.g. kubectl annotate apisixroute <name> poke=$(date +%s) --overwrite).
5. Watch the controller logs: every sync now fails with
   400 ... secrets_conf_version must be greater than or equal to (1).
6. Observe the blast radius: the out-of-band manager is not removed, and because the
   push is atomic, no route reconciles — curl against the gateway returns 404 even
   for routes unrelated to secrets.

Environment

APISIX Ingress controller version (apisix-ingress-controller version --long): 2.1.0

Git SHA: 
Go Version: go1.24.13
Building OS/Arch: /
Running OS/Arch: linux/amd64

Kubernetes cluster version (kubectl version):

Client Version: v1.33.13
Kustomize Version: v5.6.0
Server Version: v1.35.1

(reproduced on minikube, Podman driver)

APISIX: 3.17.0; installed via the apisix Helm chart 2.16.0;
provider.type: apisix-standalone

[AlinsRan]

Thanks for the detailed reproduction.

This is a constraint of the standalone model, not a bug in the sync path

PUT /apisix/admin/configs replaces the whole configuration atomically, and the controller always pushes a full snapshot. There is no incremental path, so anything not in that snapshot isn't part of the configuration the controller declares.

Out-of-band resources survive today only because ADC echoes back fields it doesn't recognize from the config it fetched. That's an implementation detail, not a contract — and it's the same path that carried a stale secrets_conf_version into the payload and produced your 400.

So we won't be making out-of-band registration a supported setup.

Unblocking it today

Restart the ADC container (or the controller pod) — the cached snapshot is rebuilt from the data plane on the next sync and reconciliation resumes. The cache is an LRU with a 1h TTL (ADC_APISIX_STANDALONE_CACHE_TTL_MS), so it also recovers on its own eventually.

What we will fix

The control plane shouldn't be blocked indefinitely by external state. ADC already exposes bypassCache on its sync API, but the controller doesn't pass it, and ADC keeps its cached snapshot on a failed sync. Wiring that up turns the deadlock into one failed sync that self-heals — robustness, not support for out-of-band writes.

$secret:// support

You're right that there's no way to declare a secrets manager through the controller today. secrets isn't in the resource model, and secretRef is a different mechanism (reads a Kubernetes Secret, injects the value). The real answer is bringing secrets into the control plane so it becomes part of the snapshot.

That's a feature and needs its own design discussion — note that once the controller owns secrets, anything registered out of band gets removed on the next sync.

[jens-skribble]

Thank you @AlinsRan for the answer. I went ahead and created a feature request #2831 to track the $secret:// support.