Skip to content

fix(standalone): stop aborting stream connections before the first config arrives - #13855

Open
AlinsRan wants to merge 1 commit into
apache:masterfrom
AlinsRan:fix/stream-router-nil-values
Open

fix(standalone): stop aborting stream connections before the first config arrives#13855
AlinsRan wants to merge 1 commit into
apache:masterfrom
AlinsRan:fix/stream-router-nil-values

Conversation

@AlinsRan

Copy link
Copy Markdown
Contributor

Description

In standalone mode the stream subsystem starts accepting connections before any configuration has reached it, and every connection that arrives in that window is aborted:

2026/08/19 10:03:22 [error] 49#49: *4026 lua entry thread aborted: runtime error:
  /usr/local/apisix/apisix/core/config_util.lua:37: attempt to index local 'tab' (a nil value)
stack traceback:
  /usr/local/apisix/apisix/core/config_util.lua: in function '(for generator)'
  /usr/local/apisix/apisix/stream/router/ip_port.lua:76: in function 'create_router'
  /usr/local/apisix/apisix/stream/router/ip_port.lua:154: in function 'match'
  /usr/local/apisix/apisix/init.lua:1328: in function 'stream_preread_phase'

The client sees an empty response, not an error, so this shows up as a stream port that silently answers nothing.

Why values is nil. config_yaml.sync_data returns early while the config source has not delivered anything yet:

conf_version = apisix_yaml[self.conf_version_key] or 0     -- 0, apisix_yaml is empty
if not conf_version or conf_version == self.conf_version then
    return true                                             -- values is left nil
end

user_routes starts as {conf_version = 0, values = nil}, so values stays nil while conf_version is 0.

Why the connection aborts. In ip_port.lua, router_ver starts as nil, so router_ver ~= user_routes.conf_version (nil ~= 0) holds and create_router(user_routes.values) is called with nil. The error is thrown before router_ver is assigned, so the next connection takes the same branch and aborts again — the worker does not leave this state on its own.

Why it does not recover. Only the polling loop in admin/standalone.lua carries configuration across the http/stream boundary (the events module cannot broadcast between subsystems, as the comment there says). It skipped an update whenever X-Last-Modified was unchanged, and that field is ngx_time() — second resolution. Two updates inside one second are indistinguishable by it, so if the poll ran between them the second update was dropped permanently: nothing re-delivers it later, because a client re-sending the same content is answered with config not changed: same digest and the stored metadata never advances.

That combination is what I hit in the field: a control plane pushed several configs inside one second, the first without stream_routes and a later one with them, and the stream workers of that instance answered nothing for the rest of the pod's life while the HTTP subsystem and the Admin API both showed the stream route present.

Fix

  • apisix/stream/router/ip_port.lua: treat a missing route list as "no stream route" instead of indexing nil, so the connection takes the normal no-match path and the router is rebuilt once the routes arrive.
  • apisix/admin/standalone.lua: compare X-Digest as well as X-Last-Modified in the polling loop. The digest changes with the content, so a second update inside the same second is no longer invisible to the other subsystem.

Reproduce

No control plane needed:

# config.yaml
deployment:
  role: traditional
  role_traditional:
    config_provider: yaml
  admin:
    admin_key:
    - key: edd1c9f034335f136f87ad84b625c8f1
      name: admin
      role: admin
nginx_config:
  worker_processes: 2
  error_log_level: info
apisix:
  proxy_mode: http&stream
  stream_proxy:
    tcp:
      - 9100
docker run -d --name apisix-repro -p 19180:9180 -p 19100:9100 \
  -v $PWD/config.yaml:/usr/local/apisix/conf/config.yaml:ro apache/apisix:dev

# connect to the stream port before pushing any configuration
printf 'GET / HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n' | nc 127.0.0.1 19100
# -> empty response, and the trace above in logs/error.log

With this patch the same connection logs ip_port.lua: match(): not hit any route and no error, and once a config carrying stream_routes is pushed the connection is routed normally.

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change — n/a, no user-facing interface changed
  • I have verified that this change is backward compatible

Note on the test: t/admin/standalone-stream.t asserts that a stream connection made before the first config push takes the no-match path and logs no abort. I could not run test-nginx locally — the openresty on this machine predates apisix_stream_metrics_zone, so pre-existing stream tests such as t/config-center-yaml/stream-route.t fail to start nginx here as well. The behaviour was verified against apache/apisix:dev with the two patched files mounted in, as described above.

…nfig arrives

In standalone mode the stream subsystem accepts connections as soon as it
starts, which can be well before any configuration has reached it, and every
one of those connections was aborted with

    core/config_util.lua:37: attempt to index local 'tab' (a nil value)
    stream/router/ip_port.lua:76: in function 'create_router'
    stream/router/ip_port.lua:154: in function 'match'

config_yaml.sync_data leaves `values` nil while `conf_version` is still 0,
because `apisix_yaml[<key>_conf_version] or 0` equals the initial conf_version
and it returns early. `router_ver` starts as nil, so `router_ver ~=
user_routes.conf_version` holds and match() calls create_router(nil). The error
is thrown before router_ver is assigned, so the branch is taken again on the
next connection: the worker never leaves this state on its own.

Treat a missing route list as "no stream route" and let the connection take the
normal no-match path.

The second half is why the worker stayed in that state. Only the polling in
admin/standalone.lua carries configuration across the http/stream boundary, and
it skipped an update whenever X-Last-Modified was unchanged. That field is
ngx_time(), so two updates in the same second are indistinguishable by it: if
the poll ran between them, the second update was dropped for good, and nothing
later re-delivered it because a client re-sending the same content is answered
with "config not changed: same digest". Compare the digest as well, which
changes with the content.

Reproduces without a client: start APISIX in standalone mode with a
stream_proxy port, connect to that port before pushing any configuration, and
the connection is aborted with the trace above.
@dosubot dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. bug Something isn't working labels Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant