feat(ai-proxy): support phase-specific HTTP timeouts - #13801
Open
ruanweihong-aaa wants to merge 2 commits into
Open
feat(ai-proxy): support phase-specific HTTP timeouts#13801ruanweihong-aaa wants to merge 2 commits into
ruanweihong-aaa wants to merge 2 commits into
Conversation
Add optional connect, send, and read timeout settings for ai-proxy and ai-proxy-multi while preserving the existing timeout fallback behavior.
nic-6443
reviewed
Aug 11, 2026
nic-6443
marked this pull request as ready for review
August 14, 2026 08:56
dosubot
Bot
added
size:L
nic-6443
approved these changes
Aug 14, 2026
There was a problem hiding this comment.
Pull request overview
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Adds phase-specific HTTP timeouts to ai-proxy / ai-proxy-multi while preserving the legacy single timeout behavior for backward compatibility.
Changes:
- Adds
connect_timeout,send_timeout, andread_timeoutto plugin schemas and documentation (EN/ZH). - Updates the AI HTTP transport to support
lua-resty-httpset_timeouts(connect, send, read)alongside the existingset_timeout(timeout)path. - Extends test coverage for schema boundaries, transport dispatch, and read-timeout regression behavior.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| t/plugin/ai-transport-http.t | Adds transport-level tests asserting set_timeouts vs set_timeout dispatch. |
| t/plugin/ai-proxy.t | Adds schema boundary tests for new fields + delayed-upstream read-timeout regression tests. |
| t/plugin/ai-proxy-multi.t | Adds schema boundary tests for new fields. |
| docs/zh/latest/plugins/ai-proxy.md | Documents new timeout fields and fallback behavior (ZH). |
| docs/zh/latest/plugins/ai-proxy-multi.md | Documents new timeout fields and fallback behavior (ZH). |
| docs/en/latest/plugins/ai-proxy.md | Documents new timeout fields and fallback behavior (EN). |
| docs/en/latest/plugins/ai-proxy-multi.md | Documents new timeout fields and fallback behavior (EN). |
| apisix/plugins/ai-transport/http.lua | Implements timeout table support via set_timeouts with legacy fallback. |
| apisix/plugins/ai-proxy/schema.lua | Adds schema fields for phase timeouts to both proxy schemas. |
| apisix/plugins/ai-proxy/base.lua | Builds connect/send/read timeout tuple (with fallback) at transport call site. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+362
to
+399
| content_by_lua_block { | ||
| local orig_http = package.loaded["resty.http"] | ||
| local orig_transport = package.loaded["apisix.plugins.ai-transport.http"] | ||
|
|
||
| package.loaded["resty.http"] = { | ||
| new = function() | ||
| return { | ||
| set_timeout = function() | ||
| error("legacy set_timeout must not be used") | ||
| end, | ||
| set_timeouts = function(_, connect, send, read) | ||
| ngx.say(connect, ",", send, ",", read) | ||
| end, | ||
| connect = function() return true end, | ||
| request = function() return {headers = {}, status = 200} end, | ||
| } | ||
| end, | ||
| } | ||
|
|
||
| package.loaded["apisix.plugins.ai-transport.http"] = nil | ||
| local transport = require("apisix.plugins.ai-transport.http") | ||
| local res, err = transport.request({ | ||
| host = "127.0.0.1", | ||
| port = 80, | ||
| path = "/", | ||
| body = {}, | ||
| }, { | ||
| connect_timeout = 101, | ||
| send_timeout = 202, | ||
| read_timeout = 303, | ||
| }) | ||
| if not res then | ||
| ngx.say(err) | ||
| end | ||
|
|
||
| package.loaded["resty.http"] = orig_http | ||
| package.loaded["apisix.plugins.ai-transport.http"] = orig_transport | ||
| } |
Comment on lines
+409
to
+442
| content_by_lua_block { | ||
| local orig_http = package.loaded["resty.http"] | ||
| local orig_transport = package.loaded["apisix.plugins.ai-transport.http"] | ||
|
|
||
| package.loaded["resty.http"] = { | ||
| new = function() | ||
| return { | ||
| set_timeout = function(_, timeout) | ||
| ngx.say(timeout) | ||
| end, | ||
| set_timeouts = function() | ||
| error("set_timeouts must not be used for numeric callers") | ||
| end, | ||
| connect = function() return true end, | ||
| request = function() return {headers = {}, status = 200} end, | ||
| } | ||
| end, | ||
| } | ||
|
|
||
| package.loaded["apisix.plugins.ai-transport.http"] = nil | ||
| local transport = require("apisix.plugins.ai-transport.http") | ||
| local res, err = transport.request({ | ||
| host = "127.0.0.1", | ||
| port = 80, | ||
| path = "/", | ||
| body = {}, | ||
| }, 456) | ||
| if not res then | ||
| ngx.say(err) | ||
| end | ||
|
|
||
| package.loaded["resty.http"] = orig_http | ||
| package.loaded["apisix.plugins.ai-transport.http"] = orig_transport | ||
| } |
| | logging.summaries | boolean | False | false | | If true, logs request LLM model, duration, request, and response tokens. | | ||
| | logging.payloads | boolean | False | false | | If true, logs request and response payload. | | ||
| | timeout | integer | False | 30000 | 1 - 600000 | Request timeout in milliseconds when requesting the LLM service. Applied per socket operation (connect / send / read block); does not cap the total duration of a streaming response. | | ||
| | timeout | integer | False | 30000 | 1 - 600000 | Default request timeout in milliseconds when requesting the LLM service. It remains the fallback for each phase below and is used for all phases when no phase-specific timeout is configured. | |
Comment on lines
+350
to
+360
| maximum = 600000, | ||
| }, | ||
| send_timeout = { | ||
| type = "integer", | ||
| minimum = 1, | ||
| maximum = 600000, | ||
| }, | ||
| read_timeout = { | ||
| type = "integer", | ||
| minimum = 1, | ||
| maximum = 600000, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
ai-proxyandai-proxy-multicurrently expose a singletimeoutvalue that is applied to the HTTP connect, send, and read phases. This makes it difficult to fail fast during connection establishment or request upload while allowing a longer timeout for LLM response generation.This PR adds three optional phase-specific timeout fields:
connect_timeout: timeout for establishing the upstream connection;send_timeout: timeout for sending the request to the upstream;read_timeout: timeout for each socket read from the upstream.All three fields are expressed in milliseconds and accept integer values from
1to600000.The change preserves backward compatibility:
set_timeout(timeout)path;timeoutvalue;Implementation details:
ai-proxyandai-proxy-multischemas;ai-proxytransport call site;lua-resty-http'sset_timeouts(connect, send, read)for phase-specific configuration while retaining the originalset_timeout(timeout)compatibility path;Compatibility
The new fields are optional. Existing users who only configure
timeout, or rely on its default value, continue to use the original code path. The upper bound of each new field is the same as the existingtimeoutupper bound.read_timeoutis a per-socket-read timeout and is not a total wall-clock limit for streaming responses. Existingmax_stream_duration_msandmax_response_bytescontrols remain responsible for total streaming duration and response-size limits.Verification
Targeted, source-matched Docker tests:
ai-proxyandai-proxy-multi, including valid boundaries and invalid values;set_timeouts(connect, send, read)and compatibility with the numericset_timeout(timeout)path;read_timeoutconfigured, exercising fallback to the existingtimeoutvalue for the omitted phases;Files=3, Tests=63, Result: PASS.Additional local dual-Gateway A/B verification used identical etcd, routes, Docker network, and controlled upstream fixtures for the pre-fix and fixed images:
200after about250 ms; fixed returned504after about50 mswithread_timeout=50;1 s; fixed timed out after about50 mswithconnect_timeout=50;60 MiBrequest and an upstream that accepted headers but did not consume the body, the fixed image timed out significantly earlier withsend_timeout=50;These results cover the tests related to this change; they are not a claim that the complete APISIX test suite was executed locally.
Which issue(s) this PR fixes:
Fixes #12072
Checklist