WebSocket mid-turn interruptions regressed sharply in 1.23.0-beta.x vs 1.22.0 (new "scope cleanup exceeded its remaining drain budget" warning)

[crystal150]

Summary

After upgrading from 1.22.0 to 1.23.0-beta.5, Codex clients using supports_websockets = true frequently see websocket closed by server before response.completed, followed by reconnect/retry loops.

Server-side, two things changed:

1. A warning that never appeared once on 1.22.0 now fires continuously: Websocket scope cleanup exceeded its remaining drain budget
2. Mid-turn interruption/resend markers rose by roughly 28–35x per request

v1.23.0 (stable) contains no WebSocket changes relative to v1.23.0-beta.5, so this is presumably still present on stable.

This is a fairly high-throughput deployment with a large SQLite store, so it may only show up under load — that might be why it survived the beta soak.

Measured comparison

Same workload, same store, same client fleet across both versions — only the image changed. Normalized per 1,000 POST /v1/responses, each measured over a multi-day window:

marker	1.22.0	1.23.0-beta.5
Websocket scope cleanup exceeded its remaining drain budget	0.00 (never observed)	12.75
websocket_interrupted_tool_outputs_injected	0.04	1.53 (~35x)
websocket_client_previous_response_full_resend_retry_prepared	0.13	3.65 (~28x)

The rate is bursty — it climbs sharply during busy hours, which is consistent with a load-dependent trigger.

Where the warning comes from

app/modules/proxy/_service/websocket/mixin.py:2609

done, _ = await asyncio.wait(
    {cleanup_task},
    timeout=max(float(cleanup_timeout), 0.0),
)
if not done:
    _facade().logger.warning("Websocket scope cleanup exceeded its remaining drain budget")

with (line 2456):

cleanup_timeout = shutdown_state.remaining_drain_timeout_seconds()
if cleanup_timeout is None:
    cleanup_timeout = _facade()._TASK_CANCEL_TIMEOUT_SECONDS

Outside of shutdown, remaining_drain_timeout_seconds() returns None, so the budget falls back to _TASK_CANCEL_TIMEOUT_SECONDS = 1.0 (app/modules/proxy/_service/http_bridge/helpers.py:194) — a hardcoded 1 second with no config or env override.

So on a healthy, non-draining server the finalization path (release_current_account_lease() plus request finalization) gets a fixed 1s and routinely exceeds it. On a busy instance with a large store, 1s seems easy to blow through.

Environment

• codex-lb 1.23.0-beta.5, Docker (ghcr.io/soju06/codex-lb:1.23.0-beta.5)
• SQLite backend
• Client: Codex VS Code extension 0.147.0-alpha.6.5, wire_api = "responses", supports_websockets = true

Notes / limitations

• I could not determine from stdout which side closes first: the distinguishing strings (Downstream websocket disconnected before response.completed / Upstream websocket closed before response.completed / Websocket scope cancelled before response.completed) are passed as error_message into request-log finalization and never reach the container log. I can pull them from the request-log DB if that would help narrow it down.
• Slow finalization writes are a plausible reason a 1s budget is exceeded, but I have not measured finalization latency, so I can't claim that's the cause rather than a correlate.
• fix(proxy): extend websocket stream budget (fix(proxy): extend websocket stream budget #1353, merged 2026-07-27) is already included in beta.5, so it does not address this.

Workaround

Setting supports_websockets = false on the client avoids the WS path entirely.

Questions

1. Is the hardcoded 1.0s cleanup budget intentional for the non-shutdown path, or should it be configurable / derived from load?
2. Is the 28–35x rise in interrupted_tool_outputs_injected / full_resend_retry_prepared a known consequence of a 1.23.0-beta change?

[kevinsslin]

Live follow-up from our codex-lb deployment:

• Production was upgraded from 1.22.0 to stable 1.23.0.
• In the post-upgrade observation window we saw approximately 15 instances of “Websocket scope cleanup exceeded its remaining drain budget”.
• The same window had 559 successful requests and 1 upstream_request_timeout / missing_response_created_timeout at approximately 60 seconds.
• Normal response.created acknowledgement latency was generally below 4 seconds, so the cleanup warning is not explained by ordinary upstream latency.

I opened a focused partial-fix Draft PR: #1723

The PR separates the normal-operation WebSocket scope cleanup observation budget (5 seconds) from the generic 1-second child-task cancellation timeout, preserves the active shutdown drain deadline, and adds a route-level regression. It intentionally does not change the response.created watchdog or retry policy, so the ~60-second missing-ack outcome remains a separate follow-up.