Skip to content

HTTP bridge cooldown startup 503s can exhaust Codex retry budget for hard turn-state recovery #1737

Description

@kevinsslin

Problem

A hard turn-state HTTP bridge can experience two consecutive eventless upstream failures:

  1. missing_response_created_timeout fires with one pending request.
  2. The bounded same-anchor recovery runs.
  3. The replacement socket later fails with response_events_seen=0.
  4. A subsequent eventless timeout records the second retry-circuit failure and opens a 60-second cooldown.
  5. Codex Desktop retries the same turn within milliseconds instead of honoring the full HTTP Retry-After / SSE retry: hint.
  6. Startup returns repeated 503 upstream_request_timeout responses until Codex exhausts its retry budget and pauses the task.

The proxy process remains healthy throughout (NRestarts=0, health 200), and this is not account stream-cap saturation.

Root cause

Turn-state-only hard continuity can omit previous_response_id while still carrying a real Codex turn-state anchor. Its replay identity is protected by the durable operation ledger, but the startup cooldown guard runs before operation registration. It classifies the request as continuity-bound without safe replay and returns 503 before the ledger can serialize a bounded recovery attempt.

The server already emits retry hints. The observed client does not wait for the entire cooldown, so additional hinting alone does not preserve the client retry budget.

Proposed bounded behavior

When server_anchored_replay_once (or the explicit indefinite mode) is enabled:

  • hold a zero-event hard turn-state request through cooldown only when a live durable session id and owner epoch exist;
  • dispatch nothing upstream while waiting;
  • clamp the wait to the request budget;
  • after cooldown, run the normal durable operation-ledger lookup and atomic recovery claim;
  • preserve the existing one-shot maximum recovery dispatch;
  • fail closed when durable ownership is unavailable, any response event/output exists, the request budget expires, or the default fail_closed mode is active.

Acceptance criteria

  • One client request remains alive through cooldown instead of receiving a startup 503 storm.
  • No upstream response.create is sent during the wait.
  • One-shot mode cannot claim or dispatch the same unknown operation more than once.
  • Missing durable owner proof and expired request budgets remain fail closed.
  • Telemetry distinguishes the operation-fenced cooldown wait.
  • Default behavior does not change.

A tested implementation is available from fork commit 5ef5d366; an upstream Draft PR will link this issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions