Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions changelog/unreleased/stream-first-byte-60s.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
### English

- The default first-byte deadline for Qoder and WorkBuddy streams is now 60s (`QODER_STREAM_TTFB_MS`, `WORKBUDDY_STREAM_TTFB_MS`; both were 30s). A slow thinking-model cold start gets longer before the request fails over to the next account. Set either variable to keep the old deadline.

### 中文

- Qoder 和 WorkBuddy 流式请求的默认首字节时限改为 60 秒(`QODER_STREAM_TTFB_MS`、`WORKBUDDY_STREAM_TTFB_MS`,原先都是 30 秒)。思考模型冷启动更慢时,请求会多等一会儿再切换到下一个账号。想保持原来的时限,设置对应的环境变量即可。
2 changes: 1 addition & 1 deletion deploy/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ console; environment variables cannot replace it.
| `QODER_MAX_RETRY_ACCOUNTS` | `4` | Maximum accounts attempted for one request (1-64) |
| `QODER_SSE_DIAGNOSTIC_MODELS` | empty | Comma-separated Qoder model IDs for redacted SSE diagnostics in Runtime Logs; `*` enables all |
| `QODER_WORKER_BASE_PORT` | `32100` | Internal child-runtime port range |
| `QODER_STREAM_TTFB_MS` | `30000` | Qoder stream first-byte deadline in ms (response head + first meaningful SSE data line after 200), Go side and worker; min `5000`; timeout fails over to the next account |
| `QODER_STREAM_TTFB_MS` | `60000` | Qoder stream first-byte deadline in ms (response head + first meaningful SSE data line after 200), Go side and worker; min `5000`; timeout fails over to the next account |
| `QODER_STREAM_TOTAL_MS` | `300000` | Qoder worker mid-stream stall timeout in ms: aborts only after this long with no new chunk (resets per chunk, not a wall-clock cap); never below the TTFB value |
| `QODER_SIGNATURE_SECRET` | built-in | Override the COSY request-signing secret for in-process Qoder; leave unset unless upstream rotates it |
| `QODER_PROXY_URL` | empty | Initial global outbound proxy: `http(s)://`, `direct`, or `none`; saved console settings take precedence |
Expand Down
4 changes: 2 additions & 2 deletions internal/providers/qoder/worker.go
Original file line number Diff line number Diff line change
Expand Up @@ -57,14 +57,14 @@ func (e TransportError) Unwrap() error { return e.Err }
// timer is stopped and the SSE body streams under the parent context's lifetime,
// so long thinking-model output is not cut off. A stuck upstream that never
// sends a first byte surfaces as ErrFirstByteTimeout so the executor can fail
// over to the next account instead of hanging forever. 30s covers slow
// over to the next account instead of hanging forever. 60s covers slow
// cold-starts and thinking-model first-token latency while keeping a
// stuck-upstream failover inside one round-trip of the client's patience.
//
// The effective value is resolved once at construction (see
// resolveFirstByteTimeout) and can be tuned per deployment with
// QODER_STREAM_TTFB_MS, mirroring the worker daemon's own knob.
const defaultFirstByteTimeout = 30 * time.Second
const defaultFirstByteTimeout = 60 * time.Second

// resolveFirstByteTimeout reads QODER_STREAM_TTFB_MS, falling back to
// defaultFirstByteTimeout for an absent or invalid value. The floor matches the
Expand Down
2 changes: 1 addition & 1 deletion internal/providers/workbuddy/ttfb.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ import (
// relay waits until the client gives up, the turn is spent, and a retry lands on
// the same account. Qoder already fails over on this (QODER_STREAM_TTFB_MS); this
// is the WorkBuddy counterpart.
const defaultFirstByteTimeout = 30 * time.Second
const defaultFirstByteTimeout = 60 * time.Second

// workbuddyStreamTTFBEnv overrides the first-byte deadline.
const workbuddyStreamTTFBEnv = "WORKBUDDY_STREAM_TTFB_MS"
Expand Down
2 changes: 1 addition & 1 deletion worker/src/upstream.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
// whole window is aborted. The TTFB default mirrors the Go-side first-byte
// deadline so worker and gateway agree on when a stuck head counts as a
// failure; both are tunable per deployment.
export const streamTtfbMs = Math.max(5_000, Number(process.env.QODER_STREAM_TTFB_MS || 30_000) || 30_000);
export const streamTtfbMs = Math.max(5_000, Number(process.env.QODER_STREAM_TTFB_MS || 60_000) || 60_000);
export const streamStallMs = Math.max(streamTtfbMs, Number(process.env.QODER_STREAM_TOTAL_MS || 300_000) || 300_000);

// fetchWithTimeout wraps fetch with a TTFB deadline on the response head and
Expand Down
Loading