codex systemError masks the real turn error as opaque UNKNOWN_UPSTREAM_ERROR

[kaizhou-lab]

Symptom

A codex conversation in AionUi repeatedly fails with only:

{"code":"UNKNOWN_UPSTREAM_ERROR","detail":"codex reported a system error","retryable":true}

The real cause is never shown, so the user cannot tell an AWS-credentials problem from a gateway 403 from a CA issue. (Live case 2026-07-27, conversation 18962466: user suspected CA certificates; the actual error was failed to load AWS credentials.)

Root cause

codex_conn.rs reconcile_terminal: on thread/status/changed → {type:"systemError"} the reader immediately synthesizes a terminal with the hard-coded text "codex reported a system error" and sets terminated. The rich follow-ups that arrive milliseconds later are then absorbed by the terminated guard (I10) and dropped:

• error{willRetry:false} carrying error.message with the real cause
• turn/completed{status:"failed"} carrying turn.error.message with the real cause

The design assumption (from the #609 protocol audit, MED) was "codex does NOT reliably follow systemError with a turn/completed, so synthesize immediately or the FSM hangs Running".

Evidence that the follow-up does arrive

1. Real user session (codex 0.145.0 rollout 019f1215-96fd-7003-a5e2-192c410b924a, 2026-07-27): 6 failed turns, and all 6 (6/6) recorded a task_complete with the full error message (stream disconnected before completion: failed to load AWS credentials: an error occurred while loading credentials; first turn: unexpected status 403 Forbidden: user not allowed to access model …). The UI showed only the opaque code for every one of them.
2. Live wire capture (codex 0.145.0 app-server, bedrock provider with no AWS credentials in the default chain; archived at ~/aion/protocols/samples/codex-cli/0.145.0/systemerror_bedrock_no_creds.jsonl):

   17:58:48.328 error{willRetry:true}  "Reconnecting... 5/5"  (5x transient retries)
   17:59:15.811 thread/status/changed  {"type":"systemError"}
   17:59:15.811 error{willRetry:false} message="stream disconnected before completion: failed to load AWS credentials…"  (same ms)
   17:59:15.816 turn/completed         turn.status="failed", turn.error.message=<same rich message>  (+5 ms)
   

3. Schema (codex 0.145.0 app-server generate-json-schema, v2/ThreadStatusChangedNotification.json): SystemErrorThreadStatus has only type — the status itself can never carry the detail, so deferring to the follow-up is the only way to surface the real cause.

Fix

Defer systemError exactly like the existing idle deferral (system_error_pending): emit nothing, let the fatal error{willRetry:false} or the authoritative turn/completed produce the terminal with the real message. Two bounded fallbacks keep the FSM from hanging Running if the follow-up ever is missing:

• a SYSTEM_ERROR_GRACE (2 s) deadline on the reader (armed only after codex has declared the thread fatally errored — this is not a mid-turn watchdog);
• the EOF flush (process death path) emits the previous opaque terminal.