fix: #206 follow-ups: redact-then-cut attempt log, Codex quota codes, MiniMax salvage, slash passwords, linear core scrub; v2.28.1

CHANGELOG.md

@@ -7,6 +7,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [2.28.1] — 2026-10-15
+
+### Fixed
+
+- **A Codex spent balance is `credits-exhausted`, in a stream and over HTTP.** `insufficient_quota`, `credit_balance_exhausted`, `organization_spend_limit_exceeded` and `project_spend_limit_exceeded` (the four codes the Codex CLI maps to a fatal `QuotaExceeded`, openai/codex `codex-rs/codex-api/src/sse/responses_error.rs` at `f82e3659`) classified `codex:stream-error:other` in a stream and `rate-limited` on a `429`, so a spent account cooled down and was retried on a timer. A new `CODEX_QUOTA_CODES` list (`apps/api/src/providers/drivers/openai-oauth-stream-errors.ts`) feeds the stream hook (only from `response.error.code` of a frame whose own JSON `type` is `response.failed`, the one place the CLI reads them, and there even beside a root `code` or `error`; a type-only frame, a root `code` or `error`, a bare `error` event, or a frame typed only by its SSE event name is retried or skipped there, so each stays `server-error`) and two new HTTP rules in `openai-oauth.ts` that apply only on a `429`, as the CLI reads them (the CLI reads the `error` envelope only; the driver also reads the `detail` envelope it reads every Codex code from) (signal `openai-oauth:insufficient_quota`; the CLI's fifth 429 code, `organization_usage_limit_exceeded`, is left as `rate-limited`, since its name says a usage window). The account goes `exhausted`. The CLI never retrying these does not by itself make them permanent (it does not retry a usage limit either); their meaning does, a balance or a spend cap, as on OpenAI's API. No production frame was captured; the test frame is the shape the CLI's own test uses.
+- **An oversized MiniMax in-body failure is no longer read as success.** A MiniMax body over the observation cap went through the salvage scanner, which kept no `base_resp`, so the driver hook saw no envelope and a `1008` (spent balance) on a 200 settled as a success. `base_resp.status_code` is now a salvage slot with its own 32-byte ceiling (`apps/api/src/services/usage/response-salvage-slots.ts`; the scanner is `response-salvage.ts`); the free-text `status_msg` is never kept, so a long one cannot push the code out, and a code over 32 bytes, or one that would cross a full budget, drops itself without evicting other evidence, is released first, before the optional `error` and `code`, when a required member needs room, and is lent to error evidence that needs the room and given back if that evidence drops anyway, rather than failing the salvage; a repeated `base_resp` (or `status_code`) replaces the earlier one, so the last wins, as in `JSON.parse` (for byte-identical keys; escaped keys are #209); a malformed code is dropped, never failing the salvage, and when the frame shows one of two structural violations at depths 1-2 (a second value under one key, or a non-string value where a key or its colon belongs, as in `"status_code":0 1008` or `"base_resp":{...} {"status_code":1008}`), no code is kept from that frame, including one kept earlier. The scanner is not a JSON validator: other malformed frames (for example stray or doubled commas or colons, a wrong closer, errors inside arrays or deeper, bad literals; the full list is on #209) can still yield the code they contain; accepted; losing it reads as before the slot existed. Also fixed in the same scanner: a root key named like an `Object` built-in (`constructor`, `toString`) holding an object threw a `TypeError`; slot lookup now reads own keys only.
+- **The attempt-failure log line redacts before it cuts.** `upstream attempt failed` cut the upstream message to `LOG_REASON_MAX_CHARS` and left redaction to the logger, so a credential across the cut could survive as a fragment. The HTTP path redacts earlier; the Agent-SDK path does not. Both log lines now share one redact-then-cut helper (`apps/api/src/services/dataplane/attempt-log.ts`). This also closes the same gap in 2.25.8's `upstream stream failure` line, whose 4,096-character window could keep a fragment once a long credential before it collapsed to `[REDACTED]`; that window is gone. Up to 64 KiB (the upstream error-body cap) the whole message is redacted before the cut, so every credential is matched with its own terminator and the words before a collapsed token are still logged. Past 64 KiB (an Agent-SDK message is not bounded by the upstream caps) the scan ends at the last whitespace before the 64 KiB mark; no credential's secret part spans whitespace (a `Bearer`/`Basic` gap comes before its token), so none can straddle the cut. A message with no whitespace at all in its first 64 KiB is one token nothing can safely be cut from, so only `…` is logged. A cut never splits a `[REDACTED]` marker, and the quote is re-redacted and pulled back until it fits, so the logger's own pass cannot push it past `LOG_REASON_MAX_CHARS`.
+- **A userinfo password holding an unencoded `/` is scrubbed** (`postgres://u:p/ss@h`), in the server redactor (`apps/api/src/logging/redact.ts`) and the boot-path scrub (`packages/core/src/scrub-credentials.ts`). It is a new first branch beside the unchanged original pattern, so it only adds matches. It is bounded to 256 characters and stops at `:`, so redaction stays linear and cheap; neither its username nor its password crosses `?`, `#`, `&`, `=` or `"`, so it can never swallow the `password=` key of a query-string secret holding an `@` (`postgresql://db:5432/app?password=p@ss`), which the query-string pattern then redacts as before, nor run across a JSON field. A slash password holding one of those characters is not caught, as before, except base64 `=` padding just before the `@`. In the boot-path scrub the padding is not allowed right after a key name of its key=value pattern (`jdbc:postgresql://h:5432/db;password=@S` stays that pattern's), since that pattern anchors on a word boundary rather than `?` or `&`. Price: a URL with a port and an `@` later in its path (`https://host:8080/a@b`) loses its host.
+- **The boot-path credential scrub is linear.** `scrubCredentials` carried the two quadratic shapes 2.25.8 removed from the server redactor; on this host the old code took 12.7 s on the new cost case's inputs (64 KB of `a-`, 64 KB of `eyJ-`, 72 KB of `eyJaaaaa-`, and the `://` runs), and over a minute on the first alone under heavy load. The scheme now stays outside the userinfo match (a linear lookbehind still requires the real scheme `main` required, so `x_y://a:password=@S` stays the key=value pattern's), and the JWT header is bounded with a lookbehind-anchored long-header pattern beside it. Timing cases in `packages/core/test/unit/scrub-credentials.test.ts`. Price, as on the server: a JWT whose header is over 512 characters and is glued straight after `-`, `_` or a letter is no longer scrubbed by this boot-path backstop.
+- Not changed: the original userinfo branch can still match across a JSON field boundary (`"url":"http://h:1","note":"x@y"`). That only mangles log text and leaks nothing; excluding `"` from it would let a password containing `"` through, and a scrubbed line costs less than a leaked one.
+- Tests: `test/unit/dataplane/attempt-log.test.ts` (new), `test/unit/redact.test.ts`, `packages/core/test/unit/scrub-credentials.test.ts`, `test/unit/providers/openai-oauth-stream-errors.test.ts`, `test/unit/providers/openai-oauth.test.ts`, `test/unit/services/usage/response-observer-codex.test.ts`, `test/unit/usage/response-salvage.test.ts`. Refs #206.
+
+### Documentation
+
+- `docs/idea/03-providers.md`: an `openai-oauth` row in the credits-exhausted signal table. `docs/idea/05-routing-and-failover.md`: the quota codes and the `base_resp` salvage slot in the in-stream row. `docs/idea/07-security.md`: slash passwords in the value-shape list. `docs/idea/08-observability.md`: `upstream attempt failed` scrubs before it cuts. `docs/idea/09-deployment.md`: the salvage members and the lent 32 bytes in `UPSTREAM_RESPONSE_OBSERVATION_MAX_BYTES`.
+
 ## [2.28.0] — 2026-10-14
 
 ### Added
@@ -1199,5 +1215,8 @@ your tooling config.
 [2.25.6]: https://github.com/developerz-ai/multi-ai-router/releases/tag/v2.25.6
 [2.25.7]: https://github.com/developerz-ai/multi-ai-router/releases/tag/v2.25.7
 [2.25.8]: https://github.com/developerz-ai/multi-ai-router/releases/tag/v2.25.8
-[2.28.0]: https://github.com/developerz-ai/multi-ai-router/releases/tag/v2.28.0
+[2.26.0]: https://github.com/developerz-ai/multi-ai-router/releases/tag/v2.26.0
+[2.26.1]: https://github.com/developerz-ai/multi-ai-router/releases/tag/v2.26.1
 [2.27.0]: https://github.com/developerz-ai/multi-ai-router/releases/tag/v2.27.0
+[2.28.0]: https://github.com/developerz-ai/multi-ai-router/releases/tag/v2.28.0
+[2.28.1]: https://github.com/developerz-ai/multi-ai-router/releases/tag/v2.28.1

Dockerfile

@@ -205,7 +205,7 @@ LABEL org.opencontainers.image.title="multi-ai-router" \
       org.opencontainers.image.url="https://github.com/developerz-ai/multi-ai-router" \
       org.opencontainers.image.documentation="https://github.com/developerz-ai/multi-ai-router/blob/main/README.md" \
       org.opencontainers.image.licenses="MIT" \
-      org.opencontainers.image.version="2.28.0" \
+      org.opencontainers.image.version="2.28.1" \
       org.opencontainers.image.revision="${ROUTER_REVISION}"
 
 # ---- PID 1 ----

apps/api/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@multi-ai-router/api",
-  "version": "2.28.0",
+  "version": "2.28.1",
   "private": true,
   "type": "module",
   "main": "src/main.ts",

apps/api/src/logging/redact.ts

@@ -100,7 +100,18 @@ const SECRET_VALUE_PATTERNS: readonly ValuePattern[] = [
   // The scheme is left out of the match (it stays in the text untouched): matching it made a long
   // `a-a-a-…` run backtrack quadratically, and bounding it let a scheme over the bound through.
   // The username may be empty: `redis://:password@cache` carries only a password.
-  { pattern: /:\/\/[^\s/:@]*:[^\s/@]+@/g, replacement: `://${REDACTED}@` },
+  // A password may hold an unencoded `/` (`postgres://u:p/ss@h`). The first branch allows it, bounded
+  // to 256 characters and stopping at `:` so a scan from each `://` stays short and the cost linear.
+  // Neither its username nor its password crosses `?`, `#`, `&`, `=` or `"`: it can never cover a
+  // `key=` of a query string (whose secret the query-string pattern below must still see) or run
+  // across a JSON field into an unrelated `@`. So it misses a slash password holding one of those,
+  // except base64 `=` padding just before the `@` (`Zk3/q+Vb9xQ==@`).
+  // The second branch is the original pattern unchanged, so this one only ever adds matches.
+  // Price: `https://host:8080/a@b` loses its host.
+  {
+    pattern: /:\/\/(?:[^\s/:@?#&="]*:[^\s@"?#:&=]{1,256}={0,2}|[^\s/:@]*:[^\s/@]+)@/g,
+    replacement: `://${REDACTED}@`,
+  },
   // A credential passed in a query string, keeping the endpoint and the parameter that carried it.
   // `code`/`code_verifier`/`state` are here because a whole OAuth callback URL logged as one string
   // is not covered by the field-name list — nothing in this router reads a query parameter by those

apps/api/src/providers/drivers/openai-oauth-stream-errors.ts

@@ -13,21 +13,58 @@ import type { FailureClassification, UpstreamFailureKind } from "../types"
  *     under `error` on an `error` event;
  *   - a flat `{type:"error", code, message}`, the code at the top level.
  *
- * The code decides when there is one. Only when there is no code is the nested error object's
+ * The code decides when there is one, with one exception below: a quota code at
+ * `response.error.code` of a JSON-typed `response.failed` frame decides even beside a root `code`
+ * or `error`, as the Codex CLI reads it. Only when there is no code is the nested error object's
  * `type` read, and only for the limit and deactivated kinds (the HTTP rules' lists; unlike them, a
- * code present always decides). The flat
- * event's own `type` is never read: it is the event name `error`, not a code.
+ * code present always decides). A quota kind counts only from `response.error.code` of a frame whose
+ * own JSON `type` is `response.failed`: the only place the Codex CLI's stream parser reads one
+ * (every other shape is retried or skipped there), so no other shape may park the account here.
+ * One deliberate widening: the CLI's typed parse also rejects such a frame when a sibling field of
+ * `response.error` has the wrong type (`plan_type: 42`) and retries it, and skips the whole frame
+ * when a root field it parses has the wrong type (`delta: 42`) or a known root key repeats; the code
+ * alone decides here.
+ * The flat event's own `type` is never read: it is the event name `error`, not a code.
  *
- * The mapping is a closed, stream-only list — the driver's HTTP `rules` are untouched:
+ * The mapping is a closed list. The driver's HTTP `rules` share the limit, deactivated and quota
+ * lists but not the request-shaped one:
  *   - request-shaped codes → `invalid-request` (failover reads `client-error`; no breaker strike);
- *   - the driver's limit codes → `rate-limited`; its deactivated code → `credits-exhausted`;
+ *   - the driver's limit codes → `rate-limited`; its deactivated and quota codes →
+ *     `credits-exhausted`;
  *   - anything else, or nothing → `server-error`, the behaviour before this hook existed.
  * `signal` is `codex:stream-error:<listed code>` or `codex:stream-error:other`, never an upstream
  * string: it becomes the usage row's `errorClass` and the `error_class` metric label.
  */
 
 export const CODEX_LIMIT_CODES: readonly string[] = ["usage_limit_reached", "rate_limit_exceeded"]
 export const CODEX_DEACTIVATED_CODES: readonly string[] = ["account_deactivated"]
+/**
+ * A spent balance or spend cap. Provenance: openai/codex `codex-rs/codex-api/src/sse/responses_error.rs`
+ * (f82e3659, 2026-10-13) maps these four codes to `QuotaExceeded`, which the CLI never retries. That
+ * alone does not separate permanent from clock-recoverable (the CLI does not retry a usage limit
+ * either); the codes' own meaning does: a balance or a spend cap, as `insufficient_quota` means on
+ * OpenAI's API (`openai-api.ts`). A ChatGPT plan's spent window arrives as `usage_limit_reached`
+ * instead. No production frame carrying one has been captured. Blast radius: an account reporting
+ * one goes `exhausted` until a human acts.
+ *
+ * Over HTTP (`openai-oauth.ts` rules): A spent balance arrives as a 429, like OpenAI's API: on
+ * status alone it would cool down and be retried on a timer forever (non-negotiable 7). Only on a
+ * 429, as the Codex CLI reads it (`codex-rs/codex-api/src/api_bridge.rs`), so a stray code on
+ * another status cannot park an account. The CLI reads the `error` envelope only; this driver also
+ * reads the `detail` envelope it reads every other Codex code from (`readCodexFacts`), a deliberate
+ * widening. The CLI's fifth HTTP code there, `organization_usage_limit_exceeded`, is left out: its
+ * name says a usage window, which a 429 already classifies as `rate-limited`. As a `type`, only
+ * `insufficient_quota`, as the CLI reads it; the other three are codes only. One more difference:
+ * these run after the limit rules, so a limit code or a `rate_limit_exceeded` type beside a quota
+ * signal reads `rate-limited` here where the CLI reads quota. That errs to cooldown; the stream
+ * hook follows the CLI instead (quota wins there).
+ */
+export const CODEX_QUOTA_CODES: readonly string[] = [
+  "insufficient_quota",
+  "credit_balance_exhausted",
+  "organization_spend_limit_exceeded",
+  "project_spend_limit_exceeded",
+]
 
 export const CODEX_STREAM_ERROR_REQUEST_CODES: readonly string[] = [
   "invalid_prompt",
@@ -44,13 +81,13 @@ const KIND_BY_CODE: ReadonlyMap<string, UpstreamFailureKind> = new Map([
     "invalid-request",
   ]),
   ...CODEX_LIMIT_CODES.map((code): [string, UpstreamFailureKind] => [code, "rate-limited"]),
-  ...CODEX_DEACTIVATED_CODES.map((code): [string, UpstreamFailureKind] => [
-    code,
-    "credits-exhausted",
-  ]),
+  ...[...CODEX_DEACTIVATED_CODES, ...CODEX_QUOTA_CODES].map(
+    (code): [string, UpstreamFailureKind] => [code, "credits-exhausted"],
+  ),
 ])
 
 const SIGNAL_PREFIX = "codex:stream-error:"
+const QUOTA_CODES: ReadonlySet<string> = new Set(CODEX_QUOTA_CODES)
 
 function record(value: unknown): Record<string, unknown> | undefined {
   return value !== null && typeof value === "object" && !Array.isArray(value)
@@ -96,7 +133,20 @@ export function inspectCodexStreamFrame(
   if (type !== "response.failed" && type !== "error") return null
 
   const nested = record(root.error) ?? record(record(root.response)?.error)
-  const code = listedCode(root, nested)
+  const listed = listedCode(root, nested)
+  // A quota code counts only from where the Codex CLI reads it: `response.error.code` of a frame
+  // whose own JSON `type` is `response.failed` (the CLI's stricter typed parse aside, see above).
+  // Every other shape (a bare `error` event, a root `code` or `error`, a frame typed only by its
+  // SSE event name) is retried or skipped there, so here it must not park the account.
+  const quotaCode =
+    root.type === "response.failed" ? text(record(record(root.response)?.error)?.code) : undefined
+  // That location decides a quota code even when a root `code` or `error` would otherwise win.
+  const code =
+    quotaCode !== undefined && QUOTA_CODES.has(quotaCode)
+      ? quotaCode
+      : listed !== undefined && QUOTA_CODES.has(listed)
+        ? undefined
+        : listed
   const kind = code === undefined ? "server-error" : (KIND_BY_CODE.get(code) ?? "server-error")
   const message = text(root.message) ?? text(nested?.message) ?? text(root.error)
   return {

apps/api/src/providers/drivers/openai-oauth.ts

@@ -19,6 +19,7 @@ import { codexSignal } from "./openai-oauth-quota"
 import {
   CODEX_DEACTIVATED_CODES,
   CODEX_LIMIT_CODES,
+  CODEX_QUOTA_CODES,
   inspectCodexStreamFrame,
 } from "./openai-oauth-stream-errors"
 
@@ -219,6 +220,7 @@ export function readCodexFacts(body: unknown): UpstreamErrorFacts {
 
 const LIMIT_CODES = CODEX_LIMIT_CODES
 const DEACTIVATED_CODES = CODEX_DEACTIVATED_CODES
+const QUOTA_CODES = CODEX_QUOTA_CODES
 
 const CODEX_MODEL_REFUSED = /model is not supported when using codex/i
 
@@ -247,6 +249,12 @@ const codex = createHttpDriver({
     codeRule("rate-limited", "openai-oauth:usage_limit_reached", LIMIT_CODES),
     typeRule("credits-exhausted", "openai-oauth:account_deactivated", DEACTIVATED_CODES),
     codeRule("credits-exhausted", "openai-oauth:account_deactivated", DEACTIVATED_CODES),
+    // A spent balance on a 429; scope and the differences from the Codex CLI: CODEX_QUOTA_CODES.
+    onStatus(
+      [429],
+      typeRule("credits-exhausted", "openai-oauth:insufficient_quota", ["insufficient_quota"]),
+    ),
+    onStatus([429], codeRule("credits-exhausted", "openai-oauth:insufficient_quota", QUOTA_CODES)),
     // "The '<model>' model is not supported when using Codex with a ChatGPT account." (prod
     // 2026-10-04): the model is wrong, not the request, so fail over instead of `invalid-request`.
     onStatus(