diff --git a/CHANGELOG.md b/CHANGELOG.md index c9ba571f..57fa4247 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [2.28.1] — 2026-10-15 + +### Fixed + +- **A Codex spent balance is `credits-exhausted`, in a stream and over HTTP.** `insufficient_quota`, `credit_balance_exhausted`, `organization_spend_limit_exceeded` and `project_spend_limit_exceeded` (the four codes the Codex CLI maps to a fatal `QuotaExceeded`, openai/codex `codex-rs/codex-api/src/sse/responses_error.rs` at `f82e3659`) classified `codex:stream-error:other` in a stream and `rate-limited` on a `429`, so a spent account cooled down and was retried on a timer. A new `CODEX_QUOTA_CODES` list (`apps/api/src/providers/drivers/openai-oauth-stream-errors.ts`) feeds the stream hook (only from `response.error.code` of a frame whose own JSON `type` is `response.failed`, the one place the CLI reads them, and there even beside a root `code` or `error`; a type-only frame, a root `code` or `error`, a bare `error` event, or a frame typed only by its SSE event name is retried or skipped there, so each stays `server-error`) and two new HTTP rules in `openai-oauth.ts` that apply only on a `429`, as the CLI reads them (the CLI reads the `error` envelope only; the driver also reads the `detail` envelope it reads every Codex code from) (signal `openai-oauth:insufficient_quota`; the CLI's fifth 429 code, `organization_usage_limit_exceeded`, is left as `rate-limited`, since its name says a usage window). The account goes `exhausted`. The CLI never retrying these does not by itself make them permanent (it does not retry a usage limit either); their meaning does, a balance or a spend cap, as on OpenAI's API. No production frame was captured; the test frame is the shape the CLI's own test uses. +- **An oversized MiniMax in-body failure is no longer read as success.** A MiniMax body over the observation cap went through the salvage scanner, which kept no `base_resp`, so the driver hook saw no envelope and a `1008` (spent balance) on a 200 settled as a success. `base_resp.status_code` is now a salvage slot with its own 32-byte ceiling (`apps/api/src/services/usage/response-salvage-slots.ts`; the scanner is `response-salvage.ts`); the free-text `status_msg` is never kept, so a long one cannot push the code out, and a code over 32 bytes, or one that would cross a full budget, drops itself without evicting other evidence, is released first, before the optional `error` and `code`, when a required member needs room, and is lent to error evidence that needs the room and given back if that evidence drops anyway, rather than failing the salvage; a repeated `base_resp` (or `status_code`) replaces the earlier one, so the last wins, as in `JSON.parse` (for byte-identical keys; escaped keys are #209); a malformed code is dropped, never failing the salvage, and when the frame shows one of two structural violations at depths 1-2 (a second value under one key, or a non-string value where a key or its colon belongs, as in `"status_code":0 1008` or `"base_resp":{...} {"status_code":1008}`), no code is kept from that frame, including one kept earlier. The scanner is not a JSON validator: other malformed frames (for example stray or doubled commas or colons, a wrong closer, errors inside arrays or deeper, bad literals; the full list is on #209) can still yield the code they contain; accepted; losing it reads as before the slot existed. Also fixed in the same scanner: a root key named like an `Object` built-in (`constructor`, `toString`) holding an object threw a `TypeError`; slot lookup now reads own keys only. +- **The attempt-failure log line redacts before it cuts.** `upstream attempt failed` cut the upstream message to `LOG_REASON_MAX_CHARS` and left redaction to the logger, so a credential across the cut could survive as a fragment. The HTTP path redacts earlier; the Agent-SDK path does not. Both log lines now share one redact-then-cut helper (`apps/api/src/services/dataplane/attempt-log.ts`). This also closes the same gap in 2.25.8's `upstream stream failure` line, whose 4,096-character window could keep a fragment once a long credential before it collapsed to `[REDACTED]`; that window is gone. Up to 64 KiB (the upstream error-body cap) the whole message is redacted before the cut, so every credential is matched with its own terminator and the words before a collapsed token are still logged. Past 64 KiB (an Agent-SDK message is not bounded by the upstream caps) the scan ends at the last whitespace before the 64 KiB mark; no credential's secret part spans whitespace (a `Bearer`/`Basic` gap comes before its token), so none can straddle the cut. A message with no whitespace at all in its first 64 KiB is one token nothing can safely be cut from, so only `…` is logged. A cut never splits a `[REDACTED]` marker, and the quote is re-redacted and pulled back until it fits, so the logger's own pass cannot push it past `LOG_REASON_MAX_CHARS`. +- **A userinfo password holding an unencoded `/` is scrubbed** (`postgres://u:p/ss@h`), in the server redactor (`apps/api/src/logging/redact.ts`) and the boot-path scrub (`packages/core/src/scrub-credentials.ts`). It is a new first branch beside the unchanged original pattern, so it only adds matches. It is bounded to 256 characters and stops at `:`, so redaction stays linear and cheap; neither its username nor its password crosses `?`, `#`, `&`, `=` or `"`, so it can never swallow the `password=` key of a query-string secret holding an `@` (`postgresql://db:5432/app?password=p@ss`), which the query-string pattern then redacts as before, nor run across a JSON field. A slash password holding one of those characters is not caught, as before, except base64 `=` padding just before the `@`. In the boot-path scrub the padding is not allowed right after a key name of its key=value pattern (`jdbc:postgresql://h:5432/db;password=@S` stays that pattern's), since that pattern anchors on a word boundary rather than `?` or `&`. Price: a URL with a port and an `@` later in its path (`https://host:8080/a@b`) loses its host. +- **The boot-path credential scrub is linear.** `scrubCredentials` carried the two quadratic shapes 2.25.8 removed from the server redactor; on this host the old code took 12.7 s on the new cost case's inputs (64 KB of `a-`, 64 KB of `eyJ-`, 72 KB of `eyJaaaaa-`, and the `://` runs), and over a minute on the first alone under heavy load. The scheme now stays outside the userinfo match (a linear lookbehind still requires the real scheme `main` required, so `x_y://a:password=@S` stays the key=value pattern's), and the JWT header is bounded with a lookbehind-anchored long-header pattern beside it. Timing cases in `packages/core/test/unit/scrub-credentials.test.ts`. Price, as on the server: a JWT whose header is over 512 characters and is glued straight after `-`, `_` or a letter is no longer scrubbed by this boot-path backstop. +- Not changed: the original userinfo branch can still match across a JSON field boundary (`"url":"http://h:1","note":"x@y"`). That only mangles log text and leaks nothing; excluding `"` from it would let a password containing `"` through, and a scrubbed line costs less than a leaked one. +- Tests: `test/unit/dataplane/attempt-log.test.ts` (new), `test/unit/redact.test.ts`, `packages/core/test/unit/scrub-credentials.test.ts`, `test/unit/providers/openai-oauth-stream-errors.test.ts`, `test/unit/providers/openai-oauth.test.ts`, `test/unit/services/usage/response-observer-codex.test.ts`, `test/unit/usage/response-salvage.test.ts`. Refs #206. + +### Documentation + +- `docs/idea/03-providers.md`: an `openai-oauth` row in the credits-exhausted signal table. `docs/idea/05-routing-and-failover.md`: the quota codes and the `base_resp` salvage slot in the in-stream row. `docs/idea/07-security.md`: slash passwords in the value-shape list. `docs/idea/08-observability.md`: `upstream attempt failed` scrubs before it cuts. `docs/idea/09-deployment.md`: the salvage members and the lent 32 bytes in `UPSTREAM_RESPONSE_OBSERVATION_MAX_BYTES`. + ## [2.28.0] — 2026-10-14 ### Added @@ -1199,5 +1215,8 @@ your tooling config. [2.25.6]: https://github.com/developerz-ai/multi-ai-router/releases/tag/v2.25.6 [2.25.7]: https://github.com/developerz-ai/multi-ai-router/releases/tag/v2.25.7 [2.25.8]: https://github.com/developerz-ai/multi-ai-router/releases/tag/v2.25.8 -[2.28.0]: https://github.com/developerz-ai/multi-ai-router/releases/tag/v2.28.0 +[2.26.0]: https://github.com/developerz-ai/multi-ai-router/releases/tag/v2.26.0 +[2.26.1]: https://github.com/developerz-ai/multi-ai-router/releases/tag/v2.26.1 [2.27.0]: https://github.com/developerz-ai/multi-ai-router/releases/tag/v2.27.0 +[2.28.0]: https://github.com/developerz-ai/multi-ai-router/releases/tag/v2.28.0 +[2.28.1]: https://github.com/developerz-ai/multi-ai-router/releases/tag/v2.28.1 diff --git a/Dockerfile b/Dockerfile index 1ab16518..8ec43949 100644 --- a/Dockerfile +++ b/Dockerfile @@ -205,7 +205,7 @@ LABEL org.opencontainers.image.title="multi-ai-router" \ org.opencontainers.image.url="https://github.com/developerz-ai/multi-ai-router" \ org.opencontainers.image.documentation="https://github.com/developerz-ai/multi-ai-router/blob/main/README.md" \ org.opencontainers.image.licenses="MIT" \ - org.opencontainers.image.version="2.28.0" \ + org.opencontainers.image.version="2.28.1" \ org.opencontainers.image.revision="${ROUTER_REVISION}" # ---- PID 1 ---- diff --git a/apps/api/package.json b/apps/api/package.json index 759e0b8d..a094a1cb 100644 --- a/apps/api/package.json +++ b/apps/api/package.json @@ -1,6 +1,6 @@ { "name": "@multi-ai-router/api", - "version": "2.28.0", + "version": "2.28.1", "private": true, "type": "module", "main": "src/main.ts", diff --git a/apps/api/src/logging/redact.ts b/apps/api/src/logging/redact.ts index 036d1a7c..93f128d3 100644 --- a/apps/api/src/logging/redact.ts +++ b/apps/api/src/logging/redact.ts @@ -100,7 +100,18 @@ const SECRET_VALUE_PATTERNS: readonly ValuePattern[] = [ // The scheme is left out of the match (it stays in the text untouched): matching it made a long // `a-a-a-…` run backtrack quadratically, and bounding it let a scheme over the bound through. // The username may be empty: `redis://:password@cache` carries only a password. - { pattern: /:\/\/[^\s/:@]*:[^\s/@]+@/g, replacement: `://${REDACTED}@` }, + // A password may hold an unencoded `/` (`postgres://u:p/ss@h`). The first branch allows it, bounded + // to 256 characters and stopping at `:` so a scan from each `://` stays short and the cost linear. + // Neither its username nor its password crosses `?`, `#`, `&`, `=` or `"`: it can never cover a + // `key=` of a query string (whose secret the query-string pattern below must still see) or run + // across a JSON field into an unrelated `@`. So it misses a slash password holding one of those, + // except base64 `=` padding just before the `@` (`Zk3/q+Vb9xQ==@`). + // The second branch is the original pattern unchanged, so this one only ever adds matches. + // Price: `https://host:8080/a@b` loses its host. + { + pattern: /:\/\/(?:[^\s/:@?#&="]*:[^\s@"?#:&=]{1,256}={0,2}|[^\s/:@]*:[^\s/@]+)@/g, + replacement: `://${REDACTED}@`, + }, // A credential passed in a query string, keeping the endpoint and the parameter that carried it. // `code`/`code_verifier`/`state` are here because a whole OAuth callback URL logged as one string // is not covered by the field-name list — nothing in this router reads a query parameter by those diff --git a/apps/api/src/providers/drivers/openai-oauth-stream-errors.ts b/apps/api/src/providers/drivers/openai-oauth-stream-errors.ts index 6a272f21..a5f62072 100644 --- a/apps/api/src/providers/drivers/openai-oauth-stream-errors.ts +++ b/apps/api/src/providers/drivers/openai-oauth-stream-errors.ts @@ -13,14 +13,24 @@ import type { FailureClassification, UpstreamFailureKind } from "../types" * under `error` on an `error` event; * - a flat `{type:"error", code, message}`, the code at the top level. * - * The code decides when there is one. Only when there is no code is the nested error object's + * The code decides when there is one, with one exception below: a quota code at + * `response.error.code` of a JSON-typed `response.failed` frame decides even beside a root `code` + * or `error`, as the Codex CLI reads it. Only when there is no code is the nested error object's * `type` read, and only for the limit and deactivated kinds (the HTTP rules' lists; unlike them, a - * code present always decides). The flat - * event's own `type` is never read: it is the event name `error`, not a code. + * code present always decides). A quota kind counts only from `response.error.code` of a frame whose + * own JSON `type` is `response.failed`: the only place the Codex CLI's stream parser reads one + * (every other shape is retried or skipped there), so no other shape may park the account here. + * One deliberate widening: the CLI's typed parse also rejects such a frame when a sibling field of + * `response.error` has the wrong type (`plan_type: 42`) and retries it, and skips the whole frame + * when a root field it parses has the wrong type (`delta: 42`) or a known root key repeats; the code + * alone decides here. + * The flat event's own `type` is never read: it is the event name `error`, not a code. * - * The mapping is a closed, stream-only list — the driver's HTTP `rules` are untouched: + * The mapping is a closed list. The driver's HTTP `rules` share the limit, deactivated and quota + * lists but not the request-shaped one: * - request-shaped codes → `invalid-request` (failover reads `client-error`; no breaker strike); - * - the driver's limit codes → `rate-limited`; its deactivated code → `credits-exhausted`; + * - the driver's limit codes → `rate-limited`; its deactivated and quota codes → + * `credits-exhausted`; * - anything else, or nothing → `server-error`, the behaviour before this hook existed. * `signal` is `codex:stream-error:` or `codex:stream-error:other`, never an upstream * string: it becomes the usage row's `errorClass` and the `error_class` metric label. @@ -28,6 +38,33 @@ import type { FailureClassification, UpstreamFailureKind } from "../types" export const CODEX_LIMIT_CODES: readonly string[] = ["usage_limit_reached", "rate_limit_exceeded"] export const CODEX_DEACTIVATED_CODES: readonly string[] = ["account_deactivated"] +/** + * A spent balance or spend cap. Provenance: openai/codex `codex-rs/codex-api/src/sse/responses_error.rs` + * (f82e3659, 2026-10-13) maps these four codes to `QuotaExceeded`, which the CLI never retries. That + * alone does not separate permanent from clock-recoverable (the CLI does not retry a usage limit + * either); the codes' own meaning does: a balance or a spend cap, as `insufficient_quota` means on + * OpenAI's API (`openai-api.ts`). A ChatGPT plan's spent window arrives as `usage_limit_reached` + * instead. No production frame carrying one has been captured. Blast radius: an account reporting + * one goes `exhausted` until a human acts. + * + * Over HTTP (`openai-oauth.ts` rules): A spent balance arrives as a 429, like OpenAI's API: on + * status alone it would cool down and be retried on a timer forever (non-negotiable 7). Only on a + * 429, as the Codex CLI reads it (`codex-rs/codex-api/src/api_bridge.rs`), so a stray code on + * another status cannot park an account. The CLI reads the `error` envelope only; this driver also + * reads the `detail` envelope it reads every other Codex code from (`readCodexFacts`), a deliberate + * widening. The CLI's fifth HTTP code there, `organization_usage_limit_exceeded`, is left out: its + * name says a usage window, which a 429 already classifies as `rate-limited`. As a `type`, only + * `insufficient_quota`, as the CLI reads it; the other three are codes only. One more difference: + * these run after the limit rules, so a limit code or a `rate_limit_exceeded` type beside a quota + * signal reads `rate-limited` here where the CLI reads quota. That errs to cooldown; the stream + * hook follows the CLI instead (quota wins there). + */ +export const CODEX_QUOTA_CODES: readonly string[] = [ + "insufficient_quota", + "credit_balance_exhausted", + "organization_spend_limit_exceeded", + "project_spend_limit_exceeded", +] export const CODEX_STREAM_ERROR_REQUEST_CODES: readonly string[] = [ "invalid_prompt", @@ -44,13 +81,13 @@ const KIND_BY_CODE: ReadonlyMap = new Map([ "invalid-request", ]), ...CODEX_LIMIT_CODES.map((code): [string, UpstreamFailureKind] => [code, "rate-limited"]), - ...CODEX_DEACTIVATED_CODES.map((code): [string, UpstreamFailureKind] => [ - code, - "credits-exhausted", - ]), + ...[...CODEX_DEACTIVATED_CODES, ...CODEX_QUOTA_CODES].map( + (code): [string, UpstreamFailureKind] => [code, "credits-exhausted"], + ), ]) const SIGNAL_PREFIX = "codex:stream-error:" +const QUOTA_CODES: ReadonlySet = new Set(CODEX_QUOTA_CODES) function record(value: unknown): Record | undefined { return value !== null && typeof value === "object" && !Array.isArray(value) @@ -96,7 +133,20 @@ export function inspectCodexStreamFrame( if (type !== "response.failed" && type !== "error") return null const nested = record(root.error) ?? record(record(root.response)?.error) - const code = listedCode(root, nested) + const listed = listedCode(root, nested) + // A quota code counts only from where the Codex CLI reads it: `response.error.code` of a frame + // whose own JSON `type` is `response.failed` (the CLI's stricter typed parse aside, see above). + // Every other shape (a bare `error` event, a root `code` or `error`, a frame typed only by its + // SSE event name) is retried or skipped there, so here it must not park the account. + const quotaCode = + root.type === "response.failed" ? text(record(record(root.response)?.error)?.code) : undefined + // That location decides a quota code even when a root `code` or `error` would otherwise win. + const code = + quotaCode !== undefined && QUOTA_CODES.has(quotaCode) + ? quotaCode + : listed !== undefined && QUOTA_CODES.has(listed) + ? undefined + : listed const kind = code === undefined ? "server-error" : (KIND_BY_CODE.get(code) ?? "server-error") const message = text(root.message) ?? text(nested?.message) ?? text(root.error) return { diff --git a/apps/api/src/providers/drivers/openai-oauth.ts b/apps/api/src/providers/drivers/openai-oauth.ts index 165da113..77db3a90 100644 --- a/apps/api/src/providers/drivers/openai-oauth.ts +++ b/apps/api/src/providers/drivers/openai-oauth.ts @@ -19,6 +19,7 @@ import { codexSignal } from "./openai-oauth-quota" import { CODEX_DEACTIVATED_CODES, CODEX_LIMIT_CODES, + CODEX_QUOTA_CODES, inspectCodexStreamFrame, } from "./openai-oauth-stream-errors" @@ -219,6 +220,7 @@ export function readCodexFacts(body: unknown): UpstreamErrorFacts { const LIMIT_CODES = CODEX_LIMIT_CODES const DEACTIVATED_CODES = CODEX_DEACTIVATED_CODES +const QUOTA_CODES = CODEX_QUOTA_CODES const CODEX_MODEL_REFUSED = /model is not supported when using codex/i @@ -247,6 +249,12 @@ const codex = createHttpDriver({ codeRule("rate-limited", "openai-oauth:usage_limit_reached", LIMIT_CODES), typeRule("credits-exhausted", "openai-oauth:account_deactivated", DEACTIVATED_CODES), codeRule("credits-exhausted", "openai-oauth:account_deactivated", DEACTIVATED_CODES), + // A spent balance on a 429; scope and the differences from the Codex CLI: CODEX_QUOTA_CODES. + onStatus( + [429], + typeRule("credits-exhausted", "openai-oauth:insufficient_quota", ["insufficient_quota"]), + ), + onStatus([429], codeRule("credits-exhausted", "openai-oauth:insufficient_quota", QUOTA_CODES)), // "The '' model is not supported when using Codex with a ChatGPT account." (prod // 2026-10-04): the model is wrong, not the request, so fail over instead of `invalid-request`. onStatus( diff --git a/apps/api/src/services/dataplane/attempt-log.ts b/apps/api/src/services/dataplane/attempt-log.ts index c566fc1a..b7bdcb76 100644 --- a/apps/api/src/services/dataplane/attempt-log.ts +++ b/apps/api/src/services/dataplane/attempt-log.ts @@ -1,5 +1,5 @@ import type { Logger } from "../../logging/logger" -import { redactValue } from "../../logging/redact" +import { REDACTED, redactValue } from "../../logging/redact" import type { AttemptOutcome } from "./attempt" /** @@ -9,9 +9,9 @@ import type { AttemptOutcome } from "./attempt" * pool failed every request: the SDK's own "Failed to authenticate" and the classifier's * `claude-sdk:unclassified` never reached the log. So the line carries three more things — * `signal`, which rule classified the failure; `reason`, the router-authored sentence the client is - * owed; and `upstreamMessage`, the provider's (or the SDK's) own complaint, bounded here and - * scrubbed by the logger (`logging/redact.ts`), which is the only thing between an upstream's echo - * and the line. + * owed; and `upstreamMessage`, the provider's (or the SDK's) own complaint, scrubbed and then + * bounded here (`scrubbedQuote`). The HTTP path redacts the message once already; the Agent-SDK + * path does not, so this line cannot rely on it. */ export function logAttemptFailure( log: Logger | undefined, @@ -27,22 +27,15 @@ export function logAttemptFailure( failureKind: outcome.failure.kind, signal: outcome.classification?.signal, reason: outcome.failure.message, - upstreamMessage: bounded(outcome.classification?.message, reasonMaxChars), + upstreamMessage: scrubbedQuote(outcome.classification?.message, reasonMaxChars), }) } -/** - * Characters past the cut still scrubbed, so a credential straddling it is matched whole: room for a - * ChatGPT access token (a JWT of under 2 KB) that begins just before the cut. - */ -const REDACT_WINDOW_MARGIN = 4096 - /** * The one line an operator reads when a relayed stream ends in an upstream error (an in-stream * Codex refusal, a MiniMax in-body failure, a protocol error). `errorClass` is the closed signal * the usage row carries; `upstreamMessage` is the driver hook's reading of the upstream's words, - * which never reaches a usage row. Redacted before it is cut, so truncation cannot split a - * credential below the length its pattern needs and leave a fragment for the logger to miss. + * which never reaches a usage row. */ export function logStreamFailure( log: Logger | undefined, @@ -54,16 +47,67 @@ export function logStreamFailure( log?.warn("upstream stream failure", { accountId, errorClass, - upstreamMessage: bounded( - // Only the part that can be kept is scrubbed, with room for a credential straddling the cut. - message === undefined - ? undefined - : redactValue(message.slice(0, reasonMaxChars + REDACT_WINDOW_MARGIN)), - reasonMaxChars, - ), + upstreamMessage: scrubbedQuote(message, reasonMaxChars), }) } +/** + * An upstream's own words for a log line: redacted before it is cut, so truncation cannot split a + * credential below the length its pattern needs and leave a fragment for the logger to miss. Up to + * `MAX_QUOTE_SCAN_CHARS` the whole text is redacted, so every credential is matched with its own + * terminator and the words before a collapsed token survive. Past it, the scan ends at the last + * whitespace before that mark: no credential's secret part spans whitespace (a Bearer/Basic gap + * comes before its token, so a cut there drops the token whole), so none can straddle the scan's + * end. A message with no whitespace at all in its first `MAX_QUOTE_SCAN_CHARS` is one token nothing + * can safely be cut from, so only `…` is logged. + */ +function scrubbedQuote(text: string | undefined, maxChars: number): string | undefined { + if (text === undefined) return undefined + if (text.length <= MAX_QUOTE_SCAN_CHARS) { + const scrubbed = redactValue(text) + return scrubbed.length <= maxChars ? scrubbed : fitted(scrubbed, maxChars - 1, maxChars) + } + const end = lastWhitespaceBefore(text, MAX_QUOTE_SCAN_CHARS) + return end > 0 ? fitted(redactValue(text.slice(0, end)), maxChars - 1, maxChars) : "…" +} + +/** The index of the last whitespace character at or before `at`, or -1. A loop, not a regex. */ +function lastWhitespaceBefore(text: string, at: number): number { + for (let index = Math.min(at, text.length - 1); index > 0; index--) { + if (WHITESPACE.test(text.charAt(index))) return index + } + return -1 +} + +const WHITESPACE = /\s/ + +/** + * The first `keep` characters plus `…`, held within `maxChars` after the logger's own redaction + * pass. + * A cut never splits a `[REDACTED]` marker, and a cut that leaves a pattern to re-expand (a + * `?token=` whose value the cut shortened) is moved back until the re-redacted text fits. + */ +function fitted(scrubbed: string, keep: number, maxChars: number): string { + let at = keep + for (let round = 0; round < FIT_ROUNDS && at > 0; round++) { + const marker = scrubbed.lastIndexOf(REDACTED, at - 1) + if (marker >= 0 && marker + REDACTED.length > at) at = marker + const quoted = redactValue(`${scrubbed.slice(0, Math.max(0, at))}…`) + if (quoted.length <= maxChars) return quoted + at -= quoted.length - maxChars + } + return "…" +} + +/** + * How much of an upstream message is redacted for its log line. Sized like the default upstream + * error-body and observation caps (64 KiB), which bound most messages; an Agent-SDK message is not + * bounded by either, which is what the whitespace cut above is for. + */ +const MAX_QUOTE_SCAN_CHARS = 65_536 +/** Attempts to pull a cut back until the re-redacted quote fits the bound. */ +const FIT_ROUNDS = 4 + /** An upstream's own words, cut to the configured length for a log line. */ export function bounded(text: string | undefined, maxChars: number): string | undefined { if (text === undefined || text.length <= maxChars) return text diff --git a/apps/api/src/services/usage/response-salvage-slots.ts b/apps/api/src/services/usage/response-salvage-slots.ts new file mode 100644 index 00000000..ad673d78 --- /dev/null +++ b/apps/api/src/services/usage/response-salvage-slots.ts @@ -0,0 +1,88 @@ +import { responseObject } from "./response-usage" + +/** + * The salvage scanner's slot tables, the byte constants its lexer reads, and its final rebuild step, + * kept apart from the byte-level state machine in `response-salvage.ts`. + */ + +// Shallow members kept verbatim, by parent key (root = ""). Everything else is skipped unread. +const SLOTS: Readonly> = { + "": ["type", "usage", "status", "incomplete_details", "error", "code"], + // MiniMax carries its failures on a 200 in `base_resp`; only the number classifies, so the + // free-text `status_msg` beside it is never kept. Kept unless it is over its ceiling, the budget is + // full (`SLOT_MAX_BYTES`), it is malformed, or the frame trips the structural check; losing it reads + // as before the slot existed. + base_resp: ["status_code"], + message: ["usage"], + response: ["usage", "status", "incomplete_details", "error"], +} +/** + * Error evidence a driver hook classifies from (`code`, the nested `error`; a root `message` is + * Anthropic's message object, so it is not a slot). Kept when it is small, + * and dropped — never failing the salvage — when it is not: a lost code reads as an unclassified + * failure, while a failed salvage would lose the failure itself. + */ +export const OPTIONAL: ReadonlySet = new Set(["error", "code", "response.error"]) +/** + * Slots with their own ceiling: dropped past it, or when the budget is full (without evicting + * anything else), released before a required member would fail the salvage, and lent to error + * evidence that needs the room (given back if that evidence drops anyway). Losing one reads as + * before the slot existed; it never fails the salvage. A MiniMax status + * code is a small integer; anything longer cannot classify anyway. + */ +export const SLOT_MAX_BYTES: ReadonlyMap = new Map([["base_resp.status_code", 32]]) +/** Each ceiling slot by the root key that parents it, so a repeated parent releases its children. */ +export const CEILING_CHILDREN: ReadonlyMap = Map.groupBy( + SLOT_MAX_BYTES.keys(), + (path) => path.split(".")[0] ?? "", +) +export const KEY_MAX_BYTES = 32 +export const QUOTE = 34 +export const BACKSLASH = 92 +export const OPEN_OBJECT = 123 +export const CLOSE_OBJECT = 125 +export const OPEN_ARRAY = 91 +export const CLOSE_ARRAY = 93 +export const COLON = 58 +export const COMMA = 44 + +/** A parent key's slot list; own keys only, so a key named `constructor` is no slot at all. */ +export function slotKeys(parent: string): readonly string[] { + return Object.hasOwn(SLOTS, parent) ? (SLOTS[parent] ?? []) : [] +} + +/** + * The rebuilt shallow payload from the captured slots, or null when a required or optional member + * does not parse. A malformed ceiling slot is lost as if it never existed. + */ +export function rebuildSalvage( + captured: ReadonlyMap, + errorObject: boolean, + decoder: TextDecoder, +): Record | null { + const parts: Record = {} + try { + for (const [path, bytes] of captured) { + let value: unknown + try { + value = JSON.parse(decoder.decode(Uint8Array.from(bytes))) + } catch (error) { + // A malformed ceiling slot is lost as if it never existed; anything else fails as before. + if (SLOT_MAX_BYTES.has(path)) continue + throw error + } + const [first, second] = path.split(".") + if (first === undefined) continue + if (second === undefined) parts[first] = value + else { + const parent = responseObject(parts[first]) ?? {} + parent[second] = value + parts[first] = parent + } + } + } catch { + return null + } + if (errorObject && responseObject(parts.error) === undefined) parts.error = {} + return parts +} diff --git a/apps/api/src/services/usage/response-salvage.ts b/apps/api/src/services/usage/response-salvage.ts index f41c1c35..d6e689c2 100644 --- a/apps/api/src/services/usage/response-salvage.ts +++ b/apps/api/src/services/usage/response-salvage.ts @@ -1,37 +1,30 @@ -import { responseObject } from "./response-usage" +import { + BACKSLASH, + CEILING_CHILDREN, + CLOSE_ARRAY, + CLOSE_OBJECT, + COLON, + COMMA, + KEY_MAX_BYTES, + OPEN_ARRAY, + OPEN_OBJECT, + OPTIONAL, + QUOTE, + rebuildSalvage, + SLOT_MAX_BYTES, + slotKeys, +} from "./response-salvage-slots" /** * Bounded salvage of an oversized JSON payload. Some upstreams echo the whole request into a * terminal frame (Codex `response.completed` carries instructions, tools and output), pushing it * past the observation cap while the usage that settles the attempt sits inside. This scanner * walks the oversized bytes once, retains only the few shallow members usage and terminal - * evidence are read from, and rebuilds a small payload from them. It never retains more than - * `maximumBytes` in total; anything it cannot account for honestly yields `null`. + * evidence are read from, and rebuilds a small payload from them. It never counts more than + * `maximumBytes`; one lent ceiling slot (32 bytes at most) is held aside beyond it. Anything it + * cannot account for honestly yields `null`. */ -// Shallow members kept verbatim, by parent key (root = ""). Everything else is skipped unread. -const SLOTS: Readonly> = { - "": ["type", "usage", "status", "incomplete_details", "error", "code"], - message: ["usage"], - response: ["usage", "status", "incomplete_details", "error"], -} -/** - * Error evidence a driver hook classifies from (`code`, the nested `error`; a root `message` is - * Anthropic's message object, so it is not a slot). Kept when it is small, - * and dropped — never failing the salvage — when it is not: a lost code reads as an unclassified - * failure, while a failed salvage would lose the failure itself. - */ -const OPTIONAL = new Set(["error", "code", "response.error"]) -const KEY_MAX_BYTES = 32 -const QUOTE = 34 -const BACKSLASH = 92 -const OPEN_OBJECT = 123 -const CLOSE_OBJECT = 125 -const OPEN_ARRAY = 91 -const CLOSE_ARRAY = 93 -const COLON = 58 -const COMMA = 44 - interface Capture { readonly path: string readonly depth: number @@ -65,36 +58,86 @@ export function createResponseSalvage(maximumBytes: number): ResponseSalvage { let capture: Capture | null = null let retained = 0 let errorObject = false + // Depths 1-2: a second value under a key, or a non-string value where a key or colon belongs, is + // invalid JSON, so the frame keeps no ceiling slot, even one kept earlier. Not a validator: other + // malformed frames can still yield a code (#209). `primitiveRun` marks continuation bytes. + const valueStarted: boolean[] = [false, false, false] + let primitiveRun = false + let ceilingBlocked = false const slotAt = (): string | null => { if (depth === 1 && objects[1] && !expectKey[1]) { const key = keys[1] - return key != null && SLOTS[""]?.includes(key) ? key : null + return key != null && slotKeys("").includes(key) ? key : null } if (depth === 2 && objects[2] && !expectKey[2]) { const parent = keys[1] const key = keys[2] if (parent == null || key == null) return null - return SLOTS[parent]?.includes(key) ? `${parent}.${key}` : null + return slotKeys(parent).includes(key) ? `${parent}.${key}` : null } return null } const optionalMaxBytes = Math.floor(maximumBytes / 4) - // Release every completed optional capture: room for a required member, never a failed salvage. - const evictOptional = () => { - for (const path of OPTIONAL) { + // Release completed captures for room: ceiling slots first, optional evidence only if still full. + const release = (paths: Iterable) => { + for (const path of paths) { + const bytes = captured.get(path) + if (bytes === undefined) continue + retained -= bytes.length + captured.delete(path) + } + } + // Ceiling slots lent to the error capture in progress: given back if it drops (it frees at least + // what it borrowed, so they fit), forgotten once it is kept. + let lent: [string, number[]][] = [] + const lend = () => { + for (const path of SLOT_MAX_BYTES.keys()) { const bytes = captured.get(path) if (bytes === undefined) continue + lent.push([path, bytes]) retained -= bytes.length captured.delete(path) } } + const giveBack = () => { + for (const [path, bytes] of lent) { + captured.set(path, bytes) + retained += bytes.length + } + lent = [] + } + const blockCeiling = () => { + ceilingBlocked = true + release(SLOT_MAX_BYTES.keys()) + lent = [] + if (capture === null || !SLOT_MAX_BYTES.has(capture.path) || capture.dropped) return + retained -= capture.bytes.length + capture.bytes = [] + capture.dropped = true + } + const evictOptional = () => { + release(SLOT_MAX_BYTES.keys()) + if (retained >= maximumBytes) release(OPTIONAL) + } const keep = (byte: number) => { if (capture === null || capture.dropped) return + const ceiling = SLOT_MAX_BYTES.get(capture.path) + // Error evidence outranks a ceiling slot: borrow its room before giving way. + if (OPTIONAL.has(capture.path) && retained >= maximumBytes && lent.length === 0) lend() if ( - OPTIONAL.has(capture.path) && - (capture.bytes.length >= optionalMaxBytes || retained >= maximumBytes) + (ceiling !== undefined && capture.bytes.length >= ceiling) || + (OPTIONAL.has(capture.path) && + (capture.bytes.length >= optionalMaxBytes || retained >= maximumBytes)) ) { + retained -= capture.bytes.length + capture.bytes = [] + capture.dropped = true + giveBack() + return + } + // A ceiling slot gives way itself at a full budget; it never evicts the evidence others hold. + if (retained >= maximumBytes && ceiling !== undefined) { retained -= capture.bytes.length capture.bytes = [] capture.dropped = true @@ -115,14 +158,28 @@ export function createResponseSalvage(maximumBytes: number): ResponseSalvage { retained -= captured.get(capture.path)?.length ?? 0 if (!capture.dropped) captured.set(capture.path, capture.bytes) else captured.delete(capture.path) + if (!capture.dropped) lent = [] capture = null } - const beginValue = (byte: number) => { + const beginValue = (byte: number, fresh: boolean) => { if (depth === 1 && objects[1] && !expectKey[1] && keys[1] === "error" && byte === OPEN_OBJECT) errorObject = true + if (fresh && depth <= 2 && objects[depth]) { + if (expectKey[depth] || valueStarted[depth]) blockCeiling() + valueStarted[depth] = true + } if (capture !== null) return + // Last wins, as in JSON.parse: a new parent value, or a repeated slot in one parent, replaces + // what the slot held. Keys are compared as raw bytes, escapes not decoded, like every slot (#209). + if (depth === 1 && objects[1] && !expectKey[1] && keys[1] != null) { + release(CEILING_CHILDREN.get(keys[1]) ?? []) + } const path = slotAt() if (path === null) return + if (SLOT_MAX_BYTES.has(path)) { + release([path]) + if (ceilingBlocked) return + } capture = { path, depth, @@ -157,6 +214,7 @@ export function createResponseSalvage(maximumBytes: number): ResponseSalvage { return } if (whitespace(byte)) { + primitiveRun = false if (capture?.kind === "primitive" && capture.depth === depth) endCapture() else keep(byte) return @@ -175,27 +233,31 @@ export function createResponseSalvage(maximumBytes: number): ResponseSalvage { } started = true } + const continuesPrimitive = primitiveRun + primitiveRun = false switch (byte) { case QUOTE: inString = true if (depth >= 1 && depth <= 2 && objects[depth] && expectKey[depth]) { keyBytes = [] keys[depth] = null + valueStarted[depth] = false keep(byte) return } - beginValue(byte) + beginValue(byte, true) keep(byte) return case OPEN_OBJECT: case OPEN_ARRAY: - if (depth > 0) beginValue(byte) + if (depth > 0) beginValue(byte, true) keep(byte) depth++ if (depth <= 2) { objects[depth] = byte === OPEN_OBJECT expectKey[depth] = byte === OPEN_OBJECT keys[depth] = null + valueStarted[depth] = false } return case CLOSE_OBJECT: @@ -222,30 +284,14 @@ export function createResponseSalvage(maximumBytes: number): ResponseSalvage { failed = true return } - if (capture === null) beginValue(byte) + primitiveRun = true + if (capture === null || !continuesPrimitive) beginValue(byte, !continuesPrimitive) keep(byte) } }, finish() { if (failed || !closed || inString) return null - const parts: Record = {} - try { - for (const [path, bytes] of captured) { - const value: unknown = JSON.parse(decoder.decode(Uint8Array.from(bytes))) - const [first, second] = path.split(".") - if (first === undefined) continue - if (second === undefined) parts[first] = value - else { - const parent = responseObject(parts[first]) ?? {} - parent[second] = value - parts[first] = parent - } - } - } catch { - return null - } - if (errorObject && responseObject(parts.error) === undefined) parts.error = {} - return parts + return rebuildSalvage(captured, errorObject, decoder) }, get retainedBytes() { return retained diff --git a/apps/api/test/unit/dataplane/attempt-log.test.ts b/apps/api/test/unit/dataplane/attempt-log.test.ts new file mode 100644 index 00000000..00d79178 --- /dev/null +++ b/apps/api/test/unit/dataplane/attempt-log.test.ts @@ -0,0 +1,265 @@ +import { describe, expect, test } from "bun:test" +import { createLogger } from "../../../src/logging/logger" +import type { AttemptOutcome } from "../../../src/services/dataplane/attempt" +import { logAttemptFailure, logStreamFailure } from "../../../src/services/dataplane/attempt-log" + +const REASON_MAX_CHARS = 40 +const FAKE_KEY = "sk-LIVEFAKEKEY-do-not-leak-1234567890abcdef" + +function captured() { + const lines: Record[] = [] + const log = createLogger({ + level: "info", + write: (line) => { + lines.push(JSON.parse(line)) + }, + }) + return { log, lines } +} + +/** A classified failure whose message no earlier stage scrubbed, as the Agent-SDK path delivers. */ +function failure(message: string | undefined): Extract { + return { + kind: "failure", + failure: { kind: "auth", status: 401, message: "the upstream rejected the credential" }, + classification: { + kind: "auth", + status: 401, + retryable: true, + signal: "claude-sdk:auth", + rateLimit: null, + message, + }, + rateLimit: null, + } as Extract +} + +describe("logAttemptFailure (#206)", () => { + test("a credential cut by the length bound is redacted first, so no fragment survives", () => { + const { log, lines } = captured() + // The cut keeps REASON_MAX_CHARS - 1 characters; this prefix puts the cut seven characters into + // the key, shorter than the `sk-` pattern needs to match on its own. + logAttemptFailure(log, "a", 1, failure(`${"r".repeat(31)} ${FAKE_KEY}`), REASON_MAX_CHARS) + const text = JSON.stringify(lines) + expect(text).not.toContain("sk-LIVE") + const quoted = lines[0]?.upstreamMessage + if (typeof quoted !== "string") throw new Error("the line carries no upstreamMessage") + expect(quoted.length).toBeLessThanOrEqual(REASON_MAX_CHARS) + }) + + test("a credential shrunk away early cannot pull a cut fragment into the kept text (#207 review)", () => { + const { log, lines } = captured() + const maxChars = 200 + // The window is maxChars + 4096. A 4,100-character bearer value collapses to `[REDACTED]`, and + // the window ends four characters before the `@` of a userinfo URL whose password is then cut + // below anything a pattern can match. + const head = `Bearer ${"b".repeat(4100)} ` + const url = "postgres://router:SuperSecretPassw0rdXYZ@db" + const filler = "f".repeat(maxChars + 4096 - head.length - (url.indexOf("@") - 4)) + logAttemptFailure(log, "a", 1, failure(`${head}${filler}${url}`), maxChars) + expect(JSON.stringify(lines)).not.toContain("SuperSecret") + const quoted = lines[0]?.upstreamMessage + if (typeof quoted !== "string") throw new Error("the line carries no upstreamMessage") + expect(quoted.length).toBeLessThanOrEqual(maxChars) + }) + + test("the words before a collapsed token are still logged (#207 review)", () => { + const { log, lines } = captured() + const message = `Failed to authenticate: upstream said Bearer ${"t".repeat(400)} ${"x".repeat(5000)}` + logAttemptFailure(log, "a", 1, failure(message), 200) + const quoted = lines[0]?.upstreamMessage + if (typeof quoted !== "string") throw new Error("the line carries no upstreamMessage") + expect(quoted.startsWith("Failed to authenticate: upstream said [REDACTED]")).toBe(true) + expect(quoted.length).toBe(200) + }) + + test("a token far longer than the window still leaves the words before it (#207 round 3)", () => { + for (const message of [ + `Failed: Bearer ${"t".repeat(30_000)} ${"x".repeat(5_000)}`, + Array.from( + { length: 400 }, + (_, i) => `sk-${String(i).padStart(12, "k")}abcdefghijklmnop`, + ).join(","), + ]) { + const { log, lines } = captured() + logAttemptFailure(log, "a", 1, failure(message), 200) + const quoted = lines[0]?.upstreamMessage + if (typeof quoted !== "string") throw new Error("the line carries no upstreamMessage") + expect(quoted.length).toBeGreaterThan(10) + expect(quoted.length).toBeLessThanOrEqual(200) + expect(quoted).not.toContain("abcdefghijklmnop") + } + }) + + test("a cut never splits a [REDACTED] marker, so the logged line stays within the bound", () => { + for (let pad = 0; pad < 20; pad++) { + const { log, lines } = captured() + // The logger's own pass re-expands a cut `?token=[RE…` to `?token=[REDACTED]`. + const message = `${"p".repeat(pad + 10)} https://h/cb?token=abcdefgh ${"x".repeat(6_000)}` + logAttemptFailure(log, "a", 1, failure(message), 40) + const quoted = lines[0]?.upstreamMessage + if (typeof quoted !== "string") throw new Error("the line carries no upstreamMessage") + expect(quoted.length).toBeLessThanOrEqual(40) + // The pull-back keeps what fits: the p-run and, while there is room, the endpoint. + const prefix = `${"p".repeat(pad + 10)} https://h/` + expect(quoted).toStartWith(prefix.length < 40 ? prefix : "p".repeat(pad + 10)) + } + }) + + test("a bound that ends inside a marker keeps the words before it and the whole marker", () => { + const { log, lines } = captured() + logAttemptFailure(log, "a", 1, failure(`abc ${FAKE_KEY} ${"x".repeat(300)}`), 15) + expect(lines[0]?.upstreamMessage).toBe("abc [REDACTED]…") + }) + + test("a credential over 4 KB across the window's end leaves no head (#207 round 4)", () => { + const message = `sk-${"k".repeat(4300)} x://svc:${"P".repeat(5000)}@db.internal/app` + const { log, lines } = captured() + logAttemptFailure(log, "a", 1, failure(message), 200) + expect(JSON.stringify(lines)).not.toContain("PPPP") + }) + + test.each([ + ["a JWT payload", `eyJ${"Q".repeat(20)}.${"Q".repeat(13_000)}.${"Q".repeat(40)}`], + ["a userinfo password", `postgres://u:${"Q".repeat(13_000)}@h`], + ["a long JWT header", `eyJ${"Q".repeat(13_000)}.${"Q".repeat(40)}.${"Q".repeat(40)}`], + ])( + "%s over 4 KB behind a collapsed token leaves nothing (#207 round 4)", + (_label, credential) => { + const { log, lines } = captured() + logAttemptFailure(log, "a", 1, failure(`Bearer ${"t".repeat(5_000)} ${credential} tail`), 200) + expect(JSON.stringify(lines)).not.toContain("QQQQ") + }, + ) + + test("past 64 KiB, a credential straddling the scan's end leaves nothing (#207 round 5)", () => { + for (const maxChars of [200, 2000]) { + const { log, lines } = captured() + const message = `Failed: Bearer ${"b".repeat(60_000)} postgres://u:${"P".repeat(8_000)}@h tail` + logAttemptFailure(log, "a", 1, failure(message), maxChars) + expect(JSON.stringify(lines)).not.toContain("PPPP") + expect(String(lines[0]?.upstreamMessage)).toStartWith("Failed: [REDACTED]") + } + }) + + test.each([65_536, 65_537])("a key ending a %i-character message is redacted", (length) => { + const key = " sk-LIVEFAKEKEY-do-not-leak-1234567890abcdef" + const { log, lines } = captured() + logAttemptFailure(log, "a", 1, failure(`${"f".repeat(length - key.length)}${key}`), 70_000) + expect(JSON.stringify(lines)).not.toContain("LIVEFAKEKEY") + }) + + test("a whitespace-free message over 64 KiB logs only an ellipsis: no straddling head leaks", () => { + const message = `sk-${"k".repeat(60_000)},postgres://u:${"P".repeat(9_000)}@db.internal/app` + const { log, lines } = captured() + logAttemptFailure(log, "a", 1, failure(message), 200) + expect(JSON.stringify(lines)).not.toContain("PPPP") + expect(lines[0]?.upstreamMessage).toBe("…") + }) + + test("a long message with nothing to redact still keeps the full bound", () => { + const { log, lines } = captured() + logAttemptFailure(log, "a", 1, failure("m".repeat(10_000)), REASON_MAX_CHARS) + expect(lines[0]?.upstreamMessage).toBe(`${"m".repeat(REASON_MAX_CHARS - 1)}…`) + }) + + test("a short message passes through whole, and an absent one stays absent", () => { + const { log, lines } = captured() + logAttemptFailure(log, "a", 1, failure("was rejected"), REASON_MAX_CHARS) + expect(lines[0]?.upstreamMessage).toBe("was rejected") + logAttemptFailure(log, "a", 1, failure(undefined), REASON_MAX_CHARS) + expect(lines[1]).toBeDefined() + expect(lines[1]).not.toHaveProperty("upstreamMessage") + }) +}) + +describe("the 64 KiB scan boundary, exactly (#207 round 6)", () => { + const quoted = (message: string) => { + const { log, lines } = captured() + logAttemptFailure(log, "a", 1, failure(message), 200) + return String(lines[0]?.upstreamMessage) + } + const dsn = `postgres://u:${"P".repeat(9_000)}@h` + + test("65,536 characters with no whitespace are redacted whole and keep their head", () => { + expect(quoted("z".repeat(65_536))).toStartWith("zzzz") + }) + test("a newline counts as whitespace for the cut", () => { + expect(quoted(`Failed\n${"f".repeat(70_000)}\nmore`)).toBe("Failed…") + }) + test("65,537 characters with no whitespace log only an ellipsis", () => { + expect(quoted("z".repeat(65_537))).toBe("…") + }) + test("whitespace exactly at 65,536 ends the scan there and keeps the long head", () => { + const message = `Failed ${"f".repeat(65_536 - 7)} ${dsn}` + expect(message.charAt(65_536)).toBe(" ") + const line = quoted(message) + expect(line).toStartWith("Failed ffff") + expect(line).not.toContain("PPPP") + }) + test("whitespace only at 65,537 falls back to the last earlier whitespace", () => { + const message = `Failed ${"f".repeat(65_537 - 7)} ${dsn}` + expect(message.charAt(65_537)).toBe(" ") + expect(quoted(message)).toBe("Failed…") + }) + test("a cut inside a standalone marker pulls back to before it", () => { + const message = `abc ${FAKE_KEY} tail ${"x".repeat(300)}` + for (let max = 6; max <= 16; max++) { + const { log, lines } = captured() + logAttemptFailure(log, "a", 1, failure(message), max) + const line = String(lines[0]?.upstreamMessage) + expect(line).not.toMatch(/\[[A-Z]*…/) + expect(line.length).toBeLessThanOrEqual(max) + } + }) +}) + +describe("both log lines scrub the whole message before cutting (#207 round 6)", () => { + type Line = (message: string, maxChars: number) => Record[] + const attempt: Line = (message, maxChars) => { + const { log, lines } = captured() + logAttemptFailure(log, "a", 1, failure(message), maxChars) + return lines + } + const stream: Line = (message, maxChars) => { + const { log, lines } = captured() + logStreamFailure(log, "a", "codex:stream-error:other", message, maxChars) + return lines + } + + test.each([ + ["upstream attempt failed", attempt], + ["upstream stream failure", stream], + ] as const)("%s: a key cut by the length bound is redacted first", (_name, line) => { + const lines = line(`${"r".repeat(31)} ${FAKE_KEY}`, REASON_MAX_CHARS) + expect(JSON.stringify(lines)).not.toContain("sk-LIVE") + }) + + test.each([ + ["upstream attempt failed", attempt], + ["upstream stream failure", stream], + ] as const)("%s: no credential head leaks", (_name, line) => { + const messages = [ + // 2.25.8's stream line logged 175 password characters for this one. + `Bearer ${"t".repeat(4_000)} postgres://u:${"P".repeat(500)}@h tail`, + `Bearer ${"t".repeat(5_000)} eyJ${"P".repeat(20)}.${"P".repeat(13_000)}.${"P".repeat(40)} tail`, + `Bearer ${"t".repeat(5_000)} postgres://u:${"P".repeat(13_000)}@h tail`, + `Failed: Bearer ${"b".repeat(60_000)} postgres://u:${"P".repeat(8_000)}@h tail`, + // The last '@' or '.' before the 64 KiB mark sits inside a credential that ends after it, so + // only a whitespace-only cut set keeps these clean. + `Failed: Bearer ${"b".repeat(60_000)} postgres://u:${"P".repeat(100)}@h${"z".repeat(10_000)}`, + `Failed: Bearer ${"b".repeat(60_000)} eyJ${"h".repeat(20)}.${"P".repeat(200)}.${"s".repeat(10_000)}`, + `Failed: Bearer ${"b".repeat(60_000)} sk-PPPP_${"z".repeat(10_000)}`, + `Failed: Bearer ${"b".repeat(60_000)} postgres://u:${"P".repeat(100)}=${"z".repeat(10_000)}@h`, + `Failed: Bearer ${"b".repeat(60_000)} postgres://u:${"P".repeat(100)}&${"z".repeat(10_000)}@h`, + `Failed: Bearer ${"b".repeat(60_000)} postgres://u:${"P".repeat(100)}?${"z".repeat(10_000)}@h`, + ] + for (const message of messages) { + for (const maxChars of [200, 2000]) { + const lines = line(message, maxChars) + expect(JSON.stringify(lines)).not.toContain("PPPP") + expect(String(lines[0]?.upstreamMessage).length).toBeLessThanOrEqual(maxChars) + } + } + }) +}) diff --git a/apps/api/test/unit/providers/openai-oauth-stream-errors.test.ts b/apps/api/test/unit/providers/openai-oauth-stream-errors.test.ts index 36202c26..afbe8d0b 100644 --- a/apps/api/test/unit/providers/openai-oauth-stream-errors.test.ts +++ b/apps/api/test/unit/providers/openai-oauth-stream-errors.test.ts @@ -102,6 +102,108 @@ describe("inspectCodexStreamFrame", () => { }) }) + test.each([ + "insufficient_quota", + "credit_balance_exhausted", + "organization_spend_limit_exceeded", + "project_spend_limit_exceeded", + ])( + "in-stream %s classifies credits-exhausted, as the Codex CLI's QuotaExceeded (#206)", + (code) => { + // The shape the Codex CLI's own test feeds its stream parser (openai/codex + // codex-rs/codex-api/src/sse/responses.rs, `quota_exceeded_error_is_fatal`). + const verdict = inspectCodexStreamFrame( + { + type: "response.failed", + response: { + status: "failed", + error: { code, message: "You exceeded your current quota" }, + }, + }, + "response.failed", + ) + expect(verdict?.kind).toBe("credits-exhausted") + expect(verdict?.signal).toBe(`codex:stream-error:${code}`) + }, + ) + + test("a quota code on a bare error event is not a spent balance: the CLI ignores that event", () => { + // openai/codex reads a quota code only from `response.failed` at `response.error.code`; a bare + // `error` event other than flex_unavailable is ignored there, so it must not park the account. + for (const frame of [ + { type: "error", code: "insufficient_quota" }, + { type: "error", error: { code: "credit_balance_exhausted" } }, + ]) { + const verdict = inspectCodexStreamFrame(frame, "error") + expect(verdict?.kind).toBe("server-error") + expect(verdict?.signal).toBe("codex:stream-error:other") + } + }) + + test.each([ + [ + "a top-level code on response.failed", + { type: "response.failed", code: "insufficient_quota" }, + "response.failed", + ], + [ + "a root error on response.failed", + { type: "response.failed", error: { code: "insufficient_quota" } }, + "response.failed", + ], + [ + "no JSON type, only the SSE event name", + { response: { error: { code: "insufficient_quota" } } }, + "response.failed", + ], + ])( + "a quota code the CLI would not read (%s) does not park the account (#207 round 4)", + (_label, frame, event) => { + // openai/codex reads a quota code only from `response.error.code` of a frame whose own JSON + // `type` is `response.failed`; every other shape is retried or skipped there. + const verdict = inspectCodexStreamFrame(frame, event) + expect(verdict?.kind).toBe("server-error") + expect(verdict?.signal).toBe("codex:stream-error:other") + }, + ) + + test("a root field does not hide a quota code at response.error.code (#207 round 6)", () => { + const verdict = inspectCodexStreamFrame( + { + type: "response.failed", + error: { message: "" }, + response: { error: { code: "organization_spend_limit_exceeded" } }, + }, + "response.failed", + ) + expect(verdict?.kind).toBe("credits-exhausted") + expect(verdict?.signal).toBe("codex:stream-error:organization_spend_limit_exceeded") + }) + + test("a quota code at response.error.code beats a listed root code, as the CLI reads it", () => { + const verdict = inspectCodexStreamFrame( + { + type: "response.failed", + code: "rate_limit_exceeded", + response: { error: { code: "insufficient_quota" } }, + }, + "response.failed", + ) + expect(verdict?.kind).toBe("credits-exhausted") + expect(verdict?.signal).toBe("codex:stream-error:insufficient_quota") + }) + + test("a quota type with no code is not a spent balance in-stream: the CLI reads only the code", () => { + // openai/codex responses_error.rs reads `code` alone; a type-only frame is retried there, so it + // must not park the account here. + const verdict = inspectCodexStreamFrame( + { type: "error", error: { type: "insufficient_quota" } }, + "error", + ) + expect(verdict?.kind).toBe("server-error") + expect(verdict?.signal).toBe("codex:stream-error:other") + }) + test("an unknown code is fail-closed server-error with the closed `other` signal", () => { const verdict = inspectCodexStreamFrame( { type: "error", code: "some_made_up_code", message: "fixture failure" }, diff --git a/apps/api/test/unit/providers/openai-oauth.test.ts b/apps/api/test/unit/providers/openai-oauth.test.ts index 52e3b353..72d05f3d 100644 --- a/apps/api/test/unit/providers/openai-oauth.test.ts +++ b/apps/api/test/unit/providers/openai-oauth.test.ts @@ -22,6 +22,7 @@ import { parseCodexQuotaHeaders, readOpenAiOAuthTokens, } from "../../../src/providers" +import { CODEX_QUOTA_CODES } from "../../../src/providers/drivers/openai-oauth-stream-errors" import { account, response } from "./fixtures" function jwt(payload: Record): string { @@ -239,6 +240,40 @@ describe("Codex failure wording", () => { expect(result?.retryable).toBe(true) expect(result?.signal).toBe("openai-oauth:account_deactivated") }) + test("a 429 carrying a quota code is a spent balance, not a cooldown (#206)", () => { + for (const body of [ + // The `error` envelope is the one the Codex CLI reads; `detail` is the one this driver reads + // every Codex code from, so both classify. + ...CODEX_QUOTA_CODES.map((code) => ({ error: { code, message: "quota" } })), + ...CODEX_QUOTA_CODES.map((code) => ({ detail: { code, message: "quota" } })), + { error: { type: "insufficient_quota", message: "quota" } }, + ]) { + const result = codex.classifyFailure(response(429, { body })) + expect(result?.kind).toBe("credits-exhausted") + expect(result?.signal).toBe("openai-oauth:insufficient_quota") + } + }) + test.each([ + [400, { detail: { code: "insufficient_quota" } }], + [400, { error: { type: "insufficient_quota" } }], + [403, { error: { type: "insufficient_quota" } }], + [500, { error: { type: "insufficient_quota" } }], + [400, { detail: { type: "insufficient_quota" } }], + ])("a quota signal on status %i does not park the account (#207 round 11)", (status, body) => { + expect(codex.classifyFailure(response(status, { body }))?.kind).not.toBe("credits-exhausted") + }) + test.each([ + { error: { type: "credit_balance_exhausted" } }, + { error: { code: "usage_limit_reached", type: "insufficient_quota" } }, + ])("a 429 the driver reads as a limit, not quota (#207 round 11): %o", (body) => { + expect(codex.classifyFailure(response(429, { body }))?.kind).toBe("rate-limited") + }) + test("a quota code on another status does not park the account (#207 review)", () => { + const result = codex.classifyFailure( + response(400, { body: { detail: { code: "insufficient_quota", message: "quota" } } }), + ) + expect(result?.kind).not.toBe("credits-exhausted") + }) test("a bare string detail still yields a message, and falls back to the status", () => { const result = codex.classifyFailure(response(401, { body: { detail: "invalid token" } })) expect(result?.kind).toBe("auth") diff --git a/apps/api/test/unit/redact.test.ts b/apps/api/test/unit/redact.test.ts index 9ef7a2d7..4ce7883f 100644 --- a/apps/api/test/unit/redact.test.ts +++ b/apps/api/test/unit/redact.test.ts @@ -467,6 +467,10 @@ describe("redactValue cost", () => { "eyJaaaaa-".repeat(8 * 1024), "eyJ-".repeat(16 * 1024), "a://b:".repeat(10 * 1024), + "://u:p/".repeat(10 * 1024), + `://u:${"p/".repeat(32 * 1024)}`, + "://".repeat(64 * 1024), + "://:".repeat(48 * 1024), ]) { const started = performance.now() redactValue(message) @@ -492,6 +496,37 @@ describe("redactValue cost", () => { expect(redactValue("redis://:onlypassword@cache:6379")).toBe(`redis://${REDACTED}@cache:6379`) }) + test("a userinfo password with an unencoded `/` is scrubbed whole (#206)", () => { + expect(redactValue("connect failed: postgres://u:p/ss@h:5432/app")).toBe( + `connect failed: postgres://${REDACTED}@h:5432/app`, + ) + expect(redactValue("redis://:a/b/c@cache")).toBe(`redis://${REDACTED}@cache`) + }) + + test("the slash branch stops at a quote, and a port-plus-`@` URL pays its documented price", () => { + const json = '{"url":"https://h:8080/a","owner":"x@y"}' + expect(redactValue(json)).toBe(json) + expect(redactValue("https://host:8080/a@b")).toBe(`https://${REDACTED}@b`) + }) + + test("a base64 slash password with `=` padding is scrubbed (#207 review)", () => { + expect(redactValue("connect failed: postgres://router:Zk3/q+Vb9xQ==@db:5432/app")).toBe( + `connect failed: postgres://${REDACTED}@db:5432/app`, + ) + }) + + test("a query-string secret holding an `@` behind a host:port stays redacted (#207 review)", () => { + for (const [input, secret] of [ + ["postgresql://db.internal:5432/app?user=app&password=p@ssw0rd", "ssw0rd"], + ["redis://cache:6379/0?password=hunter@2secret", "2secret"], + ["connect https://h:8443/cb?token=tok@en_SECRET_TAIL failed", "SECRET_TAIL"], + ["postgres://db?sslmode=require&password=Ab:c/d@ef9", "ef9"], + ["GET https://host:8080/v1&api_key=ab@cdSECRET", "cdSECRET"], + ] as const) { + expect(redactValue(input)).not.toContain(secret) + } + }) + test("a userinfo URL with a long but real scheme is still scrubbed", () => { expect(redactValue("x postgresql+asyncpg://router:s3cr3t-pw@db.internal/app")).toBe( `x postgresql+asyncpg://${REDACTED}@db.internal/app`, diff --git a/apps/api/test/unit/services/usage/response-observer-codex.test.ts b/apps/api/test/unit/services/usage/response-observer-codex.test.ts index 0b79dfd6..68a11316 100644 --- a/apps/api/test/unit/services/usage/response-observer-codex.test.ts +++ b/apps/api/test/unit/services/usage/response-observer-codex.test.ts @@ -182,6 +182,38 @@ describe("minimax closed base_resp signal stays bounded under the generic rule", }) }) +describe("a MiniMax in-body failure past the observation cap (#206)", () => { + test.each([ + [ + "before the padding", + (pad: string) => ({ base_resp: { status_code: 1008, status_msg: "x" }, content: pad }), + ], + [ + "after the padding", + (pad: string) => ({ content: pad, base_resp: { status_code: 1008, status_msg: "x" } }), + ], + [ + "with a status_msg too large to keep (#207 review)", + (pad: string) => ({ + content: pad, + base_resp: { status_code: 1008, status_msg: "m".repeat(400) }, + }), + ], + ])("base_resp %s keeps its code through salvage", (_label, body) => { + const observer = createResponseObserver({ + dialect: "anthropic", + operation: "messages", + contentType: "application/json", + maximumObservationBytes: 1024, + descriptor: miniMaxDriver.responseObservation, + }) + observer.observe(encoder.encode(JSON.stringify(body("x".repeat(4096))))) + const facts = observer.finish() + expect(facts.failure?.kind).toBe("credits-exhausted") + expect(facts.failure?.signal).toBe("minimax:base_resp-1008") + }) +}) + describe("errorClass from a driver hook, through relay-terminal", () => { test("a MiniMax in-body failure records its closed base_resp signal as errorClass", () => { const observer = createResponseObserver({ diff --git a/apps/api/test/unit/usage/response-salvage.test.ts b/apps/api/test/unit/usage/response-salvage.test.ts index 404a01f1..56c8f3e9 100644 --- a/apps/api/test/unit/usage/response-salvage.test.ts +++ b/apps/api/test/unit/usage/response-salvage.test.ts @@ -231,3 +231,260 @@ test("a repeated optional key releases the earlier capture's bytes (#188)", () = expect(scanner.retainedBytes).toBe(40) expect(scanner.finish()).toEqual({ code: "d".repeat(10), usage: { t: "u".repeat(20) } }) }) + +test("a root key named like an Object built-in never throws (#207 review)", () => { + const scanner = createResponseSalvage(1024) + const text = + '{"constructor":{"a":1},"__proto__":{"b":2},"toString":{"c":3},"usage":{"input_tokens":5}}' + for (const byte of encoder.encode(text)) scanner.push(byte) + expect(scanner.finish()).toEqual({ usage: { input_tokens: 5 } }) +}) + +test("an oversized base_resp.status_code is dropped, never fails the salvage (#207 review)", () => { + const scanner = createResponseSalvage(1024) + const text = `{"type":"response.completed","usage":{"input_tokens":5},"base_resp":{"status_code":"${"9".repeat(2048)}"}}` + for (const byte of encoder.encode(text)) scanner.push(byte) + expect(scanner.retainedBytes).toBeLessThanOrEqual(1024) + expect(scanner.finish()).toEqual({ type: "response.completed", usage: { input_tokens: 5 } }) +}) + +test("at a tiny cap, base_resp.status_code gives way rather than failing the salvage (#207 round 3)", () => { + for (let cap = 36; cap <= 60; cap++) { + const scanner = createResponseSalvage(cap) + const text = `{"usage":{"input_tokens":5,"output_tokens":3},"pad":"${"x".repeat(200)}","base_resp":{"status_code":1008}}` + for (const byte of encoder.encode(text)) scanner.push(byte) + const usage = '{"input_tokens":5,"output_tokens":3}'.length + if (usage <= cap) expect(scanner.finish()).toMatchObject({ usage: { input_tokens: 5 } }) + } +}) + +test("a full budget drops base_resp.status_code itself rather than evicting earlier error evidence (#207 round 4)", () => { + const scanner = createResponseSalvage(128) + const text = `{"error":{"code":"insufficient_quota"},"type":"${"t".repeat(94)}","base_resp":{"status_code":1008},"z":"${"q".repeat(128)}"}` + for (const byte of encoder.encode(text)) scanner.push(byte) + expect(scanner.finish()).toEqual({ error: { code: "insufficient_quota" }, type: "t".repeat(94) }) +}) + +test("a full 64 KiB budget evicts base_resp.status_code before the code main kept (#207 round 4)", () => { + const cap = 65_536 + const head = '{"code":"insufficient_quota","base_resp":{"status_code":1008},"usage":{"pad":"' + // code (20 bytes) and usage fit the cap together; with the 4-byte status code they do not. + const fill = "u".repeat(cap - 32) + const scanner = createResponseSalvage(cap) + for (const byte of encoder.encode(`${head}${fill}"}}`)) scanner.push(byte) + expect(scanner.finish()).toMatchObject({ code: "insufficient_quota", usage: { pad: fill } }) +}) + +test("an error seen after base_resp at a full budget evicts base_resp, not itself (#207 round 5)", () => { + const scanner = createResponseSalvage(64) + const text = `{"usage":{"p":"${"u".repeat(50)}"},"base_resp":{"status_code":1008},"code":"q","status":"s"}` + for (const byte of encoder.encode(text)) scanner.push(byte) + expect(scanner.finish()).toMatchObject({ code: "q", status: "s" }) +}) + +test("an error capture that drops anyway does not take base_resp with it (#207 round 6)", () => { + const scanner = createResponseSalvage(64) + const text = `{"usage":{"p":"${"u".repeat(50)}"},"base_resp":{"status_code":1008},"code":"insufficient_quota"}` + for (const byte of encoder.encode(text)) scanner.push(byte) + expect(scanner.retainedBytes).toBe(62) + expect(scanner.finish()).toMatchObject({ base_resp: { status_code: 1008 } }) +}) + +test("lent room stays with a kept capture and is not double-counted later (#207 round 7)", () => { + const prefix = `{"usage":{"p":"${"u".repeat(50)}"},"base_resp":{"status_code":1008}` + const kept = createResponseSalvage(64) + for (const byte of encoder.encode(`${prefix},"code":"ab","error":"xyz"}`)) kept.push(byte) + expect(kept.retainedBytes).toBe(62) + expect(kept.finish()).not.toHaveProperty("base_resp") + + const dropped = createResponseSalvage(64) + const text = `${prefix},"code":"insufficient_quota","error":"xyz"}` + for (const byte of encoder.encode(text)) dropped.push(byte) + expect(dropped.retainedBytes).toBe(63) + const rebuilt = dropped.finish() + expect(rebuilt).toMatchObject({ error: "xyz" }) + expect(rebuilt).not.toHaveProperty("base_resp") +}) + +test("a repeated base_resp parent, each copy with a code, keeps the last one (#207 round 7)", () => { + const scanner = createResponseSalvage(64) + const text = `{"base_resp":{"status_code":1008},"usage":{"p":"${"x".repeat(52)}"},"code":"${"c".repeat(40)}","base_resp":{"status_code":1002}}` + for (const byte of encoder.encode(text)) { + scanner.push(byte) + expect(scanner.retainedBytes).toBeLessThanOrEqual(64) + } + expect(scanner.finish()).toMatchObject({ base_resp: { status_code: 1002 } }) +}) + +test.each(['{"status_msg":"x"}', "{}", "null", "5", '"s"', "[1]"])( + "a later base_resp of %s replaces the earlier code, as JSON.parse does (#207 round 8)", + (later) => { + const scanner = createResponseSalvage(64) + const text = `{"base_resp":{"status_code":1008},"base_resp":${later},"pad":"${"x".repeat(200)}"}` + for (const byte of encoder.encode(text)) scanner.push(byte) + const rebuilt = scanner.finish() + expect(rebuilt?.base_resp).toBeUndefined() + }, +) + +test("a parent repeat after a lend, a drop and a give-back still clears the code (#207 round 8)", () => { + const scanner = createResponseSalvage(64) + const text = `{"usage":{"p":"${"u".repeat(50)}"},"base_resp":{"status_code":1008},"code":"${"c".repeat(40)}","base_resp":{}}` + for (const byte of encoder.encode(text)) { + scanner.push(byte) + expect(scanner.retainedBytes).toBeLessThanOrEqual(64) + } + expect(scanner.finish()?.base_resp).toBeUndefined() +}) + +test("a status_code repeated inside one base_resp keeps the last one, as JSON.parse does (#207 round 9)", () => { + const scanner = createResponseSalvage(64) + const text = `{"usage":{"p":"${"u".repeat(50)}"},"base_resp":{"status_code":1008,"status_code":1002},"pad":"${"x".repeat(200)}"}` + for (const byte of encoder.encode(text)) { + scanner.push(byte) + expect(scanner.retainedBytes).toBeLessThanOrEqual(64) + } + expect(scanner.finish()).toMatchObject({ base_resp: { status_code: 1002 } }) +}) + +test.each([ + "01008", + "+1008", + "NaN", + '"a\tb"', + "10 08", + "0 1008", + '"x"1008', + "{}1008", + "null 1008", + "0 :1008", + "0,:1008", + '0 "x"', + "1008,0", +])("a malformed status_code %s is dropped, never failing the salvage (#207 round 9)", (code) => { + const scanner = createResponseSalvage(1024) + const text = `{"usage":{"input_tokens":11,"output_tokens":22},"base_resp":{"status_code":${code}},"pad":"${"x".repeat(2048)}"}` + for (const byte of encoder.encode(text)) scanner.push(byte) + expect(scanner.finish()).toEqual({ usage: { input_tokens: 11, output_tokens: 22 } }) +}) + +test("an invalid UTF-8 status_code is dropped, never failing the salvage (#207 round 9)", () => { + const scanner = createResponseSalvage(1024) + const head = encoder.encode('{"usage":{"input_tokens":11},"base_resp":{"status_code":"') + const tail = encoder.encode(`"},"pad":"${"x".repeat(2048)}"}`) + for (const byte of [...head, 0xff, ...tail]) scanner.push(byte) + expect(scanner.finish()).toEqual({ usage: { input_tokens: 11 } }) +}) + +test.each([ + '"usage":{"input_tokens":01}', + '"type":tru', + '"response":{"status":+1}', + '"type":"a\tb"', + '"status":tru', + '"incomplete_details":{"a":01}', + '"response":{"usage":{"x":01}}', + '"message":{"usage":+1}', + '"error":{"a":01}', + '"response":{"error":{"a":01}}', + '"response":{"incomplete_details":{"a":01}}', + '"usage":{"input_tokens":1 2}', + '"error":{"a":1 2}', +])("a malformed non-ceiling member %s still fails the salvage (#207 round 10)", (member) => { + const scanner = createResponseSalvage(1024) + const text = `{${member},"base_resp":{"status_code":1008},"pad":"${"x".repeat(2048)}"}` + for (const byte of encoder.encode(text)) scanner.push(byte) + expect(scanner.finish()).toBeNull() +}) + +test("a malformed optional code beside a malformed status_code still fails the salvage (#207 round 10)", () => { + const scanner = createResponseSalvage(1024) + const text = `{"usage":{"input_tokens":1},"base_resp":{"status_code":NaN},"code":+1,"pad":"${"x".repeat(2048)}"}` + for (const byte of encoder.encode(text)) scanner.push(byte) + expect(scanner.finish()).toBeNull() +}) + +test("a malformed status_code before usage drops only itself (#207 round 10)", () => { + const scanner = createResponseSalvage(1024) + const text = `{"base_resp":{"status_code":01008},"usage":{"input_tokens":11,"output_tokens":22},"pad":"${"x".repeat(2048)}"}` + for (const byte of encoder.encode(text)) scanner.push(byte) + expect(scanner.finish()).toEqual({ usage: { input_tokens: 11, output_tokens: 22 } }) +}) + +test.each([ + [32, true], + [33, false], +])("a %i-byte status_code is kept: %s (the 32-byte ceiling, #207 round 11)", (length, kept) => { + const scanner = createResponseSalvage(1024) + const code = `1${"0".repeat(length - 1)}` + const text = `{"usage":{"input_tokens":1},"base_resp":{"status_code":${code}},"pad":"${"x".repeat(2048)}"}` + for (const byte of encoder.encode(text)) scanner.push(byte) + const rebuilt = scanner.finish() + if (kept) expect(rebuilt).toMatchObject({ base_resp: { status_code: Number(code) } }) + else expect(rebuilt).toEqual({ usage: { input_tokens: 1 } }) +}) + +test.each([ + '"base_resp":{"status_code":0} {"status_code":1008}', + '"base_resp":{"status_code":0}{"status_code":1008}', + '"base_resp":0 {"status_code":1008}', + '"base_resp":{"status_code":0},{"status_code":1008}', + '"base_resp" {"status_code":1008}', + '"a":1 2,"base_resp":{"status_code":1008}', +])("invalid JSON around base_resp never yields a code (#207 round 12): %s", (members) => { + const scanner = createResponseSalvage(1024) + const text = `{"usage":{"input_tokens":1},${members},"pad":"${"x".repeat(2048)}"}` + for (const byte of encoder.encode(text)) scanner.push(byte) + expect(scanner.finish()?.base_resp).toBeUndefined() +}) + +test.each([ + '"base_resp":10,"base_resp":{"status_code":1008}', + '"usage":{"input_tokens":1,"output_tokens":2},"base_resp":{"status_msg":"x","status_code":1008}', + '"list":[1,2,{"a":3}],"base_resp":{"status_code":1008}', +])("valid JSON keeps its code (#207 round 12): %s", (members) => { + const scanner = createResponseSalvage(1024) + const text = `{${members},"pad":"${"x".repeat(2048)}"}` + expect(JSON.parse(text).base_resp.status_code).toBe(1008) + for (const byte of encoder.encode(text)) scanner.push(byte) + expect(scanner.finish()).toMatchObject({ base_resp: { status_code: 1008 } }) +}) + +test("a violation inside an error capture that borrowed base_resp never gives the code back (#207 round 12)", () => { + const scanner = createResponseSalvage(64) + // A 20-digit code stays lent long enough for the violation (`1 2`) to arrive before the drop. + const text = `{"usage":{"p":"${"u".repeat(34)}"},"base_resp":{"status_code":10080000000000000000},"code":{"a":1 2,"b":"${"z".repeat(40)}"},"pad":"${"x".repeat(200)}"}` + for (const byte of encoder.encode(text)) { + scanner.push(byte) + expect(scanner.retainedBytes).toBeLessThanOrEqual(64) + } + expect(scanner.finish()?.base_resp).toBeUndefined() +}) + +test.each([ + // A violation after the code releases it; one under an earlier root key keeps the block in force. + ['"base_resp":{"status_code":1008},"a":1 2', { usage: { input_tokens: 1 } }], + ['"base_resp":{"status_code":1008},"q":{"r":1 2}', { usage: { input_tokens: 1 } }], + ['"q":{"r":1 2},"base_resp":{"status_code":1008}', { usage: { input_tokens: 1 } }], + // The block drops only the ceiling slot: other evidence stays. + [ + '"code":"quota","base_resp":{"status_code":0 1008}', + { usage: { input_tokens: 1 }, code: "quota" }, + ], +])( + "a structural violation blocks only the code, before or after it (#207 round 13): %s", + (members, expected) => { + const scanner = createResponseSalvage(1024) + const text = `{"usage":{"input_tokens":1},${members},"pad":"${"x".repeat(2048)}"}` + for (const byte of encoder.encode(text)) scanner.push(byte) + expect(scanner.finish()).toEqual(expected) + }, +) + +test("a block drops a code still being captured, so its bytes stop counting (#207 round 13)", () => { + const scanner = createResponseSalvage(1024) + const text = `{"usage":{"input_tokens":1},"base_resp":{"status_code":1008"x"},"pad":"${"x".repeat(2048)}"}` + for (const byte of encoder.encode(text)) scanner.push(byte) + expect(scanner.retainedBytes).toBe('{"input_tokens":1}'.length) + expect(scanner.finish()).toEqual({ usage: { input_tokens: 1 } }) +}) diff --git a/apps/web/package.json b/apps/web/package.json index 2a2d21f8..0a93b78f 100644 --- a/apps/web/package.json +++ b/apps/web/package.json @@ -1,6 +1,6 @@ { "name": "@multi-ai-router/web", - "version": "2.28.0", + "version": "2.28.1", "private": true, "type": "module", "scripts": { diff --git a/docs/idea/03-providers.md b/docs/idea/03-providers.md index d9352479..1fff05e5 100644 --- a/docs/idea/03-providers.md +++ b/docs/idea/03-providers.md @@ -481,6 +481,7 @@ and the router keeps selecting a credential that can no longer serve a request. | Provider | Signal the driver keys on | Why it is not just the status code | |---|---|---| | `anthropic-api` | message matches `credit balance is too low`; or `error.type` is `billing_error` | Anthropic answers a spent console balance with **`400 invalid_request_error`**. Read the status alone and it classifies as a client mistake | +| `openai-oauth` | `detail.code` / `error.code` / `type` in `usage_limit_reached`, `rate_limit_exceeded` (→ `rate-limited`); `account_deactivated`, `insufficient_quota`, `credit_balance_exhausted`, `organization_spend_limit_exceeded`, `project_spend_limit_exceeded` (→ `credits-exhausted`: on a `429`, by code, and as a `type` only `insufficient_quota`; in a stream, only from `response.error.code` of a frame whose own JSON `type` is `response.failed`) | Codex answers a spent balance with **`429`**, as OpenAI's API does, and the Codex CLI treats those four quota codes as fatal (`QuotaExceeded`, openai/codex `codex-rs/codex-api/src/sse/responses_error.rs`). The stream hook shares the lists ([05-routing-and-failover.md](05-routing-and-failover.md)) | | `openai-api` | `error.code` / `error.type` in `insufficient_quota`, `billing_hard_limit_reached`, `account_deactivated` | OpenAI returns **`429`** for a spent balance — the same status it uses for real rate limiting. Status alone marks a dead account `cooling_down` and retries it on a timer forever | | `openrouter` | message matches `insufficient credits` / `requires more credits` / `add more using`; or `error.code` is `402`; or message matches `Key limit exceeded` (a `total` cap, or one naming no window) | OpenRouter echoes the numeric HTTP status back in `error.code` rather than a string. The wording rule is what still catches it when the `402` is proxied through with another status. A per-key spend cap arrives as a **`403`** — read on status alone it was `auth`, and a valid key sat in "credential rejected" (prod, 2026-10-04). A cap that resets `daily` / `weekly` / `monthly` is the opposite case: `rate-limited`, estimated re-test | | `zai` | `error.code` in `1113`, `1112`; or message matches `insufficient balance` / `balance is insufficient` / `account balance` | Numeric vendor codes on both surfaces. `130x` is throttling and `100x` is auth — three families that all arrive as one HTTP status | diff --git a/docs/idea/05-routing-and-failover.md b/docs/idea/05-routing-and-failover.md index 73e74b02..f9e1e8dc 100644 --- a/docs/idea/05-routing-and-failover.md +++ b/docs/idea/05-routing-and-failover.md @@ -456,7 +456,7 @@ that pool — the policy never runs across the union. | Connection failure / timeout | Retry the next candidate. Count toward the failure streak. | | `4xx` other than `429` | **Do not retry.** A bad request is bad at every account; returning the upstream's error is the honest answer. | | The upstream refuses the **model name** (`model-unsupported`, driver-classified: the `claude` CLI's "There's an issue with the selected model … may not exist or you may not have access to it", Codex's "`` model is not supported when using Codex with a ChatGPT account") | **Retry the next candidate**, before any byte reaches the client. **No breaker strike, no cooldown, no recovery probe** — the account answered correctly about a name it never served, and the next account may serve that vendor. Recorded as `client_error`. When every candidate refused it, the client hears the first refusal itself: Codex's own `400`, or for a subscription a router-authored `404` "the Claude subscription does not serve the requested model". Routing's [model family](#candidate-filtering) keeps most of these from being attempted at all; this row is for what the family admits and the upstream still refuses. Until 2026-10-04 the first read `unknown` (a `502` that struck the breaker) and the second `invalid-request` (no failover). | -| **In-stream refusal on a 200 stream** (Codex `response.failed` or bare `error` event; classified by the `openai-oauth` driver hook, not the HTTP `rules` array) | Codes `invalid_prompt`, `invalid_request_error`, `context_length_exceeded`, `unsupported_parameter`, `unsupported_value`, `invalid_json_schema` → `invalid-request` (failover reads `client-error`; the breaker is **not** struck — a request-shaped refusal is bad at every account, but the account is not unwell). Codes `usage_limit_reached`, `rate_limit_exceeded` → `rate-limited`; the observer forces `rateLimit: null` so the verdict carries no estimated reset, and the breaker trips on the **first** hit (a stream-only rate limit is a fact about the account, not about the request). Code `account_deactivated` → `credits-exhausted`; the account moves to `exhausted` on the first hit (no clock, human action only). Any other code, or no code, → `server-error` (fail-closed; three in a row cool the account). The verdict's `errorClass` is the closed `codex:stream-error:` signal (or `codex:stream-error:other`); the usage row carries the signal, never the upstream message. The code decides when there is one (top level, then the nested error object's `code`): a listed code maps as above, any other code is `server-error`. Only when there is no code is the nested error object's `type` read, and only for `usage_limit_reached` / `rate_limit_exceeded` / `account_deactivated` (the same lists the HTTP rules use, which read `type` even beside a code; in-stream, a code always decides): `invalid_request_error` is OpenAI's catch-all *type*, carried by account-level refusals such as `invalid_api_key` too, so a type never marks a refusal request-shaped. The flat event's own `type` is the event name `error` and is never read as a code. A frame too large for the observation cap is salvaged with its `error` and `code` kept, so a large echoed prompt does not hide the code; an error object too large to keep is dropped and the failure classifies `server-error`, never success. | +| **In-stream refusal on a 200 stream** (Codex `response.failed` or bare `error` event; classified by the `openai-oauth` driver hook, not the HTTP `rules` array) | Codes `invalid_prompt`, `invalid_request_error`, `context_length_exceeded`, `unsupported_parameter`, `unsupported_value`, `invalid_json_schema` → `invalid-request` (failover reads `client-error`; the breaker is **not** struck — a request-shaped refusal is bad at every account, but the account is not unwell). Codes `usage_limit_reached`, `rate_limit_exceeded` → `rate-limited`; the observer forces `rateLimit: null` so the verdict carries no estimated reset, and the breaker trips on the **first** hit (a stream-only rate limit is a fact about the account, not about the request). Code `account_deactivated`, and only from `response.error.code` of a frame whose own JSON `type` is `response.failed` (the one place the Codex CLI reads them; every other shape is retried or skipped there) `insufficient_quota`, `credit_balance_exhausted`, `organization_spend_limit_exceeded`, `project_spend_limit_exceeded` → `credits-exhausted`; the account moves to `exhausted` on the first hit (no clock, human action only). Any other code, or no code, → `server-error` (fail-closed; three in a row cool the account). The verdict's `errorClass` is the closed `codex:stream-error:` signal (or `codex:stream-error:other`); the usage row carries the signal, never the upstream message. The code decides when there is one (top level, then the nested error object's `code`): a listed code maps as above, any other code is `server-error`. Exception: a quota code at `response.error.code` of a frame whose own JSON `type` is `response.failed` decides even beside a root `code` or `error`, which the Codex CLI ignores on that frame. Only when there is no code is the nested error object's `type` read, and only for the limit and deactivated codes above, never a quota code, which the Codex CLI's stream parser reads only as a code (the same lists the HTTP rules use, which read `type` even beside a code; in-stream, a code always decides): `invalid_request_error` is OpenAI's catch-all *type*, carried by account-level refusals such as `invalid_api_key` too, so a type never marks a refusal request-shaped. The flat event's own `type` is the event name `error` and is never read as a code. A frame too large for the observation cap is salvaged with its `error` and `code` kept (and MiniMax's `base_resp.status_code`, so an oversized in-body MiniMax failure still reads as one), so a large echoed prompt does not hide the code; an error object too large to keep is dropped and the failure classifies `server-error`, never success. | | `401` / `403` | **Retry the next candidate** — before any byte has reached the client, like every row above. Move the account to `needs_reauth` (OAuth), or — for an API key, or a no-auth endpoint that has grown something in front of it, neither of which has a login to re-run — to a `cooling_down` labeled **`credential-rejected`**, `ROUTING_AUTH_FAILURE_COOLDOWN_MS` long (15 min default, doubling per refused re-test up to `ROUTING_AUTH_FAILURE_MAX_COOLDOWN_MS`, 4 h), re-tested by one half-open probe when it ends and reported meanwhile as needing a human (see *A rejected key is a re-tested cooldown* below). Through v2.14.0 this row said `disabled`. A rejected credential is *account*-scoped, not request-scoped: the next candidate authenticates with its own. Until v2.9 this row read "do not retry", which meant the first request to land on a subscription whose 30-day login had expired failed `502` while five healthy subscriptions sat beside it; only the *next* request routed around the parked account. The `502` is still what the client hears when **every** candidate failed that way — every one, because the default attempt bound is the pool itself, not a number (`ROUTING_MAX_ATTEMPTS` unset). | Rules: diff --git a/docs/idea/07-security.md b/docs/idea/07-security.md index 02bc47ee..9fe68152 100644 --- a/docs/idea/07-security.md +++ b/docs/idea/07-security.md @@ -199,8 +199,8 @@ key into the admin plane. Conversely, an admin session is not accepted on `/v1/* |---|---| | Default-on | The redactor runs on every log record, not at call sites. Forgetting to redact is not a possible mistake | | Catches by field name | `Authorization` / `Proxy-Authorization` / `Cookie` / `Set-Cookie` / `x-api-key` / `x-goog-api-key` / `x-goog-user-project`, anything containing `password`, `secret`, `token`, `credential`, `verifier`, OAuth `code`, `state`, `code_verifier` | -| Catches by value shape | `mar_live_…`, `sk-…`, `Bearer …`, any JWT (`eyJ….….…` — a ChatGPT/Codex access token is one), URL userinfo (`postgres://user:pass@host`), a credential in a query string (`?…key=` / `?…token=` / `?…secret=` / `?code=` / `?code_verifier=` / `?state=` — a whole OAuth callback URL logged as one value), `AIza…`, `gh[pousr]_…` / `github_pat_…`, `xai-…`, `gsk_…` | -| Keeps the diagnostic | A connection string keeps its host and a URL keeps its path — only the credential inside is replaced. Everything else is replaced whole | +| Catches by value shape | `mar_live_…`, `sk-…`, `Bearer …`, any JWT (`eyJ….….…` — a ChatGPT/Codex access token is one), URL userinfo (`postgres://user:pass@host`; a password holding an unencoded `/` too, up to 256 characters and holding none of `:` `?` `#` `&` `=` `"` except base64 `=` padding before the `@`), a credential in a query string (`?…key=` / `?…token=` / `?…secret=` / `?code=` / `?code_verifier=` / `?state=` — a whole OAuth callback URL logged as one value), `AIza…`, `gh[pousr]_…` / `github_pat_…`, `xai-…`, `gsk_…` | +| Keeps the diagnostic | A connection string keeps its host and a URL keeps its path — only the credential inside is replaced. Everything else is replaced whole. One price: a URL with a port and an `@` later in its path (`https://host:8080/a@b`, `https://registry.npmjs.org:443/@scope/pkg`) loses its host and port, because a password may hold an unencoded `/` | | Tested unit | Pure function, its own test file, with one assertion per secret shape and a nested case three levels deep. A new provider credential shape means a new fixture | | Bodies | Request and response bodies are never logged, at any level. Prompts are user data | | Client-facing errors | No credential, no token fragment, no decrypted material, and no upstream account identity ever appears in an error returned to a client — see [06-protocol-translation.md](06-protocol-translation.md) | diff --git a/docs/idea/08-observability.md b/docs/idea/08-observability.md index 42fca944..5351bb47 100644 --- a/docs/idea/08-observability.md +++ b/docs/idea/08-observability.md @@ -632,7 +632,7 @@ of production told an operator nothing: | Line | Carries | |---|---| | `request failed` (`transport`) | `errorClass`, `errorCode`, `status`, and **`error`**: the thrown `RouterError`'s whole `cause` chain, innermost first, scrubbed and bounded by the logger's `Error` rendering. Every `RouterError` message is router-authored and is already the client-facing body, so the line learns nothing the caller was not told — it just becomes findable. A `translation_failed` line names the field that was refused. The line also carries what the request said about its body, by allowlist — `content-type`, `content-encoding`, `content-length`, `user-agent`, beside the session and client headers every completed line has — so an empty body, a compressed one, and the wrong content type are told apart without a packet capture. Values are bounded to 200 characters; no credential-bearing header is on the list. | -| `upstream attempt failed` (`transport`) | `accountId`, `attempt`, `status`, `failureKind`, plus **`signal`** (which classifier rule decided — `http-status:429`, `claude-sdk:unclassified`), **`reason`** (the router-authored sentence), and **`upstreamMessage`** (the provider's or the SDK's own words, bounded to `LOG_REASON_MAX_CHARS` and scrubbed). | +| `upstream attempt failed` (`transport`) | `accountId`, `attempt`, `status`, `failureKind`, plus **`signal`** (which classifier rule decided — `http-status:429`, `claude-sdk:unclassified`), **`reason`** (the router-authored sentence), and **`upstreamMessage`** (the provider's or the SDK's own words, scrubbed first and then bounded to `LOG_REASON_MAX_CHARS`, as on `upstream stream failure`: the Agent-SDK path hands this line an unredacted message, and a credential cut before scrubbing could leave a fragment. Up to 64 KiB the whole message is scrubbed; past it, the scan ends at the last whitespace before 64 KiB, so no credential's secret part straddles the cut (a message with no whitespace in its first 64 KiB logs only `…`). A cut never splits `[REDACTED]`, and the quote fits the bound after the logger's own pass). | | `upstream stream failure` (`relay`) | `accountId`, `errorClass` (the attempt's closed `errorClass`: a driver hook's signal such as `codex:stream-error:invalid_prompt` or `minimax:base_resp-1008`, else `upstream_protocol_error`, `upstream_incomplete`, `upstream_error` or `upstream_timeout`), and `upstreamMessage` (the driver hook's reading of the upstream's words, redacted by `logging/redact.ts` and then cut to `LOG_REASON_MAX_CHARS`; absent when no hook read one, e.g. a mid-stream transport failure). One line per failed relay, written after the breaker update and the usage row. The verdict is the first one the relay captured, so a Codex refusal followed by a client cancel logs the refusal, not `client_cancelled`. | | `translation dropped fields` (`translate`) | `ingress`, `egress`, `dropped` (the full count), and `fields` — up to twenty ` ` entries, structural only (field paths, tool names, block types, media types; never a value from the body). One line per conversion, not per field. Rationale: [06 § Known lossy edges](06-protocol-translation.md#known-lossy-edges). | diff --git a/docs/idea/09-deployment.md b/docs/idea/09-deployment.md index e2846109..a2aafe10 100644 --- a/docs/idea/09-deployment.md +++ b/docs/idea/09-deployment.md @@ -707,7 +707,7 @@ All integer settings must be finite safe integers; `PORT` cannot exceed 65535. T | Setting | Default | Purpose | |---|---|---| | `UPSTREAM_ERROR_MAX_BYTES` | `65536` | Maximum failed HTTP response body retained; range 1–33554432 bytes. Overflow cancels the body and returns a safe diagnostic. | -| `UPSTREAM_RESPONSE_OBSERVATION_MAX_BYTES` | `65536` | Positive whole bytes, maximum 1048576. Bounds protocol/usage observation state for an upstream response. An over-cap frame or body (e.g. a Codex `response.completed` echoing instructions/tools/output) is not dropped: one streaming pass retains only its shallow `type`/`status`/`incomplete_details`/`usage` members (root, `message.*`, `response.*`), together still bounded by this value, so usage and the terminal are recorded; a frame that cannot be salvaged that way becomes unavailable. Response bytes continue to relay without frame buffering or an invented provider failure. | +| `UPSTREAM_RESPONSE_OBSERVATION_MAX_BYTES` | `65536` | Positive whole bytes, maximum 1048576. Bounds protocol/usage observation state for an upstream response. An over-cap frame or body (e.g. a Codex `response.completed` echoing instructions/tools/output) is not dropped: one streaming pass retains only its shallow `type`/`status`/`incomplete_details`/`usage` members (root, `message.*`, `response.*`), the optional `error`/`code` evidence (root and `response.error`) when it is small, and MiniMax's `base_resp.status_code` (32-byte ceiling; never `status_msg`), together bounded by this value plus at most those 32 bytes held aside while lent to an error capture in progress, so usage and the terminal are recorded; a frame that cannot be salvaged that way becomes unavailable. Response bytes continue to relay without frame buffering or an invented provider failure. | | `ACTIVE_REQUEST_MAX_ENTRIES` | `65536` | Integer 1–1048576. Bounds authenticated request lifetimes per runtime; a full or closed registry refuses before body reading or upstream admission. Shutdown closes admission immediately, permits the existing HTTP drain, then settles remaining accounting before aborting work and draining writers; no extra shutdown budget. | | `ADMIN_LOGIN_MAX_CONCURRENT` | `4` | Maximum password verifications running concurrently per process. | | `ADMIN_LOGIN_MAX_TRACKED_IPS` | `10000` | Maximum tracked login sources; new sources fail closed at capacity. | diff --git a/package.json b/package.json index 7c19934a..3f3bd3ed 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "multi-ai-router", - "version": "2.28.0", + "version": "2.28.1", "private": true, "type": "module", "license": "MIT", diff --git a/packages/core/package.json b/packages/core/package.json index 23ca2dc3..c6072e71 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -1,6 +1,6 @@ { "name": "@multi-ai-router/core", - "version": "2.28.0", + "version": "2.28.1", "private": true, "type": "module", "license": "MIT", diff --git a/packages/core/src/scrub-credentials.ts b/packages/core/src/scrub-credentials.ts index 5ebed2bb..592e2882 100644 --- a/packages/core/src/scrub-credentials.ts +++ b/packages/core/src/scrub-credentials.ts @@ -13,8 +13,24 @@ type ScrubPattern = { readonly pattern: RegExp; readonly replacement: string } const PATTERNS: readonly ScrubPattern[] = [ // URL userinfo — `postgres://user:pass@host/db` is how DATABASE_URL reaches a boot error. - // The scheme and host survive: *what* refused the connection is the diagnostic half. - { pattern: /\b([a-z][a-z0-9+.-]*):\/\/[^\s/@]+@/gi, replacement: `$1://${SCRUBBED}@` }, + // The scheme and host survive: *what* refused the connection is the diagnostic half. The scheme + // is left out of the match (it stays in the text untouched): matching it made a long `a-a-a-…` + // run backtrack quadratically. A lookbehind still requires the scheme main's pattern required + // (a letter at a word boundary, then scheme characters), so text with no real scheme before `://` + // is left to the key=value pattern; it scans back only to the previous non-scheme character, so + // it stays linear. The first branch takes a password holding an unencoded `/` + // (`postgres://u:p/ss@h`), bounded to 256 characters and stopping at `:` so the scan stays short. + // Neither its username nor its password crosses `?`, `#`, `&`, `=` or `"`, so it misses a slash + // password holding one of those. One exception: up to two `=` of base64 padding just before the + // `@`, allowed only when no key name of the key=value pattern sits right before them, so it never + // covers a `key=` whose secret that pattern (anchored on `\b`, not `?` or `&`) must still see: + // `;password=@x` stays that pattern's. The second branch is the original pattern unchanged + // (username-only userinfo, or a password without `/`), so the first only ever adds matches. + { + pattern: + /(?<=\b[a-z][a-z0-9+.-]*):\/\/(?:[^\s/:@?#&="]*:[^\s@"?#:&=]{1,256}(?:(? { const line = 'migration failed: relation "accounts" already exists (migrations/0007.sql)' expect(scrubCredentials(line)).toBe(line) }) + + test("a password holding an unencoded slash is scrubbed whole (#206)", () => { + expect(scrubCredentials("connect failed: postgres://u:p/ss@db:5432/x")).toBe( + "connect failed: postgres://[REDACTED]@db:5432/x", + ) + }) + + test("a base64 slash password with `=` padding is scrubbed (#207 review)", () => { + expect(scrubCredentials("connect failed: postgres://router:Zk3/q+Vb9xQ==@db:5432/app")).toBe( + "connect failed: postgres://[REDACTED]@db:5432/app", + ) + }) + + test("userinfo needs a real scheme in front, as before the scheme left the match (#207 round 3)", () => { + for (const [input, secret] of [ + ["x_y://a:password=@SECRET1", "SECRET1"], + ["1db://pwd=@SECRET2", "SECRET2"], + ["://token=@SECRET3", "SECRET3"], + ] as const) { + expect(scrubCredentials(input)).not.toContain(secret) + } + }) + + test("padding never swallows a key=value key before an `@` (#207 round 3)", () => { + for (const [input, secret] of [ + ["jdbc:postgresql://host:5432/db;password=@Hunter2Secret", "Hunter2Secret"], + ["pg://a:/password=@xSECRET", "xSECRET"], + ["redis://cache:6379/0,token=@SSECRET", "SSECRET"], + ] as const) { + expect(scrubCredentials(input)).not.toContain(secret) + } + }) + + test("a query-string secret holding an `@` behind a host:port stays scrubbed (#207 review)", () => { + for (const [input, secret] of [ + ["postgresql://db.internal:5432/app?user=app&password=p@ssw0rd", "ssw0rd"], + ["redis://cache:6379/0?password=hunter@2secret", "2secret"], + ["connect https://h:8443/cb?token=tok@en_SECRET_TAIL failed", "SECRET_TAIL"], + ["postgres://db?sslmode=require&password=Ab:c/d@ef9", "ef9"], + ["GET https://host:8080/v1&api_key=ab@cdSECRET", "cdSECRET"], + ] as const) { + expect(scrubCredentials(input)).not.toContain(secret) + } + }) + + test("a JWT whose header is over 512 characters is still scrubbed whole", () => { + const segment = (bytes: number) => Buffer.from("k".repeat(bytes)).toString("base64url") + const jwt = `eyJ${segment(420)}.${segment(200)}.${segment(256)}` + expect(jwt.indexOf(".")).toBeGreaterThan(512) + expect(scrubCredentials(`proof ${jwt} end`)).toBe("proof [REDACTED] end") + }) +}) + +describe("scrubCredentials cost (#206)", () => { + test("long scheme-like and JWT-like runs are scanned in linear time", () => { + // The scheme quantifier and the unbounded JWT header made these quadratic: 16 KB of `a-` took + // about 4 s. The bound is loose, the regression is not. + for (const text of [ + "a-".repeat(32 * 1024), + "eyJaaaaa-".repeat(8 * 1024), + "eyJ-".repeat(16 * 1024), + "a://b:".repeat(10 * 1024), + "://u:p/".repeat(10 * 1024), + "://".repeat(64 * 1024), + "://:".repeat(48 * 1024), + `${"ab-".repeat(2000)}://`.repeat(40), + ]) { + const started = performance.now() + scrubCredentials(text) + expect(performance.now() - started).toBeLessThan(500) + } + }) }) diff --git a/packages/db/package.json b/packages/db/package.json index d0970b12..d0387dea 100644 --- a/packages/db/package.json +++ b/packages/db/package.json @@ -1,6 +1,6 @@ { "name": "@multi-ai-router/db", - "version": "2.28.0", + "version": "2.28.1", "private": true, "type": "module", "license": "MIT",