Skip to content

fix(sse): gate structural chat admission shedding on real heap pressure - #10437

Open
diegosouzapw wants to merge 2 commits into
release/v3.8.50from
fix/10183-10268-admission-503-structural
Open

fix(sse): gate structural chat admission shedding on real heap pressure#10437
diegosouzapw wants to merge 2 commits into
release/v3.8.50from
fix/10183-10268-admission-503-structural

Conversation

@diegosouzapw

Copy link
Copy Markdown
Owner

Closes #10183, Closes #10268

Root cause

Both issues share one root cause in src/shared/middleware/chatBodyAdmission.ts. 3.8.49 (#9654/#9940) replaced the 3.8.48 heap-ratio shed (heapUsed/heapLimit >= 0.75) with an unconditional CHAT_MAX_HEAVY_IN_FLIGHT=1 structural lease. A second concurrent "structurally heavy" chat request (messages >= 200, tools >= 64, or estimated tokens >= 32000 — routine for coding-agent fan-out such as Hermes/Cursor/Claude Code) is hard-rejected with a retryable HTTP 503 chat_admission_busy / structure_limit after a bounded queue wait, regardless of actual heap pressure, even on a host with ample free RAM.

Fix

Re-gate heavyweight shedding behind a real heap-pressure probe, layered on top of (not replacing) the #9654 bounded-concurrency / per-connection-lane protection and the #4380 heap-amplification-OOM valve:

  • Fast path unchanged: if heavyweight capacity is free, admit immediately.
  • When capacity is busy, only enter the bounded-wait/shed path if heapUsed / v8 heap_size_limit >= OMNIROUTE_CHAT_ADMISSION_HEAP_SHED_RATIO (default 0.75, restoring 3.8.48 semantics). A healthy heap admits the second heavy request immediately via a no-op lease instead of parking or shedding it.
  • The probe is injectable (admitChatStructure({ heapPressureCheck })) for deterministic tests; defaults to live process.memoryUsage() / v8.getHeapStatistics().

Regression tests (TDD, Hard Rule #18)

Fail→pass evidence (focused suite, 68/68 passing):

✔ bug-10183: second concurrent heavy request admitted on a healthy heap
✔ bug-10183: a genuinely pressured heap still sheds the 2nd heavy request
✔ #10268: 2nd structurally-heavy agent request is rejected 503 (chat_admission_busy) under real heap pressure
✔ #10268: 2nd structurally-heavy agent request is admitted on a healthy heap (the fix)

Gates run

  • npm run typecheck:core — clean
  • npx eslint --suppressions-location config/quality/eslint-suppressions.json <changed files> — clean
  • node scripts/check/check-file-size.mjs — OK, no new frozen-file growth
  • node scripts/check/check-test-discovery.mjs — OK, new test files discovered
  • Focused admission suite (chat-body-admission*, per-connection-admission-9654, bug-10183-*, probe-10268-*) — 68/68 passing
  • Husky pre-commit (lint-staged, docs-sync, any-budget, tracked-artifacts) — passed
  • npm run test:unit (full suite) — run under heavy shared-devbox contention from a concurrent 13-way session fan-out; no admission-related failures observed through the run, sole failure seen was a pre-existing unrelated POST /v1/search proxy timeout consistent with known load-induced flakiness (not touching this file). CI will confirm the full suite green.

⚠️ base-red inherited: #9985 — ESLint errors (2) from #10250 (unrelated i18n PT-PT translation regression, not touched by this PR).

Closes #10183, Closes #10268

3.8.49 (#9654/#9940) replaced the 3.8.48 heap-ratio shed
(heapUsed/heapLimit >= 0.75) in chatBodyAdmission.ts with an
unconditional CHAT_MAX_HEAVY_IN_FLIGHT=1 structural lease. A second
concurrent "structurally heavy" chat request (>=200 messages, >=64
tools, or >=32k estimated tokens — routine for coding-agent fan-out
like Hermes/Cursor/Claude Code) was hard-rejected with a retryable
HTTP 503 chat_admission_busy/structure_limit regardless of actual
heap pressure, even on a host with ample free RAM.

Restore the heap-conditional gate as an ADDITIONAL check layered on
top of (not a replacement for) the #9654 bounded-concurrency /
per-connection-lane protection: when heavyweight capacity is busy,
only enter the bounded-wait/shed path when a live heap-pressure probe
(heapUsed / v8 heap_size_limit >= OMNIROUTE_CHAT_ADMISSION_HEAP_SHED_RATIO,
default 0.75) confirms real pressure. A healthy heap now admits the
second heavy request immediately via a no-op lease instead of parking
or shedding it. The probe is injectable via
admitChatStructure({ heapPressureCheck }) for deterministic tests.

Regression tests:
- tests/unit/bug-10183-admission-heavy-healthy-heap.test.ts (new,
  permanent): healthy-heap 2nd heavy request now admitted (was RED);
  genuinely pressured heap still sheds it.
- tests/unit/probe-10268-structural-503.test.ts (promoted to
  permanent): the exact reported 503 chat_admission_busy shape is
  still produced under real heap pressure, and the same fan-out is
  admitted on a healthy heap.
- tests/unit/chat-body-admission.test.ts,
  tests/unit/chat-body-admission-queue.test.ts,
  tests/unit/per-connection-admission-9654.test.ts updated to inject
  heapPressureCheck: () => true where they exercise the busy/shed
  path, preserving #9654/#4380 coverage.

Gates run: npm run typecheck:core (clean), eslint --suppressions-location
config/quality/eslint-suppressions.json on changed files (clean),
scripts/check/check-file-size.mjs (OK), scripts/check/check-test-discovery.mjs
(OK), focused admission suite (68/68 passing) and npm run test:unit
(in progress at commit time under heavy shared-devbox contention from
a 13-way parallel session fan-out; no admission-related failures
observed through 1873 lines of output, the sole failure seen was a
pre-existing unrelated proxy/search timeout consistent with known
load-induced flakiness, not a regression from this change).

⚠️ base-red inherited: #9985 — ESLint errors (2) from #10250
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants