fix(sse): gate structural chat admission shedding on real heap pressure - #10437
Open
diegosouzapw wants to merge 2 commits into
Open
fix(sse): gate structural chat admission shedding on real heap pressure#10437diegosouzapw wants to merge 2 commits into
diegosouzapw wants to merge 2 commits into
Conversation
Closes #10183, Closes #10268 3.8.49 (#9654/#9940) replaced the 3.8.48 heap-ratio shed (heapUsed/heapLimit >= 0.75) in chatBodyAdmission.ts with an unconditional CHAT_MAX_HEAVY_IN_FLIGHT=1 structural lease. A second concurrent "structurally heavy" chat request (>=200 messages, >=64 tools, or >=32k estimated tokens — routine for coding-agent fan-out like Hermes/Cursor/Claude Code) was hard-rejected with a retryable HTTP 503 chat_admission_busy/structure_limit regardless of actual heap pressure, even on a host with ample free RAM. Restore the heap-conditional gate as an ADDITIONAL check layered on top of (not a replacement for) the #9654 bounded-concurrency / per-connection-lane protection: when heavyweight capacity is busy, only enter the bounded-wait/shed path when a live heap-pressure probe (heapUsed / v8 heap_size_limit >= OMNIROUTE_CHAT_ADMISSION_HEAP_SHED_RATIO, default 0.75) confirms real pressure. A healthy heap now admits the second heavy request immediately via a no-op lease instead of parking or shedding it. The probe is injectable via admitChatStructure({ heapPressureCheck }) for deterministic tests. Regression tests: - tests/unit/bug-10183-admission-heavy-healthy-heap.test.ts (new, permanent): healthy-heap 2nd heavy request now admitted (was RED); genuinely pressured heap still sheds it. - tests/unit/probe-10268-structural-503.test.ts (promoted to permanent): the exact reported 503 chat_admission_busy shape is still produced under real heap pressure, and the same fan-out is admitted on a healthy heap. - tests/unit/chat-body-admission.test.ts, tests/unit/chat-body-admission-queue.test.ts, tests/unit/per-connection-admission-9654.test.ts updated to inject heapPressureCheck: () => true where they exercise the busy/shed path, preserving #9654/#4380 coverage. Gates run: npm run typecheck:core (clean), eslint --suppressions-location config/quality/eslint-suppressions.json on changed files (clean), scripts/check/check-file-size.mjs (OK), scripts/check/check-test-discovery.mjs (OK), focused admission suite (68/68 passing) and npm run test:unit (in progress at commit time under heavy shared-devbox contention from a 13-way parallel session fan-out; no admission-related failures observed through 1873 lines of output, the sole failure seen was a pre-existing unrelated proxy/search timeout consistent with known load-induced flakiness, not a regression from this change).⚠️ base-red inherited: #9985 — ESLint errors (2) from #10250
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #10183, Closes #10268
Root cause
Both issues share one root cause in
src/shared/middleware/chatBodyAdmission.ts. 3.8.49 (#9654/#9940) replaced the 3.8.48 heap-ratio shed (heapUsed/heapLimit >= 0.75) with an unconditionalCHAT_MAX_HEAVY_IN_FLIGHT=1structural lease. A second concurrent "structurally heavy" chat request (messages >= 200,tools >= 64, or estimated tokens>= 32000— routine for coding-agent fan-out such as Hermes/Cursor/Claude Code) is hard-rejected with a retryable HTTP 503chat_admission_busy/structure_limitafter a bounded queue wait, regardless of actual heap pressure, even on a host with ample free RAM.Fix
Re-gate heavyweight shedding behind a real heap-pressure probe, layered on top of (not replacing) the #9654 bounded-concurrency / per-connection-lane protection and the #4380 heap-amplification-OOM valve:
heapUsed / v8 heap_size_limit >= OMNIROUTE_CHAT_ADMISSION_HEAP_SHED_RATIO(default0.75, restoring 3.8.48 semantics). A healthy heap admits the second heavy request immediately via a no-op lease instead of parking or shedding it.admitChatStructure({ heapPressureCheck })) for deterministic tests; defaults to liveprocess.memoryUsage()/v8.getHeapStatistics().Regression tests (TDD, Hard Rule #18)
tests/unit/bug-10183-admission-heavy-healthy-heap.test.ts(new, permanent) — RED→GREEN: a 2nd heavy request on a healthy heap is now admitted; a genuinely pressured heap still sheds it (preserves the [BUG] Large /v1/chat/completions request crashes server with Node heap OOM #4380 valve).tests/unit/probe-10268-structural-503.test.ts(promoted to permanent) — the exact reported 503chat_admission_busyshape is still produced under real heap pressure, and the same Hermes/Cursor-style fan-out is admitted on a healthy heap.tests/unit/chat-body-admission.test.ts,tests/unit/chat-body-admission-queue.test.ts,tests/unit/per-connection-admission-9654.test.tsupdated to injectheapPressureCheck: () => truewhere they exercise the busy/shed path, so existing feat(backend): Per-connection virtual admission lanes for agentic workloads #9654/[BUG] Large /v1/chat/completions request crashes server with Node heap OOM #4380 coverage is preserved rather than weakened.Fail→pass evidence (focused suite, 68/68 passing):
Gates run
npm run typecheck:core— cleannpx eslint --suppressions-location config/quality/eslint-suppressions.json <changed files>— cleannode scripts/check/check-file-size.mjs— OK, no new frozen-file growthnode scripts/check/check-test-discovery.mjs— OK, new test files discoveredchat-body-admission*,per-connection-admission-9654,bug-10183-*,probe-10268-*) — 68/68 passingnpm run test:unit(full suite) — run under heavy shared-devbox contention from a concurrent 13-way session fan-out; no admission-related failures observed through the run, sole failure seen was a pre-existing unrelatedPOST /v1/searchproxy timeout consistent with known load-induced flakiness (not touching this file). CI will confirm the full suite green.