fix(mcp): serve streamable HTTP statelessly behind MCP_STATELESS#1697
Closed
giladresisi wants to merge 2 commits into
Closed
fix(mcp): serve streamable HTTP statelessly behind MCP_STATELESS#1697giladresisi wants to merge 2 commits into
giladresisi wants to merge 2 commits into
Conversation
The MCP service climbs ~330 MB/h from a ~1 GB baseline until it is killed at ~5 GB, roughly twice a day, then restarts and repeats. The climb is load-independent, which points at reconnects rather than traffic. Mechanism: `@mastra/mcp` stores one transport (plus, transitively, one per-session Server holding the whole converted tool schema) per `initialize` request in `streamableHTTPTransports`, and removes it only from `transport.onclose` -- which fires only on an explicit client DELETE. No idle timeout exists. Connectors open a session per conversation and never DELETE, so the map grows for the life of the process. Locally each session retains ~76 KB, and 200 `initialize` requests retained 200 sessions across a forced GC. The session design costs us twice. A restart -- an OOM kill or an ordinary deploy -- wipes the map, and every still-connected client keeps presenting a session id the new process has never seen. Those clients do not recover on their own; they must be restarted, and each reconnect leaves another permanently retained session behind. So crashes beget disconnected customer agents, and reconnecting agents beget more of the leak that caused the crash. This is also a dead end upstream: the 2026-07-28 MCP specification removes the `Mcp-Session-Id` header and protocol-level sessions outright, so session-based serving has to go regardless. This commit measures, it does not fix. Nothing about transport behaviour changes; the session map is only read, via `.size` and `.keys()`. What it records, and why in this shape: - Retained heap, sampled from a `PerformanceObserver` immediately after each major GC. `heapUsed` at an arbitrary moment is mostly uncollected garbage; a leak is exactly what survives a full collection. No forced GC, no pauses. - `live` / `new` / `closed` / `abandoned>60m`, from one 15-minute sweep that diffs the session-id key set against the previous one. No per-session timers. `closed=0` beside a growing `abandoned` count is the proof sessions are never released. - `perSession~KB` and `rate=/h`, which say what one `initialize` costs and how many arrive. Together they test whether this leak accounts for the whole 330 MB/h or whether a second one is hiding behind it. - `rss`, `heapTotal` and `external`, to line the numbers up against the container's memory graph. Correlating a log line with the Railway memory graph: retainedHeap <= heapUsed <= heapTotal <= rss <= Railway's number Railway charts the whole container (every process, plus page cache); we report one process's V8 heap. Absolute values will not match, so compare *slopes* over the window between two log lines. If Railway's slope tracks `delta` on retainedHeap, the growth is heap-resident and this leak explains it. If Railway climbs faster, the excess is outside the heap -- `external` catches Buffers, and if neither moves it is reclaimable page cache. `rss - heapTotal - external` is roughly native overhead. Independently, `perSession x rate` predicts the MB/h Railway should show; agreement means the leak is the whole story. Note that V8 releases pages to the OS lazily, so `rss` is sticky and its slope can briefly outrun `retainedHeap` -- trust `retainedHeap` for leak detection and `rss` only to reconcile with the container. `uptimeSeconds` from `/mcp-metrics` (or the boot line reappearing) marks restarts, which are the cliffs in Railway's sawtooth. The boot line also prints the effective V8 heap limit: if the container dies at ~5 GB while the limit is ~2 GB, the heap cannot be what got there, and we are looking at a kernel OOM on RSS, not a V8 FATAL. One summary line per 15 minutes, suppressed entirely while idle. The `/mcp-metrics` endpoint stays 404 unless `MCP_METRICS_TOKEN` is set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Why
---
`@mastra/mcp` stores one transport per `initialize` request in
`streamableHTTPTransports`, and each transport retains a per-session Server
holding the whole converted tool schema. The only removal path is
`transport.onclose`, which fires only on an explicit client DELETE. No idle
timeout exists, and connectors open a session per conversation and never
DELETE. The map therefore grows for the life of the process: the MCP service
climbs ~330 MB/h from a ~1 GB baseline and is killed at ~5 GB about twice a day.
Sessions hurt a second way. Any restart -- an OOM kill or an ordinary deploy --
empties the map, and every connected client keeps presenting a session id the
new process has never seen. Mastra answers an unknown id with 400, where the
spec requires 404 for a terminated session (and obliges the client to
re-initialize on 404). Clients get no recovery signal, so they hang until
restarted, and each reconnect leaves another permanently retained session.
The upcoming 2026-07-28 MCP specification removes `Mcp-Session-Id` and
protocol-level sessions entirely, so session-based serving has to go anyway.
How
---
`serverless: true` routes `startHTTP` to `handleServerlessRequest`, which
serves each request with a transient server and transport and stores nothing.
Applied to all three endpoints (`/mcp-oauth`, `/mcp`, `/mcp/:id`) via a single
`mcpHttpOptions`, gated on `MCP_STATELESS=true` and defaulting to today's
session mode, so merging changes nothing until the variable is set.
This is safe because nothing here depends on session state: auth is bound per
request through `runWithContext` (AsyncLocalStorage), and no tool uses
sampling, elicitation or server-initiated notifications.
Reproduction (session mode)
---------------------------
- 1000 `initialize` requests against a bare MCPServer retained 1000 transports,
heap growing linearly, measured after a forced GC.
- 200 `initialize` requests against the real backend retained 200 sessions and
+15 MB of heap across a forced GC: ~76 KB each. Tool calls reuse a session
and retain nothing, so the cost tracks reconnects, not traffic.
- A single Claude Desktop connection retained 4 sessions within 49 seconds.
- Restarting the backend with Claude Desktop still attached broke every
subsequent tool call with `400 Bad Request: No valid session ID provided`,
while a fresh `initialize` on the same server returned 200. The client never
recovered.
Validation (stateless)
----------------------
- Same 200 `initialize` requests: 0 transports retained, +1 MB of heap.
- Real toolset over the wire -- `tools/list`, `integrationList`,
`integrationSchema`, `uploadFromUrlTool`, `integrationSchedulePostTool` --
all behave identically; drafts are created with the same shape. Exercised
via curl, via `mcp-remote`, and via Claude Desktop.
- `initialize` no longer returns `mcp-session-id`; `tools/list` sent without a
session id returns 200 where session mode returns 400.
- A stale session id from a previous boot is ignored and served normally. The
Claude Desktop client stranded by the session-mode restart above recovered
mid-flight, with no reconnect and no re-adding the connector.
Not related to the internal Postiz agent
----------------------------------------
The in-app agent never crosses this code. `copilot.controller.ts` runs
`mastra.getAgent('postiz')` in process via `@ag-ui/mastra`; the tools are plain
`createTool` functions called directly. It never sends `initialize`, never
enters `startHTTP`, and cannot create or retain a session. No `MCPClient`
exists anywhere in the repo. (Posts it creates are tagged `CreationMethod.MCP`
because `integration.schedule.post.ts` hardcodes that literal for any agent
tool call -- a naming artifact, not a code path.) Only external clients --
claude.ai, ChatGPT, mcp-remote -- reach `startHTTP`, so only they leak, and
only they are affected by this change.
Rollout: set `MCP_STATELESS=true` on the MCP service, confirm `live=0` in the
mcp-sessions log line and a flat memory graph, then the backend service.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Contribution-checker quality warning Heuristics that flagged:
If this is a genuine contribution, please add detail to your PR description and tighten the diff scope before reviewers look at it. |
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
giladresisi
changed the base branch from
chore/mcp-session-instrumentation
to
main
nevo-david
reviewed
Jul 14, 2026
nevo-david
left a comment
nevo-david
left a comment
Contributor
There was a problem hiding this comment.
Unnecessary tons of code, we just need to make it statelss
Collaborator
Author
|
Superseded by #1717 — same fix rebased onto |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
@mastra/mcpretains a transport perinitializerequest, each holding a per-sessionServerwith the whole converted tool schema, released only on an explicit clientDELETEthat connectors never send. The MCP service climbs ~330 MB/h from a ~1 GB baseline and is killed at ~5 GB about twice a day.Sessions hurt a second way. Any restart empties the map, and connected clients keep presenting a session id the new process has never seen. Mastra answers an unknown id with
400, where the spec requires404for a terminated session — and obliges the client to re-initialize on404. Clients get no recovery signal, so they hang until restarted, and each reconnect leaks another session.The 2026-07-28 spec removes protocol-level sessions outright, so this has to go regardless.
How
serverless: trueroutesstartHTTPtohandleServerlessRequest, which serves each request with a transient server and stores nothing. Applied to all three endpoints via onemcpHttpOptions, gated onMCP_STATELESS=trueand defaulting to today's session mode — merging changes nothing until the variable is set.Safe because nothing depends on session state: auth is bound per request through
runWithContext(AsyncLocalStorage), and no tool uses sampling, elicitation, or server-initiated notifications.Reproduced (session mode)
initializevs bare MCPServerinitializevs real backend400 Bad Request: No valid session ID; freshinitializereturns 200; client never recoversValidated (stateless)
initializetools/list,integrationList,integrationSchema,uploadFromUrlTool,integrationSchedulePostToolidentical; drafts same shapemcp-remote, Claude Desktopinitializemcp-session-idtools/listwithout session idNot related to the internal Postiz agent
The in-app agent never crosses this code.
copilot.controller.tsrunsmastra.getAgent('postiz')in-process via@ag-ui/mastra, calling plaincreateToolfunctions directly. It never sendsinitialize, never entersstartHTTP, and cannot create or retain a session. NoMCPClientexists anywhere in the repo.Posts it creates are tagged
CreationMethod.MCPonly becauseintegration.schedule.post.tshardcodes that literal for any agent tool call — a naming artifact, not a code path. Only external clients (claude.ai, ChatGPT,mcp-remote) reachstartHTTP, so only they leak, and only they are affected here.Rollout
Set
MCP_STATELESS=trueon the MCP service, confirmlive=0in themcp-sessionslog line and a flat memory graph, then the backend service. Rollback is unsetting the variable.