fix(http2): enforce payload limit on HTTP/2 server request body to prevent heap exhaustion DoS

[EvenLjj]

Background

HTTP/2 (h2/h2c) server path accumulated DATA frames in Http2ServerChannelHandler.onDataRead without comparing the accumulated size against ServerTransportConfig.getPayload().

The default 8 MiB payload limit (transport.payload.max) was only applied to the HTTP/1.1 pipeline via HttpObjectAggregator; the HTTP/2 / h2c pipelines had no equivalent limit.

An unauthenticated client that can reach an h2 or h2c listening port can keep a request stream open and continuously send legal DATA frames without sending END_STREAM. Because onDataRead returns the processed byte count, flow control issues WINDOW_UPDATE, so HTTP/2 flow control does not cap the application-layer request body. The server retains attacker data before routing / auth / deserialization / RPC invocation until the JVM heap is exhausted.

• CWE: CWE-770 (Allocation of Resources Without Limits or Throttling)
• CVSS v3.1: 7.5 (AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H)

Reproduction (loopback only)

A stdlib Python client sending up to 80 MiB of DATA frames on a single stream (respecting WINDOW_UPDATE, never sending END_STREAM) against an h2c target running with -Xmx64m -XX:+ExitOnOutOfMemoryError produced:

TARGET_CONNECTION_CLOSED after 12599142 bytes
Terminating due to java.lang.OutOfMemoryError: Java heap space

12,599,142 bytes exceeds the default 8 MiB (8,388,608) limit, confirming the HTTP/2 path did not enforce it.

Fix

• Http2ServerChannelHandler: accept maxContentLength (from transport.payload.max — same source as HttpObjectAggregator). In onDataRead, before writeBytes, check accumulated + frameSize > maxContentLength; on overflow release the accumulated ByteBuf, reset the stream via writeRstStream(PROTOCOL_ERROR), and do not call handleRequest. Return processed bytes so flow control stays correct.
• Release the accumulated msg after the synchronous handleRequest (this also fixes a pre-existing per-request ByteBuf leak: AbstractHttpServerTask never released the request buffer, so every h2 request leaked msg).
• Register an Http2ConnectionAdapter to release any residual buffer on stream close (connection drop / RST_STREAM / normal end) as a backstop.
• Http2ChannelHandlerBuilder / Http2ServerChannelInitializer: thread maxHttpContentLength through to the handler for all h2/h2c paths (TLS NPN, prior-knowledge, HTTP/1.1 upgrade).
• New regression test Http2ClearTextPayloadLimitTest: with payload=1024, a valid small request is served (200, method invoked); an oversized request is reset / no 200 and the service method is not invoked.

Verification

• remoting-http main + test-integration test code compiles.
• New test asserts the limit is enforced; server logs HTTP/2 request body exceeded max payload 1024 bytes, reset stream ....
• Full h2/h2c integration suite (Http2ClearText*, H2c*, deadline) — 9 tests, 0 failures, no regression.

Compatibility

Behavior now matches the HTTP/1.1 HttpObjectAggregator(maxContentLength) path. Default transport.payload.max (8 MiB) is unchanged; HTTP/2 simply enforces it too. Users can still override via -Dtransport.payload.max, META-INF/sofa-rpc/rpc-config.json, or ServerConfig.setPayload(...).

Operational note

Before releasing this fix, do not expose h2/h2c listening ports directly to untrusted networks; consider a fronting proxy with request-body limits and conservative JVM memory / OOM policy.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR hardens the HTTP/2 (h2/h2c) server transport by enforcing the existing transport.payload.max request-body limit on the HTTP/2 DATA-frame accumulation path, preventing heap exhaustion DoS scenarios and improving ByteBuf lifecycle cleanup.

Changes:

• Enforce a per-stream max accumulated request-body size in Http2ServerChannelHandler.onDataRead, resetting the stream when exceeded and releasing accumulated buffers.
• Thread maxHttpContentLength through the HTTP/2 server pipeline construction for TLS negotiation, prior-knowledge, and HTTP/1.1 upgrade paths.
• Add an h2c integration regression test validating that oversized payloads do not produce a 200 response and do not invoke the service method.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
test/test-integration/src/test/java/com/alipay/sofa/rpc/transport/http/Http2ClearTextPayloadLimitTest.java	Adds regression coverage for enforcing payload on h2c request bodies.
remoting/remoting-http/src/main/java/com/alipay/sofa/rpc/transport/http/Http2ServerChannelInitializer.java	Passes maxHttpContentLength into all HTTP/2 handler build paths to apply the limit consistently.
remoting/remoting-http/src/main/java/com/alipay/sofa/rpc/transport/http/Http2ServerChannelHandler.java	Implements accumulated DATA-frame size checks, stream reset behavior, and ByteBuf cleanup on end-of-stream/stream-close.
remoting/remoting-http/src/main/java/com/alipay/sofa/rpc/transport/http/Http2ChannelHandlerBuilder.java	Propagates the configured max content length into the HTTP/2 server channel handler.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.