Clarify 400 error related to HTTP/2

[jglick]

In some Jenkins-related code I ran into an obstacle that a request made by a java.net.http client directly to the Jenkins controller worked fine, but when the same request was run through an Undertow reverse proxy it would fail. The symptom was that Jetty would respond with a 400 error page with no explanation (and no server-side log messages).

Earlier I had seen a similar symptom specific to ALB on AWS which I suspect had the same root cause, but unconfirmed; worked around at that time by switching to the java.net client, something I just hit upon by accident.

jenkinsci/winstone#532 looked like it might help in the newer (Undertow) case, but it did not.

After

diff --git jetty-core/jetty-server/src/main/java/org/eclipse/jetty/server/Response.java jetty-core/jetty-server/src/main/java/org/eclipse/jetty/server/Response.java
index 82dcc63e4f8..68825b1ba8d 100644
--- jetty-core/jetty-server/src/main/java/org/eclipse/jetty/server/Response.java
+++ jetty-core/jetty-server/src/main/java/org/eclipse/jetty/server/Response.java
@@ -588,6 +588,7 @@ public interface Response extends Content.Sink
         {
             status = httpException.getCode();
             message = httpException.getReason();
+            cause.printStackTrace();
         }
         writeError(request, response, callback, status, message, cause);
     }
diff --git jetty-core/jetty-server/src/main/java/org/eclipse/jetty/server/handler/ErrorHandler.java jetty-core/jetty-server/src/main/java/org/eclipse/jetty/server/handler/ErrorHandler.java
index f694767d37e..03dead4314c 100644
--- jetty-core/jetty-server/src/main/java/org/eclipse/jetty/server/handler/ErrorHandler.java
+++ jetty-core/jetty-server/src/main/java/org/eclipse/jetty/server/handler/ErrorHandler.java
@@ -359,6 +359,7 @@ public class ErrorHandler implements Request.Handler
 
     protected void writeErrorHtmlMessage(Request request, Writer writer, int code, String message, Throwable cause, String uri) throws IOException
     {
+        Thread.dumpStack();
         writer.write("<h2>HTTP ERROR ");
         String status = Integer.toString(code);
         writer.write(status);

I was able to determine that the error was coming from the line of code patched here, introduced in d1e6c77 and retained in #8685. While I do not exactly understand the issue, configuring the client to avoid HTTP/2

builder.version(HttpClient.Version.HTTP_1_1)

solved the problem.

The error page could perhaps include somewhat more information, but I wanted to err on the side of caution when disclosing information to a client; this should be just enough to clue you in that something related to HTTP/2 is involved. Otherwise a 400 error could point to any of a vast array of mistakes.

[jglick]

I should mention that in my case the connection is being made using plain-text HTTP (since this is in a test environment, not production), that the Jenkins controller is not configured to listen to HTTP/2, and the problem occurs whether or not Undertow is configured to allow HTTP/2. Debugging Jetty showed that it was receiving Upgrade: h2c. The only apparent workaround when using the java.net.http client is to explicitly ask for HTTP/1.1.

[jglick]

Another update: yesterday I tested the same java.net.http-based client connecting to a Jenkins controller behind ALB, and again saw the 400 error until I worked around it by forcing the client to use HTTP/1.1. I have not yet tracked down whether the bug is in the JDK client library; Jetty; the way Jenkins embeds Jetty; or the reverse proxy. But the fact that both Undertow and ALB are affected in the same way hints that the reverse proxy, while a triggering factor, is not to blame. (nginx as configured by the Kubernetes ingress-nginx controller does not trigger the problem.)

[sbordet]

@jglick are you able to turn on DEBUG logging in Jetty?
Alternatively, a wireshark network trace?

If the upgrade fails at the line you modified in this PR, then it is a protocol error from the proxy: either the Upgrade header is invalid (e.g. wrong case for h2c), or Connection: Upgrade is missing.

Or, it is a legit Jetty bug, but we need to know what's going on.

[jglick]

are you able to turn on DEBUG logging in Jetty?

Sadly, no. I set the system property described in Jetty docs, but it did nothing. Perhaps something wrong with the Jenkins embedding w.r.t. logging framework bridges (Jenkins uses java.util.logging). Instead I inserted some printlns into Jetty sources and built a local snapshot, which is how I found that Upgrade: h2c was being sent. At the time I do not think I recorded other headers (since I was not familiar with this aspect of HTTP) but I suspect you are correct that Connection: Upgrade was missing, which would lend weight to the hypothesis that both reverse proxies I mentioned are buggy in the same way.

a wireshark network trace?

Yes, that would be the logical next step. Not sure if I can get to it today.

[sbordet]

@jglick BTW, we unfortunately cannot accept this PR until you sign the ECA.

Let us know if you're willing to do so, otherwise we'll take over: this PR has merit and should be merged, but we cannot merge it in the current state.

[jglick]

OK using

sudo tshark -i lo -f "tcp and portrange 30001-65535" -Y "http" -O http

(no idea what these options mean but Copilot came up with it and it seems to work) and rerunning my test reproducing the 400, I see the java.net.http client contacting Undertow (eliding some irrelevant-seeming headers)

POST /jenkins/… HTTP/1.1
Connection: Upgrade, HTTP2-Settings
HTTP2-Settings: AAEAAEAAAAIAAAAAAAMAAAAAAAQBAAAAAAUAAEAAAAYABgAA
Upgrade: h2c
User-Agent: Java-http-client/25.0.2

and Undertow contacting Jenkins/Jetty

POST /jenkins/… HTTP/1.1
X-Forwarded-Server: localhost
Upgrade: h2c
User-Agent: Java-http-client/25.0.2
Connection: keep-alive
HTTP2-Settings: 

So indeed it seems Undertow is mangling things; if it is not trying to support HTTP/2 then it should just be dropping Connection, HTTP2-Settings, and Upgrade. That ALB, which is quite widely used, would be doing the same would be surprising to me, but I did not try to verify that. (Running a sniffer on some EC2 instance used for EKS sounds very hard, but I suppose Jetty code could be instrumented to dump all headers.)

Anyway, the upshot for Jetty is that it seems to be correct in responding with a 400, and is at fault only for neglecting to offer any hint of what is wrong.

[jglick]

I apparently do not have an ECA on file. It would be a fair amount of work to get one. (#1824 and #4438 were merged without it, but not #13309.) This is just a one-line change that I filed as a PR for concreteness but you can just treat it as a bug report since the code change is obvious if you agree.

[sbordet]

I confirm the request from Undertow is wrong, lacking Connection: Upgrade.

Anyway, the upshot for Jetty is that it seems to be correct in responding with a 400, and is at fault only for neglecting to offer any hint of what is wrong.

Correct, we'll fix that.

I apparently do not have an ECA on file. It would be a fair amount of work to get one.

You mean between you and the company that employs you?
Otherwise, let us know if we can help.

In any case, I'll take this over.

[sbordet]

Replaced by #14755.

[jglick]

You mean between you and the company that employs you?

Right, I need to ask lawyers. Doable, just not worth it for a trivial one-line edit.