[Bug] Internal components attempt TLS handshakes against plain-HTTP listeners in behind-proxy mode (Invalid HTTP method: '\x16\x03\x01...' log noise)

[wwwlde]

[Bug] Internal components attempt TLS handshakes against plain-HTTP listeners in behind-proxy mode (Invalid HTTP method: '\x16\x03\x01...' log noise)

Describe the bug

With Behind Reverse Proxy mode active (startup log confirms Reverse proxy mode — binding to 127.0.0.1 only and SSL disabled (handled by reverse proxy)), an internal PegaProx component keeps performing HTTPS requests against its own listeners, which are now plain HTTP. Every attempt fails and both sides dump binary noise into the log.

Evidence from the log payload itself:

• The "invalid method" bytes \x16\x03\x01 are a TLS ClientHello;
• its SNI is localhost → the client connects by hostname, not by IP;
• the cipher list matches Python ssl defaults → the client is PegaProx's own Python code;
• source is always 127.0.0.1 → same container;
• both the main WSGI server (:5000, "Invalid HTTP method") and the VNC WebSocket server (:5001, "Expected GET method") receive these handshakes.

Important timing detail: the first entries appear ~1 second after startup, before any external request had been proxied to the app (the reverse proxy was still returning 502 at that moment). So at least the startup-time occurrences cannot originate from X-Forwarded-Proto — the https scheme must come from persisted config or a computed self-URL.

Suspected cause: the instance first booted with default built-in HTTPS, then was switched to behind-proxy mode. The internal self-URL used for VNC/SSH console WebSocket callbacks (cf. PEGAPROX_URL, default http://127.0.0.1:5000) appears to stay https://localhost:... — persisted from the TLS-enabled era or derived from SSL config — and is not switched back to http:// when behind-proxy mode disables SSL. Secondary candidate: the ACME auto-renewal thread (started unconditionally per the log), though the observed frequency is higher than a daily renewal check.

Functionality is NOT affected (UI, SSE, noVNC, node shell, LXC terminal all work) — the impact is log pollution that masks real errors and trips log-based alerting, plus an internal operation that silently never succeeds.

Workaround being verified: explicitly setting PEGAPROX_URL=http://127.0.0.1:5000 in the container environment.

Steps to Reproduce

1. Fresh install from the official Docker image; let it boot once with default built-in HTTPS enabled.
2. Enable Behind Reverse Proxy (Settings → Server) + set PEGAPROX_BEHIND_PROXY=true; recreate the container.
3. Watch container logs: TLS ClientHello dumps from 127.0.0.1 appear against :5000 and :5001, starting ~1 s after boot and recurring afterwards.

Expected behavior

When behind-proxy mode disables SSL, internal self-requests (console WS callbacks, any self-probes) switch to http://, or read the effective scheme from a single source of truth. Failing that, a single warning line ("skipping X: SSL disabled") instead of repeated binary handshake dumps.

Environment

• PegaProx Version: Beta 0.9.14
• Installation Method: Docker (official ghcr.io/pegaprox/pegaprox:latest)
• OS:
• Browser: Edge 150 (macOS) — not relevant, reproducible with no browser attached
• Behind Reverse Proxy? Yes: Traefik (TLS termination) → nginx (path routing per examples/nginx.conf, sharing the container's network namespace) → PegaProx
• UI?

Logs

Trusted proxies: 172.28.0.0/24,10.0.4.0/26
Reverse proxy mode — binding to 127.0.0.1 only
SSL disabled (handled by reverse proxy)
Starting PegaProx with Gevent WSGIServer (32 greenlets)
HTTP on http://127.0.0.1:5000
WARNING: Running without HTTPS - noVNC console may not work!
...
VNC WebSocket Server ready on ws://127.0.0.1:5001
SSH WebSocket server subprocess started (PID: 21)
...
<gevent._socket3.socket at 0x7f76057fdda0 object, fd=34, family=2, type=1, proto=0>: (from ('127.0.0.1', 34302)) Invalid HTTP method: '\x16\x03\x01\x05ð\x01\x00\x05ì\x03\x03¾Þ(>GWHº\x1d9\x15\x89\x9c\x8c®J¿\x08Ðc4\rô\x11×\x1f%\x06Æ*2\x18 ...\x00\x00\tlocalhost\x00\x0b\x00\x04\x03\x00\x01\x02\x00\n'
<gevent._socket3.socket at 0x7f76046f0f30 object, fd=34, family=2, type=1, proto=0>: (from ('127.0.0.1', 40540)) Expected GET method; Got command='\x16\x03\x01\x05ð\x01\x00\x05ì\x03\x03±S^ôd\x16\x0eÿæÃ'; path='\x12ÑíÃCW2ÜÓ\x86uÀXbõc|@\x81'; raw='\x16\x03\x01\x05ð\x01\x00\x05ì\x03\x03±S^ôd\x16\x0eÿæÃ\x85\x12ÑíÃCW2ÜÓ\x86uÀXbõc|@\x81\n'

Screenshots

N/A — log-only issue.

Checklist

• I have searched existing issues to make sure this is not a duplicate
• I am using the latest version of PegaProx

---

Related observation (Docker + behind-proxy UX): PEGAPROX_BEHIND_PROXY=true alone switches the backend (bind 127.0.0.1, SSL off), but the frontend kept building console WebSocket URLs on page port + 1 (wss://host:444/...) until the UI toggle in Settings → Server was flipped — after which console WS correctly went through 443 per examples/nginx.conf. If intentional, a note on the Reverse Proxy docs page that the UI toggle (not only the env var) is required would save Docker users some debugging; PEGAPROX_URL is also worth mentioning there.

Happy to provide full logs, compose files, or run a patched build to verify a fix.

[mkellermann97]

Really good writeup — you were on the right track, just the wrong culprit. It's the Docker HEALTHCHECK.

It probes https://localhost:5000/api/health first, unconditionally, every 30s. In behind-proxy mode the built-in SSL is off (gevent serves plain HTTP), so that first attempt is a TLS ClientHello — SNI=localhost, Python default ciphers, from 127.0.0.1 — hitting the plain-HTTP socket. That's your \x16\x03\x01 noise, and the "~1s" is just the interval firing right after the start-period. Health was never actually affected (we fall back to http), but the log spam is real.

The internal self-URL you suspected (PEGAPROX_URL) is actually already http:// in behind-proxy mode — reverse_proxy forces the SSL context to None, so the SSH/console callbacks use http. And the ACME thread is out too (daily, talks only to Let's Encrypt). So it was the healthcheck all along.

Fixed on Testing (ships next release): the healthcheck now picks its scheme from PEGAPROX_BEHIND_PROXY and dials 127.0.0.1 instead of the localhost hostname — no stray TLS handshake in proxy mode.

I also fixed your related observation. PEGAPROX_BEHIND_PROXY=true now drives the frontend as well: the console (VNC/SSH) WebSocket URLs are built from reverse_proxy_enabled, which previously only the Settings→Server toggle set — so the env var alone left consoles dialing port+1/+2. They now route through the proxy port without needing the UI toggle, which should also clear the :5001 handshakes (the browser was picking wss against a plain-ws console listener). I'll add a PEGAPROX_URL + reverse-proxy-env note to the docs.

opencollective.com/pegaprox if PegaProx is earning its keep — and thanks, genuinely one of the easier bugs to chase thanks to the detail you put in.

— Marcus