Nginx HTTP Tunnel Connect Module

Note

Throughput can be limited due to configurations. Try tune TCP kernel buffers, use BBR congestion control, set larger http3_stream_buffer_size, client_body_buffer_size and client_max_body_size values can boost performance greatly.

This provides a subset features of HTTP CONNECT over HTTP/1.1, HTTP2, and HTTP/3.

Table of contents

• Tested nginx versions
• Why implement this?
• How to build?
• Example conf file
• Detailed config directives
• Performance Comparison

Intro

Naive Proxy used padding scheme is implemented in here,
works on h2 and h3 only. HTTP/1.1 is not targeted and it is
recommended to use nginx/1.31.* version.

- [x] HTTP/1.1 CONNECT 
- [x] HTTP/2 CONNECT
- [x] HTTP/3, QUIC CONNECT
- [x] Non CONNECT method can co-exist with other HTTP methods
- [x] Proxy authentication
- [x] Map based ACL
- [x] Naive Style Padding Scheme
- [x] Connect-udp with capsule protocol, without QUIC DATAGRAM

Module build relies on some nginx patches.
See patches, apply them to nginx source code,

The patches do following:

1. Let nginx upstream module only open TCP/UDP connection
without any input, using `ignore_input` flag.

2. Let h2/h3 pseudo header parsing pass nginx core.

Read scripts/ to find out more information about
map based acl.

Features that are in WIP:

Refactors that are in WIP:

Any PR or Issues are welcomed. If you are using Agent coding, please review
before submitting it.

Tested Nginx Versions

• Nginx 1.29/28 and some privious version should work, but not recommended to use due to some CVE in nginx.

• Nginx 1.30.* is tested to work.

• Nginx 1.31.0 - 1.31.2 is tested to work, apply header_parsing.patch and upstream-1.31.patch

How to build

First fetch source code

mkdir -p build
cd build

git clone https://github.com/nginx/nginx.git
git switch --detach {version} # Checkout on specific version

git clone https://github.com/ZihaoFU245/ngx_http_tunnel_module.git
git switch --detach {version} # Checkout on specific version

cd nginx

Apply patches in patches/ folder, for nginx versions below 1.30.* and including 1.30.. Apply header_parsing.patch and upstream.patch For 1.31. and above inclusion, apply upstream-1.31.patch instead.

On nginx 1.31.0 and newer, nginx core handles proxy authentication for CONNECT requests. This module therefore does not build its auth.c auth path on those versions.

git apply ../ngx_http_tunnel_module/patches/header_parsing.patch

git apply ../ngx_http_tunnel_module/patches/upstream.patch
# OR
git apply ../ngx_http_tunnel_module/patches/upstream-1.31.patch
# BASED ON YOUR NGINX VERSION

Check nginx build dependency before continue. Below is an example build configuration.

./auto/configure \
	--prefix=/usr/local/share/nginx \
	--sbin-path=/usr/local/sbin/nginx \
	--conf-path=/etc/nginx/nginx.conf \
	--pid-path=/run/nginx.pid \
	--lock-path=/var/lock/nginx.lock \
	--error-log-path=/var/log/nginx/error.log \
	--http-log-path=/var/log/nginx/access.log \
	--http-client-body-temp-path=/var/lib/nginx/body \
	--http-fastcgi-temp-path=/var/lib/nginx/fastcgi \
	--http-proxy-temp-path=/var/lib/nginx/proxy \
	--http-scgi-temp-path=/var/lib/nginx/scgi \
	--http-uwsgi-temp-path=/var/lib/nginx/uwsgi \
	--with-compat \
	--with-threads \
	--with-file-aio \
	--with-pcre-jit \
	--with-http_ssl_module \
	--with-http_v2_module \
	--with-http_v3_module \
	--with-http_realip_module \
	--with-http_auth_request_module \
	--with-http_mp4_module \
	--with-http_gunzip_module \
	--with-http_gzip_static_module \
	--with-cc-opt='-g -O2 -fPIC -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=3' \
	--with-ld-opt='-Wl,-z,relro -Wl,-z,now -fPIC' \	
	--add-dynamic-module=../ngx_http_tunnel_module
	# use --add-module to compile it in nginx binary

Finally, run make -j$(nproc).

Implementation tricks

1. It is chosen to build as a nginx module as it can reuse nginx core functionalities, including but not limited to, mature h2/h3 implementation, multiplexing, limit conn, upstream module.

2. This module relies on nginx upstream and http_v2, see header file.

Example Configuration file

Tip

Check examples/ for minimum configuration files.

Important

Strongly recommended to use headers-more module in nginx to clear Proxy-Authenticate header in 405 response, specifically for nginx v1.31.0 and above. Otherwise, any MITM with active probing capability may discover the header, and classify nginx as a forward proxy.

Below is a detailed config flags illustration.

load_module ngx_http_tunnel_module.so;  # If build as an dynamic module

user www-data;
worker_processes auto;
worker_cpu_affinity auto;

events {
	worker_connections 1024;
}

http {
	tcp_nopush on;
	tcp_nodelay on;
	server_tokens off;

    # If you are using with a file server
    sendfile on;
	include mime.types;

	ssl_protocols TLSv1.2 TLSv1.3;
	ssl_prefer_server_ciphers off;

	limit_conn_zone $binary_remote_addr zone=addr:1m;

	# ---------------------------------------------
	# A map is used in ACL for O(1) lookup,
	# $connect_target_host variable provides the raw
	# authority for classic CONNECT and the parsed
	# target authority for CONNECT-UDP. It is your job
	# to regex match these authorities. Test the ACL
	# before production, some clients put authority
	# as raw ip, raw host, or even host:port.
	# This can be tricky, be careful!
	#
	# Allowed mapping values:
	# 0/1: deny/allow
	# 2/3: deny/allow + logging
	# 
	# Example:
	# Blocking a single hostname:
	# ~^example\.com(:[0-9]+)?$ 
	# --------------------------------------------
	map $connect_target_host $is_granted {
		default 1;								# default allow

		~(^|:)fr\.a2dfp\.net(:|$)       0;		# deny
		~(^|:)static\.a-ads\.com(:|$)   2;		# deny + log
	}

	# Use Proxy Auth but on CONNECT only
	map $request_method $connect_auth_realm {
        default  off;
        CONNECT  "proxy";
    }

	server {
		listen 0.0.0.0:443 ssl;
		listen 0.0.0.0:443 quic reuseport;

		server_name example.com;

		http2 on;
        http3 on;

        http3_stream_buffer_size 256k;
        quic_gso on;

		add_header Alt-Svc 'h3=":443"; ma=86400';

		limit_conn addr 100;
		limit_conn_status 429;

		client_body_buffer_size 256k;
		client_max_body_size 16M;

		# Resolver must be set, otherwise will result in 502
		resolver 1.1.1.1 8.8.8.8;

		ssl_certificate  fullchain.pem;
		ssl_certificate_key privkey.pem;

		tunnel_connect;                        	# Enable tunnel module
		tunnel_connect_buffer_size 64k;         # Buffer size for tunnel relay 
		# tunnel_connect_tcp_nodelay on;        # TCP_NODELAY for CONNECT only

		# NGINX 1.30 AND OLDER ONLY
		# tunnel_connect_proxy_auth_user_file /path/to/.htaccess;
		# tunnel_connect_probe_resistance off;
		# tunnel_connect_probe_resistance_allow_methods "";

		# NGINX 1.31.0 AND NEWER
		auth_basic $connect_auth_realm;
		auth_basic_user_file /path/to/.htaccess;
		error_page 407 =405 @probe_resistance;

		location @probe_resistance {
            # On Auth Failure, this header is still present, use
            # headers more module to remove this header entirely.
            more_clear_headers 'Proxy-Authenticate';
		    return 405;
		}

        tunnel_connect_padding off;             # Opt in padding scheme for h2/h3
        tunnel_connect_upstream_timeout 60s;
		tunnel_connect_idle_timeout 120s;

		# 0: deny, 1: allow, 2: deny + log, 3: allow + log.
		# $connect_target_host is the effective target authority.
		tunnel_connect_acl_eval_on $is_granted;

		tunnel_connect_udp off; 						# Enable connect udp with capsule protocol
		tunnel_connect_udp_path $request_uri; 			# Path variable for parsing masque path

		location / {
		    proxy_pass http://localhost:8090;

			# To avoid The Discriminative Power of Cross-layer 
			# RTTs in Fingerprinting Proxy Traffic
			# It is recommended to either proxy_pass to a server
			# running on the same nginx instance or
			# run a file server here directly.
		}
    }

	server {
		listen 127.0.0.1:8090;

		location / {
			return 200 "Hello there!\n";
		}
	}
}

Config Directives

1. tunnel_connect_pass: Enable tunnel module, allow HTTP connect.

2. tunnel_connect_buffer_size: Size, a symmetric buffer size for tunnel module, for upstream buffer and client buffer. Default to 64k, with minimum 1k.

3. tunnel_connect_proxy_auth_user_file: path to .htaccess file. This will only support nginx version 1.30 and below.

4. tunnel_connect_probe_resistance: Default to off. When auth failed, it will return 405. Only nginx 1.30 and below are supported.

5. tunnel_connect_probe_resistance_allow_methods: Default to an empty string. And will not send "Allow" header when it is empty, used with tunnel_connect_probe_resistance.

6. tunnel_connect_padding: Default to off. Enable padding scheme defined by naiveproxy.

7. tunnel_connect_upstream_timeout: Time, default to 60s. On timeout, internal server error will be send. This happens when upstream server has no response.

8. tunnel_connect_idle_timeout: Time, default to 120s. On timeout, tunnel will be closed.

9. tunnel_connect_tcp_nodelay: Default to off. Enable TCP_NODELAY for CONNECT tunnels without enabling nginx core tcp_nodelay for normal HTTP requests. If nginx core tcp_nodelay is on, CONNECT tunnels already use TCP_NODELAY.

10. tunnel_connect_acl_eval_on: Complex value. Accept, integers from 0-3. 0/1 means access deny/allow. 2/3 means access deny/allow with logging.

11. tunnel_connect_udp: Default to off. With connect udp over MASQUE capsule protocol. Client send header must present capsule-protocol = ?1, otherwise 400 rejected.

12. tunnel_connect_udp_path: Complex value, MASQUE encode target host and port in path. Default to $request_uri.

Performance Comparison

This compares memory usage (RSS) between Caddy forward proxy and this nginx tunnel module.

The test is under 5 rounds of 100 tunnels, each transfer data at 1Mbps, total 100 Mbps for 30 seconds with 5 seconds rest interval.

Nginx is more memory efficient and memory growth is more deterministic. buffer size is used at 64k, the default value. Valgrind shows no definitely leak, all memory pools are properly deleted on CONNECT finalize.

benchmark tools and more details