rf-bypass

Ansible automation for a RU-resident forward-proxy layer that tunnels selected egress traffic through WireGuard + Cloak to out-of-country exit nodes, with automatic failover and optional multipath load balancing.

What this project does

RU-side hosts (ru_nodes) run Envoy as a forward HTTP/HTTPS proxy. Envoy-owned traffic is marked fwmark 0x1 and policy-routed via table 100 through one or more WireGuard tunnels, each wrapped in Cloak to obfuscate the handshake as TLS. Tunnels terminate on exit nodes (exit_nodes) hosted abroad. Clients reach the proxy either directly on 3128/tcp or via a PAC/WPAD-served proxy chain.

A separate playbook (setup-gre-forward.yml) lets non-HTTP traffic ingress over GRE tunnels on the RU node and share the same WG egress path (fwmark 0x2, same table 100, SNAT to wg_ip).

Observability: each RU node exposes a read-only Envoy admin listener on 0.0.0.0:9902 (configurable via envoy_admin_lan_port). It proxies the safe admin paths (/stats, /clusters, /listeners, /server_info, /ready, /runtime) back to the local-only admin on 127.0.0.1:9901, gated by an RBAC filter that allows only RFC1918 source IPs. Destructive endpoints (/quitquitquit, /runtime_modify, /drain_listeners, …) are not routed and return 404. Prometheus scrape URL: http://<ru-node>:9902/stats/prometheus.

Core design

• Single source of truth for topology: group_vars/all/wg.yml. Each wg_connections entry pairs a ru_node with an exit_node, assigns a ru_interface (wg0, wg1, …), a failover_priority, and an optional weight for multipath.
• Three route layers in table 100, lowest metric wins:
  1. Watchdog routes — priority × 10 metric, added/removed based on liveness pings.
  2. Fallback — metric 1000, always present, points at the highest-priority tunnel.
  3. /32 host routes for exit-node endpoints, pinned to their physical NIC.
• Zero-downtime transitions: ip route replace is atomic; wg-failover-watch.sh layers metric routes above the metric-1000 fallback so a failing tunnel never leaves the host without a route.
• Two routing modes — per-host, toggled by wg_load_balance:
  • LB mode (default, wg_load_balance=true): watchdog installs a single weighted multipath route at metric 5 with one nexthop per healthy tunnel. Kernel distributes flows via fib_multipath_hash_policy (default 1 = L4 hash; set 0 if an upstream peer breaks with per-flow distribution).
  • Priority mode (wg_load_balance=false): watchdog installs per-tunnel routes at metric priority × 10. Kernel picks the lowest-metric healthy tunnel; others stand by. Classic active-standby. Both modes sit above the always-present metric-1000 fallback, so the host is never without a route.
• Domain allowlist at the proxy: Envoy's Lua filter parses resources/resources.txt (the same list PAC consumes) plus optional resources/proxy-only.txt. Unlisted domains get 403.
• Cloak obfuscation: WG client → ck-client (127.0.0.1) → ck-server on exit:443 → WG server. Exit nodes expose only port 443/TCP publicly.

See group_vars/all/wg.yml for the tunnel map and the header comments there for how to add a new ru_node or a standby tunnel.

A connection can be marked monitor_only: true — it is built and pinged like a normal tunnel, but excluded from all routes. See Adding a monitor-only tunnel below.

⚠️ Security notice — read before using this codebase

The original (internal) repository commits plaintext WireGuard and Cloak private keys under wg-keys/ and wg-cloak-keys/. This is an explicit threat-model choice for a closed internal git server, where ops convenience (zero-touch redeploy from any operator's clone) is valued higher than compartmentalisation of key material.

In this snapshot the key directories have been removed. If you intend to reuse this codebase, decide before you do anything else:

• Never commit private keys to a public or shared repository. If you adopt this layout as-is, restrict the repo to operators who already have prod access — i.e. treat the repo itself as a secret.
• Better: refactor setup-wg-cloak.yml to store keys outside the git tree (ansible-vault, an external secret store, generated-on-host with the public key fetched back). This is left as an exercise — out of scope of the migration that produced this snapshot.

About key regeneration. setup-wg-cloak.yml's local key-gen tasks create files only if they're missing. So:

• Fresh install with empty wg-keys/ and wg-cloak-keys/ → new keys are generated locally, then deployed to both sides — works.
• Existing prod cluster, run with empty key dirs → new keys generated, then pushed to one side, but the peer side already has the old configuration under its name, so the handshake breaks until both sides are re-keyed simultaneously. Do not delete the production key dirs. Only the snapshot in this repo has them removed.

Inventory and topology are placeholders

inventory.ini contains commented-out example hosts only. Fill in real hostnames, IPs and wg_ips before running anything. group_vars/all/wg.yml and group_vars/all/gre.yml carry the topology that was active at snapshot time — review and adjust them.

Runbook

One-time setup

SSH pubkey auth is required on every host — inventory no longer carries passwords.

# Exit nodes (root). If not yet key-based:
ssh-copy-id -o PreferredAuthentications=password root@<exit-node-ip>

# RU nodes: your user must have sudo + pubkey installed
ssh-copy-id <user>@<ru-node-ip>

Environment variables

RU nodes pull their SSH user and sudo password from the environment (group_vars/ru_nodes.yml uses lookup('env', ...)). Exit nodes are pinned to root in the inventory and need no env vars.

export RU_SSH_USER=your.username
export RU_BECOME_PASS='...'

If either is unset, the RU connection/sudo will fail fast with an empty value — that's intentional.

Playbooks — when to run which

Playbook	When to run
setup-wg-cloak.yml	Initial tunnel bring-up, or after topology changes in group_vars/all/wg.yml. Creates WG + Cloak on both ends. Keys are generated once and never overwritten.
setup-envoy-wg.yml	Main RU-node deploy: installs Envoy, configures fwmark/policy routing, installs the watchdog, applies the domain allowlist.
update-envoy-config.yml	Only the Envoy config + reload — run after edits to resources/resources.txt or resources/proxy-only.txt. Skips install, routing, watchdog.
setup-pac-wpad.yml	Deploy PAC/WPAD hosting so clients auto-discover the proxy chain.
setup-gre-forward.yml	GRE ingress for non-HTTP traffic. Independent of Envoy; reuses table 100 and the WG egress.

Typical full deploy on a fresh ru_node:

ansible-playbook -i inventory.ini setup-wg-cloak.yml
ansible-playbook -i inventory.ini setup-envoy-wg.yml
ansible-playbook -i inventory.ini setup-pac-wpad.yml        # run this to deploy proxy autoconfig using PAC/WPAD conventions. Require system-level PAC config with automatic or manual URL.
ansible-playbook -i inventory.ini setup-gre-forward.yml     # optional
ansible-playbook -i inventory.ini setup-monitoring.yml      # per-tunnel external-IP probes

Tag-scoped runs are supported on setup-envoy-wg.yml:

ansible-playbook -i inventory.ini setup-envoy-wg.yml --tags envoy
ansible-playbook -i inventory.ini setup-envoy-wg.yml --tags verify

Updating the domain allowlist

Edit resources/resources.txt (shared with PAC) and/or resources/proxy-only.txt (proxy-only superset), then:

ansible-playbook -i inventory.ini update-envoy-config.yml

This regenerates /etc/envoy/envoy.yaml and hot-reloads Envoy. It does not touch installation, routing, or the watchdog — fastest feedback loop for allowlist churn. Also re-run setup-pac-wpad.yml if PAC-visible entries changed and you want clients updated too.

Adding a new exit node

1. Provision the exit_node (fresh Debian/Ubuntu, root SSH with your pubkey in /root/.ssh/authorized_keys, UDP egress allowed, port 443/TCP inbound open).
2. Add it to [exit_nodes] in inventory.ini:

   newexit  ansible_host=<public-ip>  wg_ip=10.230.0.N

   Pick a free wg_ip in 10.230.0.0/28.
3. Append wg_connections entries in group_vars/all/wg.yml — one per ru_node that should tunnel through it. Pick the next free ru_interface (wgN) on each ru_node and assign a failover_priority (lower = preferred in priority mode; any non-overlapping integer is fine in LB mode):

   - ru_node: ru_node1
     exit_node: newexit
     ru_interface: wg3
     failover_priority: 3
     weight: 1

4. Deploy both ends, then refresh RU-node routing/watchdog:

   ansible-playbook -i inventory.ini setup-wg-cloak.yml
   ansible-playbook -i inventory.ini setup-envoy-wg.yml

Adding a new RU node

1. Provision the ru_node (Ubuntu, sudo user with pubkey auth; sudo password exported as $RU_BECOME_PASS).
2. Add it to [ru_nodes] in inventory.ini with a unique wg_ip:

   ru_node3  ansible_host=<ru-node-3-lan-ip>  wg_ip=10.230.0.12

3. Append one wg_connections entry per exit tunnel it should use (group_vars/all/wg.yml).
4. Optionally add wg_load_balance=false as a host_var to put this node in priority mode instead of LB.
5. Run the full deploy sequence, targeting just the new host to save time:

   ansible-playbook -i inventory.ini setup-wg-cloak.yml     --limit ru_node3,exit_nodes
   ansible-playbook -i inventory.ini setup-envoy-wg.yml     --limit ru_node3
   ansible-playbook -i inventory.ini setup-pac-wpad.yml     --limit ru_node3   # if serving PAC locally
   ansible-playbook -i inventory.ini setup-gre-forward.yml  --limit ru_node3   # if GRE ingress needed

   (Include exit_nodes in the first --limit so the peer side of the new tunnel is configured.)

Adding a monitor-only tunnel

A monitor-only tunnel is brought up exactly like a normal one (WG + Cloak on both ends, watchdog pings the peer every 10 s, UP/DOWN events appear in journalctl -t wg-failover), but it is never added to any kernel route — no multipath nexthop in LB mode, no metric route in priority mode, never picked as the metric-1000 fallback. Use it to verify that a standby exit is reachable before you decide to route real traffic through it.

1. Provision the exit node and append a wg_connections entry as for a normal tunnel, then add monitor_only: true:

   - ru_node: ru_node1
     exit_node: exit_node5
     ru_interface: wg3
     failover_priority: 9         # required by the sort, otherwise ignored
     monitor_only: true

2. Run setup-wg-cloak.yml then setup-envoy-wg.yml as usual.
3. Confirm pings: sudo journalctl -u wg-failover-watch -t wg-failover -f.

To promote a monitor-only tunnel to a routed one, remove the monitor_only key (do not set it to false — only presence is checked) and re-run setup-envoy-wg.yml. The metric-1000 fallback's _primary_conn selection skips monitor-only entries, so at least one routed connection must remain per ru_node.

The watchdog pings the peer's wg_ip over the tunnel interface — that proves WG handshake + peer is up, not that the exit's own internet egress works. If you need the latter, add a second probe through the tunnel to e.g. 1.1.1.1.

Editing an existing connection

• Change weight or failover_priority: edit group_vars/all/wg.yml, then setup-envoy-wg.yml is enough — it re-renders the policy script and the watchdog picks up the new metrics/weights.
• Retire a connection: remove its entry from wg_connections, re-run setup-wg-cloak.yml (to tear down WG/Cloak units on both peers) and setup-envoy-wg.yml (to refresh routes and watchdog). The stale interface is cleaned up idempotently.
• Switch a ru_node between LB and priority modes: set/unset wg_load_balance in inventory, then setup-envoy-wg.yml — the watchdog and policy script are regenerated for the chosen mode.

Routing modes in detail

Aspect	LB mode (default)	Priority mode
Toggle	wg_load_balance=true (or unset)	wg_load_balance=false per host
Active tunnels	All healthy, simultaneously	One (lowest priority × 10 metric)
Watchdog route	Single multipath route at metric 5 with one nexthop per healthy tunnel	Per-tunnel routes at metric priority × 10
Role of weight	Multipath weight — nexthop … weight N. Flows are distributed ~proportionally to weight. Default 1 = equal share.	Ignored.
Role of failover_priority	Tie-breaker only — feeds the metric-1000 fallback (lowest priority wins).	Primary ordering — lowest wins; others stand by.
Failover behaviour	Failing nexthop is removed from the multipath route; healthy ones keep carrying traffic. No flow re-hash on survivors.	Failing tunnel's metric route is removed; next-lower metric takes over.
Hash policy	net.ipv4.fib_multipath_hash_policy (1 = L4, 0 = L3). Set via envoy_multipath_hash_policy.	N/A
When to prefer	Multiple exits with similar latency; aggregate throughput matters; you're OK with per-flow path churn on topology change.	One strongly preferred exit (geo, cost, trust); others are hot-standby; sticky single-path desirable.

Weights — how to pick them: weights are integers, interpreted as relative shares. [1, 1] → 50/50. [3, 1] → ~75/25 on the first tunnel. There's no bandwidth awareness; you're telling the kernel what ratio to aim for, not a bitrate. Start at 1 for all, and only change if you have a concrete reason (one exit has twice the egress pipe, or you want to shift load away from an expensive provider).

Priorities — how to pick them: pick non-overlapping integers across a given ru_node's connections. 1 = most preferred. In LB mode priorities matter only if all tunnels are down — the metric-1000 fallback uses the lowest-priority tunnel. In priority mode they're the whole story.

Repository layout

group_vars/
  all/               topology (wg.yml), GRE map (gre.yml), proxy CIDRs
  ru_nodes.yml       env-var lookups for RU user + become password
inventory.ini        hosts, minimal non-secret vars
resources/           domain lists feeding PAC and Envoy's Lua allowlist
templates/           Envoy, PAC, WG configs
wg-keys/             WireGuard keypairs (generated on first run)
wg-cloak-keys/       Cloak UID + keypairs (generated on first run)
setup-*.yml          playbooks (see runbook table)

Security notes

• No credentials in inventory. SSH pubkey auth only; RU sudo password comes from $RU_BECOME_PASS at runtime.
• WG/Cloak private keys live under wg-keys/ and wg-cloak-keys/ in plaintext. They are per-host, generated locally, and carry no value outside the tunnel mesh — acceptable for this repo's threat model.
• Envoy's allowlist is a coarse guard, not an authN boundary. The proxy is assumed to be reached only from trusted internal networks.