cc-chaos-proxy

WebSocket-aware network chaos proxy for Creditcoin / Substrate JSON-RPC. Built on Rama — a Toxiproxy-style alternative that relays WebSocket frames (not raw TCP), so subxt subscriptions stay alive.

Why not Toxiproxy?

Toxiproxy is a TCP relay. Substrate/subxt keeps multiple long-lived JSON-RPC subscriptions over WebSocket; TCP-level proxying causes SubscriptionDropped within hundreds of milliseconds even with no toxics applied.

This proxy:

• Completes proper WS handshakes on both sides
• Relays ping/pong/close immediately (required by jsonrpsee/subxt)
• Applies toxics only on application (text/binary) frames
• Exposes a Toxiproxy-compatible HTTP control API on :8474 (same as Toxiproxy — no binary protocol)

Quick start

cargo build --release

# Terminal 1: cc3-node + anvil running on :9944 and :8545
./target/release/cc-chaos-proxy

# Terminal 2: point attestor at proxies
#   cc3-url: ws://127.0.0.1:19944
#   eth-url: ws://127.0.0.1:18545

Default listeners:

Proxy	Listen	Upstream
cc3	127.0.0.1:19944	ws://127.0.0.1:9944/
eth	127.0.0.1:18545	ws://127.0.0.1:8545/

Control API: http://127.0.0.1:8474

Toxics

Type	Description
latency	Delay application frames (optional jitter)
timeout	Close connections after N ms (rolled once per connection)
bandwidth	Throttle by bytes/sec (sleep proportional to frame size)
limit_data	Hang forever once byte cap is exceeded
reset	Drop connection on next matching frame
rate_limit	Infura-style quota: allow N requests, then 429 / disconnect

All frame toxics support stream: upstream (client→node), downstream (node→client), or both.

Optional toxicity (0.0–1.0) controls how often a toxic applies per frame (Toxiproxy-compatible). Defaults to 1.0.

Infura-style rate limiting

Simulates provider API key quotas — works for a while, then busts:

• First N upstream JSON-RPC frames pass through
• Frame N+1 on an existing WebSocket → connection dropped
• New WebSocket handshakes while over quota → HTTP 429 with JSON-RPC error body
• Quota is global per proxy (shared across connections)
• window_ms: 0 = quota never resets until toxic is removed; set window_ms for a sliding reset

# Allow 20 requests, then throttle (Infura-like)
cargo run --bin cc-chaos-cli -- toxic-add eth \
  --name infura-cap --kind rate_limit --max-requests 20

# Same, quota resets every 60s
cargo run --bin cc-chaos-cli -- toxic-add eth \
  --name infura-cap --kind rate_limit --max-requests 20 --window-ms 60000

Control API

# List proxies (includes `running` and `toxics`)
curl -s http://127.0.0.1:8474/proxies | jq

# Version
curl -s http://127.0.0.1:8474/version | jq

# Toggle cc3 proxy off (drops application frames, rejects new connections)
curl -X POST http://127.0.0.1:8474/proxies/cc3

# Create a dynamic proxy (starts listener immediately)
curl -X POST http://127.0.0.1:8474/proxies \
  -H 'content-type: application/json' \
  -d '{"name":"custom","listen":"127.0.0.1:29999","upstream":"ws://127.0.0.1:9944/"}'

# Delete proxy (stops listener)
curl -X DELETE http://127.0.0.1:8474/proxies/custom

# Add 30s downstream latency
curl -X POST http://127.0.0.1:8474/proxies/cc3/toxics \
  -H 'content-type: application/json' \
  -d '{"name":"latency","type":"latency","latency_ms":30000,"stream":"downstream"}'

# Force-close connections after 8s
curl -X POST http://127.0.0.1:8474/proxies/cc3/toxics \
  -H 'content-type: application/json' \
  -d '{"name":"timeout","type":"timeout","timeout_ms":8000,"stream":"upstream"}'

# Throttle bandwidth
curl -X POST http://127.0.0.1:8474/proxies/eth/toxics \
  -H 'content-type: application/json' \
  -d '{"name":"slow","type":"bandwidth","rate_bytes_per_sec":32768,"stream":"downstream"}'

# Rate limit (Infura-style)
curl -X POST http://127.0.0.1:8474/proxies/eth/toxics \
  -H 'content-type: application/json' \
  -d '{"name":"cap","type":"rate_limit","max_requests":20,"window_ms":0,"stream":"upstream"}'

# Remove toxic
curl -X DELETE http://127.0.0.1:8474/proxies/cc3/toxics/latency

# Clear all toxics (optionally per proxy)
curl -X POST 'http://127.0.0.1:8474/reset'
curl -X POST 'http://127.0.0.1:8474/reset?proxy=eth'

CLI

cc-chaos-cli wraps the HTTP control API (like toxiproxy-cli):

cargo run --bin cc-chaos-cli -- list
cargo run --bin cc-chaos-cli -- toggle cc3

# Latency
cargo run --bin cc-chaos-cli -- toxic-add cc3 --name slow --latency 30000 --stream downstream

# Timeout
cargo run --bin cc-chaos-cli -- toxic-add cc3 --name kill --kind timeout --timeout 8000

# Bandwidth
cargo run --bin cc-chaos-cli -- toxic-add eth --name slow-net --kind bandwidth --rate 32768

# Limit data
cargo run --bin cc-chaos-cli -- toxic-add cc3 --name cap --kind limit_data --bytes 1048576

# Reset connection
cargo run --bin cc-chaos-cli -- toxic-add cc3 --name drop --kind reset

# Rate limit (aliases: 429, infura)
cargo run --bin cc-chaos-cli -- toxic-add eth --name infura --kind rate_limit --max-requests 20

# Remove toxic / clear all
cargo run --bin cc-chaos-cli -- toxic-remove cc3 --name slow
cargo run --bin cc-chaos-cli -- reset
cargo run --bin cc-chaos-cli -- reset --proxy eth

Environment

Variable	Default	Description
CHAOS_CONTROL	127.0.0.1:8474	Control API bind
CHAOS_API	http://127.0.0.1:8474	Control API URL for CLI
CC3_PROXY_LISTEN	127.0.0.1:19944	CC3 proxy listen
CC3_UPSTREAM	ws://127.0.0.1:9944/	CC3 upstream
ETH_PROXY_LISTEN	127.0.0.1:18545	ETH proxy listen
ETH_UPSTREAM	ws://127.0.0.1:8545/	ETH upstream
CHAOS_NO_CC3	—	Disable default cc3 proxy
CHAOS_NO_ETH	—	Disable default eth proxy
RUST_LOG	info	Log level

Integration with test-reconnect.sh

Point the attestor at proxy ports instead of direct chain ports:

# In attestor_zombienet / zombienet config:
--cc3-url ws://127.0.0.1:19944
--eth-url ws://127.0.0.1:18545

Start cc-chaos-proxy before the harness, then drive toxics via the control API or cc-chaos-cli.

Limitations

• WebSocket only — plain HTTP JSON-RPC gets 426 Upgrade Required
• Not a full Toxiproxy reimplementation (no slicer, slow_close, etc.)
• rate_limit simulates provider throttling, not a crashed RPC node

License

MIT OR Apache-2.0