Skip to content

PRODUCTION.md

Latest commit

 

History

History
692 lines (537 loc) · 21.4 KB

PRODUCTION.md

File metadata and controls

692 lines (537 loc) · 21.4 KB
Raw

HiveRelay Production Deployment Guide

Prerequisites

  • Node.js >= 20
  • A server with a public IP (VPS, dedicated, or cloud instance)
  • UDP port access (HyperDHT uses UDP for peer discovery)
  • At least 1 GB RAM, 10 GB disk (more disk = more apps you can seed)

Option 1: Bare Metal / VPS

Install

From npm (stable latest is 0.24.4; the release-candidate lane publishes to the next dist-tag):

npm install -g p2p-hiverelay        # stable
npm install -g p2p-hiverelay@next   # release-candidate lane

Or from the signed source tag (see the stable guide for the current tag):

git clone https://github.com/bigdestiny2/P2P-Hiverelay.git /opt/hiverelay
cd /opt/hiverelay
npm ci --omit=dev

Create system user

sudo useradd -r -s /usr/sbin/nologin -d /var/lib/hiverelay hiverelay
sudo mkdir -p /var/lib/hiverelay
sudo chown hiverelay:hiverelay /var/lib/hiverelay

Configure

# Create config (optional — defaults work fine)
sudo -u hiverelay mkdir -p /var/lib/hiverelay
cat <<EOF | sudo tee /home/hiverelay/.hiverelay/config.json
{
  "storage": "/var/lib/hiverelay",
  "maxStorageBytes": 53687091200,
  "regions": ["NA"],
  "apiPort": 9100,
  "enableRelay": true,
  "enableSeeding": true,
  "enableMetrics": true,
  "enableAPI": true
}
EOF

Signed directory (opt-in registry)

SignedDirectory (hiverelay-signed-directory) is an always-on, openly-writable registry of signed records keyed by author pubkey. First consumer: marketplace offer discovery (sellers publish a signed Offer; buyers list + verify). Generic enough for any "signed record set by author" use case (job boards, file-share directories, signed gossip feeds).

Trust model — identical to a topic-swarm announce: the relay can omit, reorder, or refuse records; it cannot forge them. All records are signed over SHA256(authorPubkey || timestamp_LE || payload) using Ed25519. Buyers MUST verify signatures + timestamps client-side; the relay never inspects payload semantics, by design.

OFF by default. Enable per node by adding to config.json:

{
  "signedDirectory": {
    "enabled": true,
    "maxEntryBytes": 8192,
    "ttlSeconds": 86400,
    "maxEntriesPerAuthor": 1,
    "publishRatePerMinute": 5,
    "maxTotalEntries": 10000,
    "clockSkewToleranceSeconds": 60
  }
}

The defaults are the storage-policy commitments documented in issue #33:

  • maxEntryBytes (8 KB) — per-record cap
  • ttlSeconds (24h) — TTL eviction; tradeoff between always-on discovery and storage churn
  • maxEntriesPerAuthor (1) — newest-timestamp-wins; one live record per authorPubkey
  • publishRatePerMinute (5, per-peer) — symmetric with circuit-relay's reserve rate limit
  • maxTotalEntries (10000) — bounds the in-memory store for storage-constrained operators
  • clockSkewToleranceSeconds (60) — critical: rejects entries timestamped more than 60s in the future. Without this, a malicious author with timestamp = 2099-01-01 permanently outranks legitimate updates from the same pubkey. Bidirectional bound.

Replication is automatic between enabled relays. A publish to relay A is broadcast via NOTIFY to all peer-relay channels A has open; peer relays validate and store (no re-broadcast — single-hop). No configuration needed beyond signedDirectory.enabled on each participating relay.

Storage is in-memory only in v1. Short TTL (24h default) + cross- relay replication means individual-relay restarts lose state but the network as a whole stays warm. Hyperbee-backed persistence is tracked as a follow-up.

Operator subsidy (Phase 1 — accrual only, opt-in)

The relay-local half of the bootstrap subsidy from docs/OPERATOR-INCENTIVES-Y1.md (Prong 2). When enabled, the node accrues a capped sats estimate for blind-peer work (default 500 sats/day ≈ $180/yr at $100k/BTC) and records per-epoch evidence (uptime, connections, seeded/anchored counts). No money moves through the relay — there is no wallet in the app. The operator sets a payout destination they control (lightning address, BOLT12 offer, or on-chain address) from the dashboard, and the subsidy coordinator independently verifies the relay's Ed25519-signed claim (GET /api/subsidy/claim) before paying. The relay's own numbers are an estimate; the coordinator's verification (Operator Score gates, sybil checks, held schedule) decides the actual payout.

OFF by default. Enable per node via config.json:

{
  "subsidy": {
    "enabled": true,
    "rateSatsPerDay": 500,
    "epochMs": 600000,
    "payoutDestination": null
  }
}

or via env for containerized installs: HIVERELAY_SUBSIDY_ENABLED=1 (optionally HIVERELAY_SUBSIDY_DESTINATION=you@your-ln-provider.com). State persists to <storage>/subsidy.json with atomic writes. Surfaces: dashboard earnings cards, /api/subsidy (status), /status (subsidy block).

/status exposes a signedDirectory block when enabled:

{
  "signedDirectory": {
    "enabled": true,
    "entries": 47,
    "maxTotalEntries": 10000,
    "attachedChannels": 11,
    "ttlSeconds": 86400,
    "totalPublished": 312,
    "totalRejected": 4,
    "totalReplicated": 89,
    "totalEvicted": 0,
    "rejectedReasons": { "BAD_SIGNATURE": 2, "RATE_LIMITED": 2 }
  }
}

Verify with npx brittle test/unit/signed-directory.test.js.

Forward relay (opt-in transport)

ForwardRelay (hiverelay-forward) lets this node relay app connections for peers behind NAT / UDP-blocking, and is the building block for onion routing (see docs/forward-relay.md). It is OFF by default — enable it per node by adding to config.json:

{
  "forwardRelay": {
    "enabled": true,
    "maxForwardsPerPeer": 5,
    "maxForwardBytes": 67108864
  }
}

It only ever reaches DHT pubkeys (never an arbitrary IP — not an internet open proxy), is bounded by per-peer concurrency + per-forward (64 MB) + per-frame (64 KB) caps, and honours the node's SwarmFirewall. Enable it deliberately: it lets peers reach other DHT peers through your node. Verify with npx brittle test/integration/forward-relay.test.js.

Storage-proof service (opt-in trustless verification)

The storage-proof builtin service lets a user trustlessly verify your relay actually holds their app — not just claims to in the catalog. On request it produces a signed Hypercore Merkle proof for a randomly-sampled block of a seeded drive's metadata core; the client verifies each proof against the drive key alone (forged content is rejected) plus your relay's swarm-identity signature (attribution + a fresh nonce, so proofs can't be replayed). It backs client.proveSeeded(driveKey, { relay, samples }).

It is OFF by default. Enable it per node:

  • Node runtime — add storage-proof to config.plugins (or the Services tab / services.json):

    {
  "plugins": ["storage-proof"]
}

  • Bare / appliance runtime — add storage-proof to config.services, or set the env var:

    HIVERELAY_STORAGE_PROOF=1

Once enabled it is reachable over the existing service RPC by anyone (route policy storage-proof.prove = public) — that is intentional: trustless verification is only useful if any user can run it. The safety properties that make this safe to expose:

  • Privacy gate — blind / privacy-redacted drives return NOT_SEEDED, the same error as a key the relay doesn't hold. A signed proof is cryptographic, relay-attributable evidence of possession, so serving one for a blind drive would defeat the catalog's deliberate redaction. The gate reuses the catalog's own AppRegistry._shouldRedactEntry predicate, so prove() can never become a possession oracle.
  • Rate limits — a sybil-resistant global proof-work token bucket caps total proof work regardless of how many ephemeral identities connect, with a bounded per-caller bucket on top. Tokens are spent only for real proof work; cheap rejects (bad input / not-seeded / blind) never spend them, so a not-seeded flood can't starve honest callers.
  • Never reads attacker keys — the service only proves keys present in appRegistry and never calls store.get() on caller-supplied input, so it can't be coerced into allocating unbounded phantom Hypercores.

v1 proves the drive metadata core; blobs-core proofs are a follow-up. Verify with npx brittle test/unit/storage-proof-service.test.js.

Install systemd service

sudo cp /opt/hiverelay/hiverelay.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable hiverelay
sudo systemctl start hiverelay

Verify

sudo systemctl status hiverelay
journalctl -u hiverelay -f          # Follow logs
curl http://127.0.0.1:9100/health   # Health check
curl http://127.0.0.1:9100/status   # Full status

Option 2: Docker

Base image: use glibc, not musl

If you're building your own image from the repo Dockerfile, or rolling your own from scratch, use a glibc-based base (node:22-bookworm-slim, node:20-bookworm-slim, Ubuntu, RHEL-family, etc.). Alpine/musl is not currently supported because:

  • udx-native ships prebuilds for linux-x64/linux-arm64 (glibc) but not for linux-x64-musl/linux-arm64-musl. On Alpine, require-addon detects musl via /etc/alpine-release and looks for a musl prebuild that doesn't exist → first import crashes with Cannot find module '/prebuilds/linux-x64-musl/udx-native.node'.
  • sodium-native has the same gap.
  • Building from source on Alpine works but requires cmake-bare + cmake-napi + python3 + make + g++ in both build and runtime stages and roughly doubles image size.

Trade-off: bookworm-slim is ~50 MB larger than Alpine but loads the native prebuilds directly, so the runtime image stays clean and starts without surprises.

Tracked in #21. Upstream context: holepunchto/udx-native — add musl prebuilds for parity with the P2P/relay/CLI ecosystem's common Alpine deployments.

Quick start

docker run -d \
  --name hiverelay \
  --restart unless-stopped \
  -v hiverelay-data:/data \
  -p 9100:9100 \
  ghcr.io/bigdestiny2/p2p-hiverelay:latest \
  start --storage /data --region NA

With docker-compose

cd /opt/hiverelay
docker compose up -d
docker compose logs -f

Host networking (recommended for best DHT performance)

docker run -d \
  --name hiverelay \
  --restart unless-stopped \
  --network host \
  -v hiverelay-data:/data \
  ghcr.io/bigdestiny2/p2p-hiverelay:latest \
  start --storage /data

Host networking gives HyperDHT direct UDP access without NAT translation, which improves peer discovery and hole-punching reliability.

Logging

HiveRelay uses structured JSON logging via pino.

Log levels

Set via environment variable:

HIVERELAY_LOG_LEVEL=info    # Default: info
HIVERELAY_LOG_LEVEL=debug   # Verbose (development)
HIVERELAY_LOG_LEVEL=warn    # Quiet (production)

View logs

# systemd
journalctl -u hiverelay -f
journalctl -u hiverelay --since "1 hour ago"

# Docker
docker logs -f hiverelay

# Pretty-print (install pino-pretty globally)
journalctl -u hiverelay -o cat | npx pino-pretty

Log rotation (systemd)

journald handles rotation automatically. To configure retention:

sudo vi /etc/systemd/journald.conf
# SystemMaxUse=500M
# MaxRetentionSec=30day
sudo systemctl restart systemd-journald

Monitoring

Prometheus

Scrape http://localhost:9100/metrics from your Prometheus instance:

# prometheus.yml
scrape_configs:
  - job_name: hiverelay
    static_configs:
      - targets: ['your-server:9100']
    scrape_interval: 30s

Available metrics:

  • hiverelay_uptime_seconds — Node uptime
  • hiverelay_seeded_apps — Number of apps being seeded (appRegistry size)
  • hiverelay_bytes_stored — Total bytes stored
  • hiverelay_bytes_served — Total bytes served to peers
  • hiverelay_cores_seeded — Cores routed through Seeder.seedCore() (registry log core + any standalone cores). Does NOT include appRegistry-managed Hyperdrive cores — see the registry counters below.
  • hiverelay_app_registry_entries — Total appRegistry entries
  • hiverelay_app_registry_anchored — Entries with blob blocks fully replicated locally
  • hiverelay_app_registry_unanchored — Entries waiting on the repair pass
  • hiverelay_app_registry_cores — Underlying Hypercores managed via appRegistry (≈ 2× entries: meta + blob per Hyperdrive)
  • hiverelay_connections — Current active connections
  • hiverelay_active_circuits — Active relay circuits
  • hiverelay_process_heap_bytes — V8 heap usage
  • hiverelay_process_rss_bytes — Resident set size

Why both cores_seeded and app_registry_cores exist: the coresSeeded counter pre-dates the appRegistry-managed download-range work (PR #24 / v0.8.21). It only counts what's routed through Seeder.seedCore, which is the seedingRegistry's log core and a small number of standalone seeds. Dashboards that read it for "how much is this relay serving?" undercount by 1000× on a busy relay; switch to hiverelay_app_registry_cores for that question, and keep hiverelay_cores_seeded for "is the registry log seeding healthy?"

Health checks

# Simple health check (for load balancers, uptime monitors)
curl -f http://127.0.0.1:9100/health

# Full status
curl http://127.0.0.1:9100/status

# Peer list
curl http://127.0.0.1:9100/peers

Disk-usage signal

/status includes a disk field with the relay's storage volume state:

{
  "disk": {
    "usedPct": 47,
    "usedBytes": 24052555776,
    "freeBytes": 26789236736,
    "totalBytes": 51190108160,
    "mountPath": "/data",
    "status": "ok",
    "checkedAt": 1779543201000
  }
}

status is one of:

  • ok — under diskWarnThreshold (default 85%)
  • warn — at or above warn threshold
  • critical — at or above diskCriticalThreshold (default 95%)

The check runs every 30 seconds in the background (df -kP); no I/O on the status hot path. Override thresholds via config:

{
  "diskWarnThreshold": 80,
  "diskCriticalThreshold": 92,
  "diskRefreshIntervalMs": 60000
}

Set "diskHealthGate": true to make /health return 503 when disk is critical, so load balancers / uptime monitors can drain traffic before the volume actually fills. Default is false (preserves existing /health semantics — always 200 unless the process is down).

This signal exists because #27 traced a 15-hour debugging fire to a 1 GB Fly.io volume hitting 100%. No relay-side counter surfaced the capacity pressure; the visible symptoms all looked like code bugs. This is the cheap signal that closes the observability gap.

API Security

The HTTP API binds to 0.0.0.0 by default for remote access compatibility (relays are public infrastructure). To restrict to localhost, set apiHost: '127.0.0.1' in config.

TLS with Caddy (Recommended)

Caddy provides automatic HTTPS with Let's Encrypt certificates:

# Install Caddy
sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update && sudo apt install caddy

# Configure
sudo tee /etc/caddy/Caddyfile <<'EOF'
relay-us.p2phiverelay.xyz {
    reverse_proxy 127.0.0.1:9100

    header {
        Strict-Transport-Security "max-age=31536000; includeSubDomains"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "DENY"
    }
}
EOF

sudo systemctl enable caddy
sudo systemctl restart caddy

Caddy auto-obtains and renews TLS certificates. No manual cert management needed.

TLS with NGINX (Alternative)

server {
    listen 443 ssl http2;
    server_name relay.yourdomain.com;

    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;
    ssl_protocols TLSv1.3;

    location / {
        proxy_pass http://127.0.0.1:9100;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

API Authentication (Optional — Private Relays)

For private relay operators who want to restrict who can seed:

# Generate and set API key
export HIVERELAY_API_KEY=$(openssl rand -hex 32)
# Add to systemd: Environment=HIVERELAY_API_KEY=<key>

When HIVERELAY_API_KEY is set:

  • All write endpoints (POST /seed, /unseed, /registry/*) require Authorization: Bearer <key> header
  • Read endpoints remain open (health, metrics, catalog, gateway)
  • Ownership signatures and registration challenges become available (opt-in, verified when provided)

Note: Public relays (like the official HiveRelay network) do NOT use API keys. Relays are open infrastructure — anyone can seed. Rate limiting and storage limits prevent abuse.

Rate Limiting

Built-in rate limiting protects against abuse:

  • HTTP API: 60 requests/minute per IP, 64KB max request body
  • P2P Protocol: Token bucket rate limiter per peer key
  • Directory listings: Max 1000 entries with timeout protection

Firewall

HiveRelay needs outbound UDP for HyperDHT. No specific inbound ports need to be opened — HyperDHT handles NAT traversal automatically.

# If you want to explicitly allow (optional, usually not needed):
# UDP — HyperDHT (ephemeral ports)
sudo ufw allow out proto udp

# API — only if exposing via reverse proxy
sudo ufw allow 9100/tcp

Storage Management

Default max storage: 50 GB. Configure with --max-storage:

hiverelay start --max-storage 100GB

Monitor disk usage:

curl http://127.0.0.1:9100/status | jq '.seeder.totalBytesStored'
du -sh /var/lib/hiverelay

The relay node tracks storage per-core and will reject new seed requests when approaching the limit.

Scaling

Vertical

Increase --max-connections (default 256) and --max-storage. A single node can handle hundreds of concurrent peers.

Horizontal

Run multiple relay nodes on different machines. They discover each other automatically via the DHT — no coordination needed. Each node independently:

  • Announces on the discovery topic
  • Accepts seed requests based on local capacity
  • Responds to proof-of-relay challenges

Resource guidelines

Workload RAM Disk CPU Connections
Light (< 10 apps) 512 MB 10 GB 1 core 64
Medium (10-50 apps) 1 GB 50 GB 2 cores 256
Heavy (50+ apps) 2 GB 200 GB 4 cores 512

Updating

# Bare metal
cd /opt/hiverelay
git pull
npm ci --omit=dev
sudo systemctl restart hiverelay

# Docker
docker compose pull
docker compose up -d

Troubleshooting

Node won't start

# Check logs
journalctl -u hiverelay --no-pager -n 50

# Check port conflict
ss -tlnp | grep 9100

# Check storage permissions
ls -la /var/lib/hiverelay

No peers connecting

# Check DHT connectivity
curl http://127.0.0.1:9100/peers

# Check if UDP is blocked
# HyperDHT needs outbound UDP access

High memory usage

# Check metrics
curl http://127.0.0.1:9100/metrics | grep process

# Reduce connections
hiverelay start --max-connections 128

# systemd will enforce MemoryMax=2G and restart if exceeded

Apps not appearing in catalog

If you seed apps but they don't show in /catalog.json:

  1. Check manifest.json exists — Apps MUST have a /manifest.json file in their root:

    {
  "id": "my-app",
  "name": "My App",
  "description": "App description",
  "version": "1.0.0",
  "categories": ["utility"]
}

  2. Convert Pear keys correctly — Pear uses z-base-32 encoding, but the API expects hex:

    // Convert pear://KEY to hex
const z32 = require('z32')
const hexKey = z32.decode('om5cpdjjp4g4wa15r9wjhjiex9jjmcsacwsw44hzsrtsz171ykfy').toString('hex')
// Result: 82f6c68d296e8daa625b27e89e26a87fd295b2d8652d4d6b97b1236bcbb2028a

  3. Check gateway can access drive — The relay must be able to replicate the drive:

    # Test gateway access
curl http://localhost:9100/v1/hyper/HEX_KEY/manifest.json

TLS/HTTPS issues

Problem: Caddy fails to obtain certificates or shows certificate errors.

Check domain configuration:

# Verify DNS resolves to this server
dig +short relay-us.p2phiverelay.xyz

# Check Caddy is using correct domain (NOT p2p-hiverelay.xyz with hyphen)
cat /etc/caddy/Caddyfile | grep -E "^\w+\.p2p"

Common fixes:

  • Ensure domain in Caddyfile matches DNS (use p2phiverelay.xyz not p2p-hiverelay.xyz)
  • Clear Caddy's certificate cache if switching from staging: rm -rf ~/.local/share/caddy/certificates
  • Check rate limits: Let's Encrypt allows 50 certificates per domain per week

App Deployment Checklist

When deploying Pear apps to HiveRelay:

  • App has manifest.json with id, name, description, version
  • App is staged with pear stage dev .
  • Pear key is converted from z-base-32 to hex format
  • App is seeded via POST /seed {"appKey": "hex", "appId": "..."}
  • Verify with GET /catalog.json — app should appear within 5 seconds
  • Test app loads via GET /v1/hyper/HEX_KEY/index.html