Skip to content

Latest commit

 

History

History
62 lines (49 loc) · 15.7 KB

File metadata and controls

62 lines (49 loc) · 15.7 KB

dash-proxy Roadmap

Proxy-side roadmap for the dash fork. The cross-repo release sequencing, strategic frame, and gem-side items live in the twin ROADMAP.md in ../kamal; this file carries the proxy work with code anchors. Strategy in one line: own what basecamp rejected (headers #62, rate limiting #20, compression #19, PROXY protocol #31), port what they left stuck (#63 on-demand TLS, #204 mTLS, #216 basic auth, #199 min-TLS), and ship the table-stakes resilience knobs every other proxy has.

R1 — Foundations & fixes (v0.9.2.2) — all S-sized

Item Anchor
Start CertificateRenewalManager — DONE (#84). The registry stack was deleted rather than started; background renewal now runs in certRenewer, started by DynamicDomainManager.Start() and covering every managed certificate internal/server/domain_renewal.go
Wire the 3 certificate Prometheus metrics — expiry gauge, renewals counter, totals gauge have setters but no callers internal/metrics/metrics.go:81-101 → call from cert managers
Slowloris / server-timeout defaults — no ReadHeaderTimeout/ReadTimeout/WriteTimeout/IdleTimeout on any listener (all Go zero = unlimited) discussion basecamp#196; internal/server/server.go:141,150,192; make configurable via run flags, ship safe defaults

R2 — Timeouts & resilience

Item Evidence Anchor
Per-route timeouts — per-(host, path-prefix) override of target timeout; each binding is already a candidate carrier #53; SSE cluster #46/#54/#137/#186 pathBinding (internal/server/service_map.go:15) + thread into Target.createProxyHandler (internal/server/target.go:293)
Whole-request deadline--target-timeout maps to Transport.ResponseHeaderTimeout only (target.go:302); long bodies can run forever #53 deadline middleware in createMiddleware (internal/server/service.go:458); must exempt WebSocket/SSE (response_buffer_middleware.go:86 bypass logic)
Retries / hold-until-healthy — retry idempotent methods on next target; brief hold during redeploy blips #71; Caddy lb_try_duration LoadBalancer.StartRequest (internal/server/load_balancer.go:174); replay needs request buffering (internal/server/buffer.go)
Custom upstream 502/503 error pages nginx proxy_intercept_errors, Apache ProxyErrorOverride DONE — --intercept-errors (internal/server/error_intercept_middleware.go). Errors the proxy raises already rendered through ErrorPageMiddleware; the gap was error responses the target wrote itself
Upstream pool tuning — MaxConnsPerHost, IdleConnTimeout, keep-alives inventory DONE — --target-max-conns, --target-max-idle-conns, --target-idle-conn-timeout, --target-dial-timeout, --target-disable-keep-alives (internal/server/target_pool.go). The transport was a bare struct literal, so the unset fields were Go zero rather than http.DefaultTransport's values: idle conns never expired and were never staleness-checked before reuse, and there was no dial timeout at all. Zero in TargetOptions now means "proxy default", resolved server-side so restored state files and older RPC clients get it too

R3 — Security & access

Item Evidence Anchor
Basic auth per service port PR #216 (open); kamal#1604 DONE — --basic-auth <user>:<pass> (internal/server/basic_auth.go). Deliberately not a createMiddleware middleware: that chain wraps serviceRequestWithTarget, which is where the HTTPS redirect lives, so a middleware there challenges before the 301 and the browser sends the password in cleartext. The check sits inline after handleRedirectsIfNeeded. Credentials are hashed CLI-side (salted SHA-256), so no plaintext crosses the RPC socket or reaches the state file. Per-path scoping is served by deploying a --path-prefix service with its own credential
IP allow list (CIDR) discussions #143/#144 DONE — --allow-ip/--trusted-proxy on deploy, --metrics-allow-ip on run (internal/server/ip_allow_list.go). Allow-only; a static deny list is the wrong tool for the abuse-blocking people reach for it with. Not a createMiddleware middleware — that chain includes the cert manager's handler, so filtering there breaks ACME HTTP-01 and certificates fail to renew weeks later. Runs as the first check in serviceRequestWithTarget, before the HTTPS redirect. Matches the connecting peer, never a header, unless the peer is inside --trusted-proxy. The logging_middleware.go:70 anchor this row used to name is the trap: that function's remote_addr is raw X-Forwarded-For with no trust check
Per-IP rate limiting (token bucket + burst + allowlist) rejected #20 DONE — --rate-limit/--rate-limit-burst/--rate-limit-exempt (internal/server/rate_limit.go). Per-service rather than the global buildHandler chain, so different services can carry different budgets. Clients are resolved through the same forwardedResolver as --allow-ip (extracted from ip_allow_list.go), because keying on the connecting peer behind a load balancer puts everyone in one bucket. IPv6 counted per /64: a client can pick any address in its own, so per-address counting is both an escape and an unbounded map. Sits after the redirect (a 301 never reaches the upstream) and before the basic auth challenge (so guessing is bounded). Bucket map capped at 50k with lazy eviction of refilled buckets — no janitor goroutine to leak on redeploy
PROXY protocol rejected #31, discussion #41 go-proxyproto listener wrap in server.go; run flag
mTLS (--tls-client-ca-path) port PR #204 (open); kamal#1628 tls.Config.ClientCAs/ClientAuth on HTTPS listener (server.go:158)

R4 — TLS & custom domains

Item Evidence Anchor
On-demand TLS with ask endpoint DONE (PR #50) internal/server/tls_on_demand.go; per-handshake gate in Router.GetCertificate, which asks the on-demand endpoint before any shared manager sees the name
Min-TLS version / ciphers (no MinVersion today → TLS 1.2 default) port PR #199 server.go:158 TLS config; run flag
Cert observability — dashboards/alerts on the R1-wired metrics internal/metrics

R5 — Traffic shaping & headers

Item Evidence Anchor
Header rules (req/resp add/remove/set) rejected #62/#25 DONE — --{set,add,remove}-{request,response}-header on deploy (internal/server/header_rules.go), applied remove → set → add. Request rules run last in Target.rewrite, so a rule outranks the X-Forwarded headers the proxy set; they do not reach health checks, which use their own client (health_check.go:82). Response rules run in ReverseProxy.ModifyResponse, which only sees what the target produced — error pages, the TLS/canonical-host 301, and 401/429 rejections come from the proxy and keep its headers. Host is rejected on the request side: Go carries it in Request.Host, so a rule naming it would silently do nothing. CORS/HSTS/CSP presets not shipped; generic set rules express all three
Weighted canary (--target=b;weight=5) kamal#941, #8 LoadBalancer.nextTarget (load_balancer.go:216) — currently pure round-robin
Cookie session affinity #26 DONE — --session-affinity, --session-affinity-cookie on deploy (internal/server/session_affinity.go). The pin sits in front of LoadBalancer.nextTarget and only ever replaces a selection, so an unpinned deployment runs the rotation it always ran. The cookie carries an HMAC of the target address under a key minted per load balancer, not the address: a cookie tells its holder nothing about the topology, and a guessed address cannot be confirmed by recomputing the digest. That key is per-NewLoadBalancer, so pins reset on deploy and on restart — the targets a pin named are gone by then anyway. A pin naming a target no longer in the healthy pool falls through to the rotation and is re-issued, so a dead pin can never 502. Reads served by a --read-target are not pinned (a replica holds no per-instance session state); writes are, and the existing kamal-writer cookie then keeps that client's reads on its pinned writer. Honoured on the first attempt only — --target-try-duration retries rotate past a pinned target that cannot serve
Redirect/rewrite rules #35; kamal discussions #1214/#97 DONE — --redirect '<pattern>=<replacement>[;status=<code>]' and --rewrite '<pattern>=<replacement>' on deploy (internal/server/redirect_rules.go). Both pieces of evidence only ask for www→apex, which --canonical-host already answers; what shipped is the path-level gap the issue names. Patterns are RE2 anchored to the whole path with $1 expansion. Redirects fold into redirectURLIfNeeded alongside the TLS/canonical hop, so a path move on a TLS service costs the client one redirect, not two; a rule resolving to the request's own URL is dropped so a catch-all cannot loop. A relative replacement is always rebuilt as scheme://host/..., which is what stops a captured //evil.com from becoming a scheme-relative Location. Rewrites apply last in serviceRequestWithTarget, after the health-check exemptions and the allow list, and skip isInternalRequest — the TLS on-demand probe runs the same chain, and a catch-all pointing it at the app's index would approve a certificate for any host
Compression (gzip/zstd/brotli) rejected #19 DONE — --compress, --compress-min-length, --compress-content-type on deploy (internal/server/compression.go, compression_middleware.go), wrapped outermost in Service.createMiddleware. Per-service, not per-target, because that is the only chain that sees proxy-written responses too. Encoding is chosen by the client's q values, ties by the order given. The decision is held until the body's size and type are known, which means a flush before any body must not settle it — ReverseProxy schedules exactly that after the headers of every unknown-length response (flushInterval returns -1), and it races the first write. text/event-stream is excluded outright, so the streaming bypass (response_buffer_middleware.go:86) never sees a compressed stream. The built-in error pages render above the router (server.go:338) and stay uncompressed; --error-pages ones do not
Shared response cache (RFC 9111 + stale-while-revalidate) #27 DONE — --cache and --cache-{max-body,max-ttl,vary-header,vary-cookie,allow-set-cookie} on deploy, --cache-store{,-timeout} and --cache-memory-size on run, kamal-proxy cache purge over a CachePurge RPC (internal/server/cache_*.go). Store is pluggable: in-process LRU or a Redis every proxy shares, always failing open — an unreachable store reads as a miss, never a 5xx. The cache is wired inside serviceRequestWithTarget, not in createMiddleware, so basic auth, the allow list, the rate limit and redirects all run before it; a hit is only ever handed to a client the target would have been asked for. Storing needs an explicit public plus a lifetime — freshness alone is not consent — and Set-Cookie needs --cache-allow-set-cookie on top. Vary is handled by refusing to guess: dimensions named in --cache-vary-header/--cache-vary-cookie are in the key, and a response varying on anything else is passed through uncached rather than risking one client's variant answering another's request. Accept-Encoding is the one exception — the cache sits inside --compress and stores what the target produced, so one entry serves every encoding, and a target-encoded body (Content-Encoding set) is refused instead. Concurrent misses coalesce through inflightGroup, which also caps a burst of stale hits at one background revalidation; the leader settles early with nil the moment a response proves unstorable, so followers are never held behind a stream they cannot be given a copy of. Rollout traffic keys separately — a canary is running different code
Scale-to-zero port PR #197 (open) PauseController states (pause_controller.go) are the natural base
Observability batch — log format selection, OTel traceparent, metrics path excludes #213 counter-proposal DONE — all three. Metrics path excludes shipped first as --exclude-metrics-path on deploy (service.go:175); the "counter-proposal" won and is merged upstream as #213, so nothing was left to port. --log-format json|text (internal/server/log_format.go) swaps the handler for the whole process, not just the access log, because both go through slog.Default(). --trace-context off|propagate|generate (internal/server/trace_context.go) reads the W3C traceparent and logs trace_id/span_id/trace_flags; the middleware sits inside WithLoggingMiddleware in Server.buildHandler, which is what creates the request context it writes to. The logged span_id is the caller's, never one the proxy minted — kamal-proxy exports no spans, so an invented id would parent the app to a span no backend has. generate marks new traces sampled (01) because an unsampled parent silently switches off tracing the app would otherwise have done itself, and drops a tracestate whose traceparent it replaced. The flag is --trace-context, not --traceparent, because OTel already defines TRACEPARENT as an env var carrying a real trace context and every flag here claims the matching env name
Liveness endpoint for external monitors #25 DONE — GET /.kamal-proxy/ping200 (internal/server/ping_handler.go), mounted outermost and unconditionally in Server.buildHandler. It reuses the /.kamal-proxy/ namespace but not DynamicDomainManager.WrapHandler, which only mounts when dynamic domains are configured — the endpoint has to answer on a proxy with zero services, which is the whole point. Outermost is what keeps it out of the access log: the logging middleware never sees it, at the cost of no request ID and no error page on that path. No readiness variant: RestoreLastSavedState runs before Start opens a listener and BeginDrain closes the listeners, so readiness here could only ever be 200 — the TCP state is the real signal. Not mounted on the metrics port, which is opt-in and IP-restricted
Scale-to-zero port PR #228 (open; supersedes the closed #197 the issue names) PauseController states (pause_controller.go) are the natural base. Upstream maintainer blessed the architecture in discussion #222: opt-in direct Docker socket behind a ContainerLifecycle interface, so a restricted host-side start/stop service can replace it later. Mounting docker.sock into the internet-facing proxy is root-equivalent on the host — opt-in only, and only when the feature is enabled
Observability batch — log format selection, OTel traceparent, metrics path excludes #213 counter-proposal logging_middleware.go:81 (fixed JSON today); request_id_middleware.go

Implementation notes (apply to every feature)

  • New per-service knobs go in ServiceOptions (service.go:82), per-target in TargetOptions (target.go:65), one-shot in DeploymentOptions (service.go:76); flags register in internal/cmd/deploy.go / run.go; RPC arg structs in internal/server/commands.go:19-63.
  • ServiceOptions/TargetOptions are JSON-persisted across restarts — new fields must round-trip Service.MarshalJSON/UnmarshalJSON (service.go:273/294) and be default-safe against old state files.
  • Never rename module/binary/RPC/socket (see CLAUDE.md Never Do #1). Image tags stay four-segment vX.Y.Z.N, all four segments ours to choose.
  • Anything reachable from deploy.yml also needs gem-side plumbing — see the flag-mapping table workflow in ../kamal (one file for plain options: lib/kamal/configuration/proxy.rb; three when the loadbalancer tier must carry it too).