A proxy-wide health check route

[igor-alexandrov]

Cloud load balancers like AWS ALB or DigitalOcean droplet balancer does not allow specifying HOST header that is getting sent to the instance. In a multi-application setup, this means two things:

1. We should always have once application in kamal-proxy with proxy: { host: nil } setting.
2. The health status of the instance is determined from the status of a single application, which is wrong.

Based on these two facts, I suggest adding an ability to specify a proxy-wide health check route that will work for all instances.

[kczpl]

Hi! I've encountered the same problem and would be happy to contribute to solving it.

Before Kamal v2.0, I resolved this issue using Traefik labels to route traffic based on specific User-Agent headers. Since I have both a FE and BE applications hosted on the same server, I need to separate my health checks:

 traefik.http.routers.app-web-staging.rule: "Host(`api.domain.com`) || (Headers(`User-Agent`, `ELB-HealthChecker/2.0`) && PathPrefix(`/health`))"

I believe we should add an option that allows routing based on headers and paths, not just the host, as I don’t see any other way to handle managed load balancers' health checks due to the limited request information they provide.

Alternatively, we could add a way to easily expose the container port, like 0.0.0.0:3000, and target the health check there. Since we're dealing with a managed service, which is usually protected by security groups and other measures.

[igor-alexandrov]

As a workaround for now, you can deploy the smallest (around 80Kb) image https://hub.docker.com/r/lipanski/docker-static-website without the host option and then deploy all your apps as usual with the host option.

[kczpl]

Thank you! I temporarily changed the range (200-499) of my health checks since it’s pointless anyway, but I didn’t know that image exists. :))

[dhh]

Help me understand the problem here. The proxy/ssl|host settings are only relevant when you're running with a single box. Why would there be a load balancer in front then?

[igor-alexandrov]

The problem is that almost all balancers allow to define only an HTTP path and port as a healthcheck. This works well when you have a single app on the host.

Imagine yourself having a balancer. It sends a /up HTTP GET request without a HTTP_HOST header (because your web interface does not allow you to configure this) to check that your instance is up and running. kamal-proxy receives the request and tries to respond, but because you have multiple app containers configured and each of them has a host option populated (which makes sense) it cannot find a container to process the request.

Currently, kamal-proxy will respond with HTTP 404, which makes sense because it will not wind a container to route traffic to.

The idea is to add a default traffic point like /up that will respond with the host machine HTTP status when no HTTP_HOST header is defined.

[Rohland]

Help me understand the problem here. The proxy/ssl|host settings are only relevant when you're running with a single box. Why would there be a load balancer in front then?

Running into this with the upgrade to 2.0 (very excited to drop Traefik!). With Kamal 1*, we have TLS terminating on our load balancers (AWS ELB), however, we maintained TLS right through to hosts using self signed certificates with Traefik. This ensures traffic is encrypted from client => load balancer => host. In some ways you're right, it doesn't make sense to leverage Kamal Proxy's auto SSL stuff in this setup, but it would be nice to support encryption across all network hops whilst keeping the proxy set up simple.

#17 might address this scenario, but still requires setting a host (as far as I can tell). Not sure it will work with something like the following. Will test this once kamal supports the custom TLS stuff that appears to now be supported in Kamal Proxy.

proxy:
  ssl: true
  ssl_certificate: /....
  ssl_private_key: /...
  hosts: *

[kczpl]

The proxy/ssl|host settings are only relevant when you're running with a single box. Why would there be a load balancer in front then?

I have applications deployed horizontally, with three web app servers behind an AWS ALB, which handles SSL termination and health checks.

The ALB health check request looks like this:

GET / HTTP/1.1
Host: 172.x.x.x
Connection: close
User-Agent: ELB-HealthChecker/2.0
Accept-Encoding: gzip, compressed

unfortunately, there is no option to add custom headers (like Host) to the managed health checks (damn you, AWS).
The issue is that I want to route this managed health check to my app so that it can respond with whether the app is healthy or not. However, I don't currently see a way to do this.

[mooktakim]

This is related to the issue I submitted in the kamal repo:
basecamp/kamal#1096

I think the solution should be to add a "default" host to the proxy.

Useful also when you want an app to show when you visit the IP address.

[mooktakim]

We really need "default host" option. I don't understand why this isn't already available.
kamal-proxy currently is unusable behind a load balancer.

I am looking to move away from kamal.

It's not enough to just allow single server deployment, we need to be able to put it behind load balancer for obvious reasons.

[cblunt]

I just stumbled across this issue setting up multiple dev instances on a single (development) server. The server is running on GCP with a load balancer in front and the healthcheck pinging /up without a host header.

It's not ideal, but for now I've found the following workaround. Hope it helps!

---

1. Customise an nginx image that returns 200/OK to /up (configure as necessary)

FROM nginx

RUN <<EOF 
echo "server {
    listen       80;
    server_name  localhost;

    location /up {
        default_type text/plain;
	return 200 "OK";
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}" > /etc/nginx/conf.d/default.conf
EOF

2. Build and deploy the image using the kamal-proxy container:

$ docker build --tag nginx-healthcheck .
$ docker run -d -it --network kamal --name gcp-healthcheck-1 nginx-healthcheck
$ docker exec -it kamal-proxy kamal-proxy deploy gcp-healthcheck-1 --target gcp-healthcheck-1:80 --health-check-path "/up" --deploy-timeout 5s

This is OK for internal/dev servers, but probably not production. As the host is not specified (--host *), kamal-proxy will respond to any URL that points to the server and isn't connected to an application.

[rachelgraves]

I just ran into this as well. We're using DigitalOcean's load balancer and have multiple sites on our droplets. In this instance I switched DigitalOcean's load balancer to use TCP and port 22.

Out of interest what's to be gained by pinging kamal-proxy in this instance? We can't know that each site is up given the limitations of DigitalOcean's load balancer so all we'd know with a default host is that kamal-proxy is running, not anything we care about behind it.

[mooktakim]

its not perfect solution, but at least you can know each node is "up and running". so the load balancer doesn't remove them from active list.

in a production env you'd run one app for server, but if you have multiple apps per server you "assign" default app, which can be used as the load balancer healthcheck.

[mooktakim]

I suppose if you really want to be fancy you could create a custom "healthchecker" app that is the default, which can check if all the other apps are successfully running.

[anatoly-pashin-axon21]

I was also wondering how to use AWS ALB to split load to multiple servers. And the solution was to remove “hosts” from the deployment config. Now my app is responding to any host and the ALB checks pass.

https://kamal-deploy.org/docs/configuration/proxy/#hosts:

If no hosts are set, then all requests will be forwarded, except for matching requests for other apps deployed on that server that do have a host set.

[mooktakim]

I see, so essentially this makes this app the "default host" app. This is exactly what i was looking, thank you

[mooktakim]

Hmm, this would prevent SSL cert from being created by kamal for the domain i think.
If ALB provides SSL, that would be fine.

[anatoly-pashin-axon21]

Yes, I use AWS certificates, ALB does HTTPS termination.

[dlibanori]

What if kamal forward health check to an app using a query parameter? You register a health check at your load balancer and kamal forward it to app. Example: at ALB you register health check as path /up?host=app-1.example.com and kamal-proxy forward /up to app-1 container. For app-2 you register path /up?host=app-2.example.com and so on

[mooktakim]

I use Caddy in front of kamal proxy and it allows me to add default host. Caddy does all the handling and /up still works.

I describe much of the deployment in this blog article i wrote about subdomain SaaS:
https://mooktakim.com/blog/building-saas-with-rails-and-kamal/

[dlibanori]

Actually we can write a very simple application to do that: it split /up?host=app-1.example.com and make a request to app-1.example.com/up. This app uses kamal without proxy.host[s], so it will get all ALB health check requests and direct it the right container. But now we have to deploy all our applications + the health check forward-er.

But I think it should be out-of-the-box feature in kamal

[dlibanori]

@mooktakim nice, but I still think it should be included inside kamal/kamal-proxy

[mooktakim]

I agree, it's a basic feature!

[kevinmcconnell]

One approach I've been thinking about for this is to publish each app's health check to an app-specific path, and skip the Host checking on requests to those paths. That way you could configure downstream LBs to health check whichever apps you want. I think this would support all the situations mentioned in this discussion.

I have an initial spike of this in #227. Still feeling it out a bit, but I'd be interested to know if anyone has a case that this approach wouldn't solve?