TLS handshake error-Let's Encrypt

[Jainam-17-18]

I have deployed my application and it's fine when ssl: false under proxy.

Now I have to do ssl: true.

proxy:
  ssl: true
  host: subdomain.domain.com
  app_port: 8081
  response_timeout: 300
  healthcheck:
    interval: 3
    path: /healthcheck
    timeout: 3

But getting error:

{"time":"xxxx","level":"INFO","msg":"http: TLS handshake error from 103.136.75.230:61871: acme/autocert: unable to satisfy \"https://acme-v02.api.letsencrypt.org/acme/authz/xxxxxxxx/xxxxxxxx\" for domain \"mysubdomain.domain.com\": no viable challenge type found"}

Server Setup:
Have checked my server's both Port 80 and 443 is showing opened on https://portchecker.co/ .
Server transfer request to VM where the application had been deployed.

I have a server's port (abcd) which map with NAT to VM's port (443).
I also have another port (wxyz) which map with NAT to VM's port (80).

[dipth]

I have the same problem.

config/environments/production.rb

Rails.application.configure do
  config.assume_ssl = true
  config.force_ssl = true
end

config/deploy.yml

proxy:
  ssl: true
  host: alpha.REDACTED.com

Cloudflare DNS

Cloudflare TLS

Proxy log

2025-01-11T15:45:58.312132198Z {"time":"2025-01-11T15:45:58.311506309Z","level":"INFO","msg":"http: TLS handshake error from 162.158.134.133:40122: acme/autocert: unable to satisfy \"https://acme-v02.api.letsencrypt.org/acme/authz/REDACTED/REDACTED\" for domain \"alpha.REDACTED.com\": no viable challenge type found"}

Firewall

I am able to connect to the server using http via Cloudflare:

╰─⠠⠵ curl -I http://alpha.REDACTED.com                   
HTTP/1.1 301 Moved Permanently
Date: Sat, 11 Jan 2025 16:02:28 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Location: https://alpha.REDACTED.com/
cf-cache-status: DYNAMIC
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=REDACTED"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 9006219e5ae9be56-CPH
alt-svc: h3=":443"; ma=86400
server-timing: cfL4;desc="?proto=TCP&rtt=10642&min_rtt=10642&rtt_var=5321&sent=1&recv=3&lost=0&retrans=0&sent_bytes=0&recv_bytes=78&delivery_rate=0&cwnd=245&unsent_bytes=0&cid=0000000000000000&ts=0&x=0"

I am also able to connect to the server using http directly:

╰─⠠⠵ curl -I -H "Host: alpha.REDACTED.com" REDACTED_IP
HTTP/1.1 301 Moved Permanently
Connection: close
Content-Type: text/html; charset=utf-8
Location: https://alpha.REDACTED.com/

[dipth]

I was able to find a solution in the thread here:
basecamp/kamal-proxy#26

Turns out that I had a Cloudflare WAF page rule that initiated a browser challenge to visitors from some country that happened to be the country that LetsEncrypt issues a request from to verify domain ownership.
Adding a condition to that rule to skip the rule for the path that LetsEncrypt uses, fixed the issue.

[fatihtas]

for anyone facing this issue with CloudFlare.

this is the solution.

proxy:
  ssl: true
  host: namazeven.com
  forward_headers: true
  healthcheck:
    path: /up
    interval: 3
    timeout: 10

production.rb:

config.assume_ssl = true 
config.force_ssl = true

remove your current DNS settings for @ and www. or other subdomains. Than add again, without any proxy on CloudFlare.

then deploy your app, then open proxies.
I have tried almost like 40 different variations. this solved it. i had to create new servers and make sure my deployment worked on them while upgrading Kamal. It worked on new servers with subdomain redirecting to the new ip, but did not worked for old server.

CloudFlare had to be reseted it turns out, as I mentioned above.

[fatihtas]

3 months later. my website suddenly stopped working out of blue. I spent 3 days on and off fixing this issue. and I found my own answer back here! and it worked again!!
But with this one caveat.. You have to wait some time to unlock acme missing certificate timeout.

I have totally forgotten about having this issue before.. i guess i am quite old now.

[andremleblanc]

I had the same issue, working for five months then suddenly stopped. I just toggled the Cloudflare proxy, thanks to this comment, and then it started working again.

[sedubois]

I ran into a similar issue where my site's certificate expired (managed by Kamal / Let's Encrypt) and wasn't renewing. The symptom was 522 timeouts when visiting the site (which is behind Cloudflare).

It wasn't Kamal's fault (although it would have been great if it informed me somehow of the issue), it was because I added a rule in the Cloudflare WAF/security settings (Security > Security rules), which blocked Let's Encrypt's validation servers. The fix is to make sure /.well-known/acme-challenge remains allowed even if other request paths are blocked:

"When incoming requests match…"

• (some condition when the request should trigger the security rule)
• AND URI Path does not contain /.well-known/acme-challenge

As I was missing the second part, LE validation servers couldn't reach the site, the ACME challenge failed, certificate renewal failed silently, and expired after 90 days. After expiry, LE's rate limiting of 5 failed validations/domain/hour got triggered, and the account also got paused.

Recovered like this:

1. Fixed the WAF rule to allow /.well-known/acme-challenge/
2. Unpaused pause by visiting the URL shown in the kamal logs: ssh root@<your-server-ip> "docker logs kamal-proxy 2>&1 | grep -iE 'rateLimited|unpause|paused'" (looks like https://portal.letsencrypt.org/sfe/v1/unpause?jwt=...)
3. Waited up to one hour for rate limit to clear
4. Cleared autocert cache: rm -rf /var/lib/docker/volumes/kamal-proxy-config/_data/certs/* (perhaps unnecessary)
5. Restarted kamal-proxy container: docker restart kamal-proxy (perhaps unnecessary)
6. Triggered certificate acquisition with an HTTPS request
7. Profit: site loaded again without timeout