kamal-proxy healthcheck connection refused, but server is running. What could be the cause?

[lorenzharfst]

Greetings!

At the time of writing this I've spent about 6h with the same issue and I finally decided to start a thread on it.

• I have this (still in its infancy) app with some models, authentication (w/ Devise) and one home#index route.
• This app works when run locally with bin/rails server.
• I have checked whether the environment configuration might affect the connection, but it's all default values and nothing stood out.
• I have made sure ports 80 (and 3000, just in case) are open with ufw status
• I have gotten into the deployed container and tried accessing /up via curl and it worked, so it does work locally

I would greatly appreciate a nudge in the right direction, because I'm honestly at a loss! The only thing I'm kinda fixating on is that kamal's docker run doesn't include a -p arg, but I assume that's proxy/tunneling magic?

Instead of explaining my configuration, I think it would be best if I just showed the files.

deploy.yml

# Name of your application. Used to uniquely configure containers.
service: event_reminder

# Name of the container image (use your-user/app-name on external registries).
image: event_reminder

# Deploy to these servers.
servers:
  web:
    - 192.168.1.46
  # job:
  #   hosts:
  #     - 192.168.0.1
  #   cmd: bin/jobs

# Enable SSL auto certification via Let's Encrypt and allow for multiple apps on a single web server.
# If used with Cloudflare, set encryption mode in SSL/TLS setting to "Full" to enable CF-to-app encryption.
#
# Using an SSL proxy like this requires turning on config.assume_ssl and config.force_ssl in production.rb!
#
# Don't use this when deploying to multiple web servers (then you have to terminate SSL at your load balancer).
#
# Where you keep your container images.
registry:
  # Alternatives: hub.docker.com / registry.digitalocean.com / ghcr.io / ...
  server: localhost:5555

  # Needed for authenticated registries.
  # username: your-user

  # Always use an access token rather than real password when possible.
  # password:
  #   - KAMAL_REGISTRY_PASSWORD

# Inject ENV variables into containers (secrets come from .kamal/secrets).
env:
  secret:
    - RAILS_MASTER_KEY
    - DB_HOST
    - DB_PORT
    - DB_USER
    - DB_PASSWORD
  clear:
    # Run the Solid Queue Supervisor inside the web server's Puma process to do jobs.
    # When you start using multiple servers, you should split out job processing to a dedicated machine.
    SOLID_QUEUE_IN_PUMA: true

    # Set number of processes dedicated to Solid Queue (default: 1)
    # JOB_CONCURRENCY: 3

    # Set number of cores available to the application on each server (default: 1).
    # WEB_CONCURRENCY: 2

    # Match this to any external database server to configure Active Record correctly
    # Use event_reminder-db for a db accessory server on same machine via local kamal docker network.
    # DB_HOST: 192.168.0.2

    # Log everything from Rails
    # RAILS_LOG_LEVEL: debug

# Aliases are triggered with "bin/kamal <alias>". You can overwrite arguments on invocation:
# "bin/kamal logs -r job" will tail logs from the first server in the job section.
aliases:
  console: app exec --interactive --reuse "bin/rails console"
  shell: app exec --interactive --reuse "bash"
  logs: app logs -f
  dbc: app exec --interactive --reuse "bin/rails dbconsole --include-password"

# Use a persistent storage volume for sqlite database files and local Active Storage files.
# Recommended to change this to a mounted volume path that is backed up off server.
volumes:
  - "event_reminder_storage:/rails/storage"

# Bridge fingerprinted assets, like JS and CSS, between versions to avoid
# hitting 404 on in-flight requests. Combines all files from new and old
# version inside the asset_path.
asset_path: /rails/public/assets

# Configure the image builder.
builder:
  arch: amd64

  # # Build image via remote server (useful for faster amd64 builds on arm64 computers)
  # remote: ssh://docker@docker-builder-server
  #
  # # Pass arguments and secrets to the Docker build process
  # args:
  #   RUBY_VERSION: 4.0.1
  # secrets:
  #   - GITHUB_TOKEN
  #   - RAILS_MASTER_KEY

# Use a different ssh user than root
ssh:
  user: _[!REDACTED, BUT SSH CONNECTION WORKS!]_
  keys: _[!REDACTED, BUT SSH CONNECTION WORKS!]_

# Use accessory services (secrets come from .kamal/secrets).
# accessories:
#   db:
#     image: mysql:8.0
#     host: 192.168.0.2
#     # Change to 3306 to expose port to the world instead of just local network.
#     port: "127.0.0.1:3306:3306"
#     env:
#       clear:
#         MYSQL_ROOT_HOST: '%'
#       secret:
#         - MYSQL_ROOT_PASSWORD
#     files:
#       - config/mysql/production.cnf:/etc/mysql/my.cnf
#       - db/production.sql:/docker-entrypoint-initdb.d/setup.sql
#     directories:
#       - data:/var/lib/mysql
#   redis:
#     image: valkey/valkey:8
#     host: 192.168.0.2
#     port: 6379
#     directories:
#       - data:/data

Dockerfile

# syntax=docker/dockerfile:1
# check=error=true

# This Dockerfile is designed for production, not development. Use with Kamal or build'n'run by hand:
# docker build -t event_reminder .
# docker run -d -p 80:80 -e RAILS_MASTER_KEY=<value from config/master.key> --name event_reminder event_reminder

# For a containerized dev environment, see Dev Containers: https://guides.rubyonrails.org/getting_started_with_devcontainer.html

# Make sure RUBY_VERSION matches the Ruby version in .ruby-version
ARG RUBY_VERSION=4.0.1
FROM docker.io/library/ruby:$RUBY_VERSION-slim AS base

# Rails app lives here
WORKDIR /rails

# Install base packages
RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y curl libjemalloc2 libvips postgresql-client && \
    ln -s /usr/lib/$(uname -m)-linux-gnu/libjemalloc.so.2 /usr/local/lib/libjemalloc.so && \
    rm -rf /var/lib/apt/lists /var/cache/apt/archives

# Set production environment variables and enable jemalloc for reduced memory usage and latency.
ENV RAILS_ENV="production" \
    BUNDLE_DEPLOYMENT="1" \
    BUNDLE_PATH="/usr/local/bundle" \
    BUNDLE_WITHOUT="development" \
    LD_PRELOAD="/usr/local/lib/libjemalloc.so"

# Throw-away build stage to reduce size of final image
FROM base AS build

# Install packages needed to build gems
RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y build-essential git libpq-dev libyaml-dev pkg-config && \
    rm -rf /var/lib/apt/lists /var/cache/apt/archives

# Install application gems
COPY vendor/* ./vendor/
COPY Gemfile Gemfile.lock ./

RUN bundle install && \
    rm -rf ~/.bundle/ "${BUNDLE_PATH}"/ruby/*/cache "${BUNDLE_PATH}"/ruby/*/bundler/gems/*/.git && \
    # -j 1 disable parallel compilation to avoid a QEMU bug: https://github.com/rails/bootsnap/issues/495
    bundle exec bootsnap precompile -j 1 --gemfile

# Copy application code
COPY . .

# Precompile bootsnap code for faster boot times.
# -j 1 disable parallel compilation to avoid a QEMU bug: https://github.com/rails/bootsnap/issues/495
RUN bundle exec bootsnap precompile -j 1 app/ lib/

# Precompiling assets for production without requiring secret RAILS_MASTER_KEY
RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile




# Final stage for app image
FROM base

# Run and own only the runtime files as a non-root user for security
RUN groupadd --system --gid 1000 rails && \
    useradd rails --uid 1000 --gid 1000 --create-home --shell /bin/bash
USER 1000:1000

# Copy built artifacts: gems, application
COPY --chown=rails:rails --from=build "${BUNDLE_PATH}" "${BUNDLE_PATH}"
COPY --chown=rails:rails --from=build /rails /rails

# Entrypoint prepares the database.
ENTRYPOINT ["/rails/bin/docker-entrypoint"]

# Start server via Thruster by default, this can be overwritten at runtime
EXPOSE 80
# Adding 3000 because server is unreachable and I'm desperately trying random stuff
EXPOSE 3000
CMD ["./bin/thrust", "./bin/rails", "server"]

kamal-proxy error:

{"time":"2026-04-25T12:50:22.448649532Z","level":"INFO","msg":"Healthcheck failed","url":"http://5ca04cd80df6:80/up","error":"Get \"http://5ca04cd80df6:80/up\": dial tcp: lookup 5ca04cd80df6 on 127.0.0.53:53: read udp 127.0.0.1:54592->127.0.0.53:53: read: connection refused"}

Server output:

=> Booting Puma
=> Rails 8.1.3 application starting in production 
=> Run `bin/rails server --help` for more startup options
Puma starting in single mode...
* Puma version: 8.0.0 ("Into the Arena")
* Ruby version: ruby 4.0.1 (2026-01-13 revision e04267a14b) +YJIT +PRISM [x86_64-linux]
*  Min threads: 3
*  Max threads: 3
*  Environment: production
*          PID: 18
* Listening on http://0.0.0.0:3000
Use Ctrl-C to stop

[lorenzharfst]

Uhhh...

So after days of checking every corner it turns out kamal proxy just was in a bad mood, a kamal proxy reboot fixed the issue.