Skip to content

Update puma service on deployment with up to date service config - #2118

Open
ianheggie-oaf wants to merge 8 commits into
mainfrom
feature/2116-update-puma-service-template
Open

Update puma service on deployment with up to date service config#2118
ianheggie-oaf wants to merge 8 commits into
mainfrom
feature/2116-update-puma-service-template

Conversation

@ianheggie-oaf

@ianheggie-oaf ianheggie-oaf commented Jul 31, 2026

Copy link
Copy Markdown
Member

Description

Update puma service to reflect recent capistrano puma gem

  • Use latest template and fixed template error from gem
  • notify systemd rather than simple type (so it will restart if puma process freezes rather than just dies)
  • Check linger is set but doesn't try and fix (we dont have password less sudo)
  • Always updates systemd service config

Based on (rather than stack as 3 PRs depend directly on it):

Motivation and Context

Fixes service config was not being updated in normal deploy (and puma:enable calls sudo by default which deploy user doesn't have access to):

How Has This Been Tested?

  • Checked affected area manually on my own / staging system

Developed and tested on Ubuntu 24.04.4 LTS x86_64 with ruby installed by mise, mysql 8.0, redis 5:7.0, make 4.3

Deployed using updated capistrano config.

Checked systemd was being alerted to status (each 5 seconds) and that it restarts a process that dies:

deploy@ip-172-31-2-63:~$ sleep 3 ; systemctl --user show planningalerts_puma_production -p WatchdogTimestamp ; date ; echo === ; sleep 3 ; systemctl --user show planningalerts_puma_production -p WatchdogTimestamp ; date ; echo === ; sleep 3 ; systemctl --user show planningalerts_puma_production -p WatchdogTimestamp ; date ; echo === ; sleep 3 ; systemctl --user show planningalerts_puma_production -p WatchdogTimestamp ; date ; echo === ;
WatchdogTimestamp=Fri 2026-07-31 22:31:11 AEST
Fri Jul 31 22:31:14 AEST 2026
===
WatchdogTimestamp=Fri 2026-07-31 22:31:16 AEST
Fri Jul 31 22:31:17 AEST 2026
===
WatchdogTimestamp=Fri 2026-07-31 22:31:16 AEST
Fri Jul 31 22:31:20 AEST 2026
===
WatchdogTimestamp=Fri 2026-07-31 22:31:21 AEST
Fri Jul 31 22:31:23 AEST 2026
===
deploy@ip-172-31-2-63:~$ systemctl --user show planningalerts_puma_production -p MainPID
MainPID=37771
deploy@ip-172-31-2-63:~$ (sleep 10; kill -STOP "$(systemctl --user show planningalerts_puma_production -p MainPID --value)") & \
journalctl --user -u planningalerts_puma_production -f
[1] 64609
Jul 31 21:12:44 ip-172-31-2-63 systemd[593]: Starting Puma HTTP Server for planningalerts (production)...
Jul 31 21:12:54 ip-172-31-2-63 systemd[593]: Started Puma HTTP Server for planningalerts (production).
Jul 31 21:49:22 ip-172-31-2-63 systemd[593]: Reloading Puma HTTP Server for planningalerts (production)...
Jul 31 21:49:22 ip-172-31-2-63 systemd[593]: Reloaded Puma HTTP Server for planningalerts (production).
Jul 31 21:52:08 ip-172-31-2-63 systemd[593]: Reloading Puma HTTP Server for planningalerts (production)...
Jul 31 21:52:52 ip-172-31-2-63 systemd[593]: Reloaded Puma HTTP Server for planningalerts (production).
Jul 31 21:53:40 ip-172-31-2-63 systemd[593]: Reloading Puma HTTP Server for planningalerts (production)...
Jul 31 21:53:57 ip-172-31-2-63 systemd[593]: Reloaded Puma HTTP Server for planningalerts (production).
Jul 31 22:04:15 ip-172-31-2-63 systemd[593]: Reloading Puma HTTP Server for planningalerts (production)...
Jul 31 22:04:37 ip-172-31-2-63 systemd[593]: Reloaded Puma HTTP Server for planningalerts (production).

Jul 31 22:33:36 ip-172-31-2-63 systemd[593]: planningalerts_puma_production.service: Watchdog timeout (limit 10s)!
Jul 31 22:33:36 ip-172-31-2-63 systemd[593]: planningalerts_puma_production.service: Killing process 37771 (ruby) with signal SIGABRT.
Jul 31 22:33:36 ip-172-31-2-63 systemd[593]: planningalerts_puma_production.service: Killing process 60721 (ruby) with signal SIGABRT.
Jul 31 22:33:36 ip-172-31-2-63 systemd[593]: planningalerts_puma_production.service: Killing process 60758 (ruby) with signal SIGABRT.
Jul 31 22:33:36 ip-172-31-2-63 systemd[593]: planningalerts_puma_production.service: Killing process 60788 (skylightd) with signal SIGABRT.
Jul 31 22:33:36 ip-172-31-2-63 systemd[593]: planningalerts_puma_production.service: Killing process 60856 (ruby) with signal SIGABRT.
Jul 31 22:33:37 ip-172-31-2-63 systemd[593]: planningalerts_puma_production.service: Main process exited, code=dumped, status=6/ABRT
Jul 31 22:33:37 ip-172-31-2-63 systemd[593]: planningalerts_puma_production.service: Failed with result 'watchdog'.
Jul 31 22:33:37 ip-172-31-2-63 systemd[593]: planningalerts_puma_production.service: Consumed 22min 12.656s CPU time.
Jul 31 22:33:38 ip-172-31-2-63 systemd[593]: planningalerts_puma_production.service: Scheduled restart job, restart counter is at 1.
Jul 31 22:33:38 ip-172-31-2-63 systemd[593]: Stopped Puma HTTP Server for planningalerts (production).
Jul 31 22:33:38 ip-172-31-2-63 systemd[593]: planningalerts_puma_production.service: Consumed 22min 12.656s CPU time.
Jul 31 22:33:38 ip-172-31-2-63 systemd[593]: Starting Puma HTTP Server for planningalerts (production)...
Jul 31 22:33:48 ip-172-31-2-63 systemd[593]: Started Puma HTTP Server for planningalerts (production).
  • Ran automated tests on my own system
  • Confirmed it passed the GitHub actions tests

Screenshots (if appropriate):

Types of Changes

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality) - makes sure service config is up to date and loaded, and will persist over reboot
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.

@ianheggie-oaf ianheggie-oaf self-assigned this Jul 31, 2026
@ianheggie-oaf
ianheggie-oaf force-pushed the feature/2116-update-puma-service-template branch from 4eb146e to d75daf8 Compare July 31, 2026 12:18
@ianheggie-oaf
ianheggie-oaf marked this pull request as ready for review July 31, 2026 12:35
@ianheggie-oaf
ianheggie-oaf requested a review from a team as a code owner July 31, 2026 12:35

@benrfairless benrfairless left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good change overall, and the testing writeup is exactly right: SIGSTOPping the master to prove the watchdog actually fires is much better evidence than assuming Type=notify works. Moving the memory and timeout tuning out of the template and into puma_service_unit_props is the right direction too.

Requesting changes on three things:

  1. MemoryMax has gone from 1680M back to 1500M while the comment above it still says 1,680. Looks like an accidental revert of 311c52d.
  2. puma_systemctl_user is being used as a username, but the gem treats it as a :system/:user mode flag. It works by luck today, and the coupling with check_lingering isn't visible from either file.
  3. The template file is now byte-identical to the gem's, so it'd be better deleted than kept in sync.

Two more worth a look before merge: puma:install runs after puma:smart_restart, so Type= and WatchdogSec= won't actually take effect until the next reboot or manual restart, and check_lingering raises after the deploy has already published and restarted.

One framing note that might save some time. A few of the new settings aren't doing what the description implies. Type=<%= service_unit_type %> is an unchanged context line in the template diff and the gem's helper already returns "notify" for Puma 6.1+, puma_systemd_watchdog_sec matches the gem default of 10, and the hardcoded log paths equal what the gem derives from shared_path. The real payload here is puma_enable_lingering, false, the puma:install hook, and puma_service_unit_props. The simple to notify switch comes from finally regenerating a unit file that had been frozen at 6.0.0.beta.1, not from the new settings.

Detail is in the three file comments. Nothing security-related found, and I'm not raising a test coverage gap given there's no practical harness for Capistrano hooks in this repo.

Comment thread config/deploy.rb
Comment thread config/deploy/templates/puma.service.erb Outdated
Comment thread lib/capistrano/tasks/puma_check_lingering.rake
Base automatically changed from feature/2116-support-ed25519-ssh-keys to main August 3, 2026 12:18
@ianheggie-oaf
ianheggie-oaf force-pushed the feature/2116-update-puma-service-template branch from 4dda37d to 2b5d608 Compare August 3, 2026 12:18
Comment thread lib/capistrano/tasks/puma_check_lingering.rake Outdated
@ianheggie-oaf
ianheggie-oaf force-pushed the feature/2116-update-puma-service-template branch from 9c0bb86 to a7b7fab Compare August 6, 2026 17:43
@ianheggie-oaf

Copy link
Copy Markdown
Member Author

Requesting changes on three things

@benrfairless - I have updated these, rebased and its ready for reviewing again

Comment thread lib/capistrano/tasks/puma_check_lingering.rake Outdated
@ianheggie-oaf
ianheggie-oaf marked this pull request as draft August 7, 2026 06:02
@ianheggie-oaf
ianheggie-oaf force-pushed the feature/2116-update-puma-service-template branch from 2030676 to 0c68e64 Compare August 7, 2026 06:35
@ianheggie-oaf
ianheggie-oaf marked this pull request as ready for review August 7, 2026 06:41
@ianheggie-oaf

Copy link
Copy Markdown
Member Author

@benrfairless back to you for review pls

@ianheggie-oaf
ianheggie-oaf force-pushed the feature/2116-update-puma-service-template branch from 0c68e64 to 83c08e0 Compare August 7, 2026 10:18
@ianheggie-oaf
ianheggie-oaf dismissed benrfairless’s stale review August 7, 2026 10:20

Requested changes have been done, but was unable to mark it resolved in github

@ianheggie-oaf
ianheggie-oaf force-pushed the feature/2116-update-puma-service-template branch from 83c08e0 to 7362b08 Compare August 7, 2026 10:45
@ianheggie-oaf
ianheggie-oaf force-pushed the feature/2116-update-puma-service-template branch from ee23ee7 to 62f0c70 Compare August 7, 2026 12:41
namespace :puma do
desc "Check lingering is already set"
task :check_lingering do
on roles(fetch(:puma_role)) do |host|

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The puma:check_lingering task defaults to the :web role, but production servers use the :app role, causing the task to silently skip execution without performing its check.
Severity: MEDIUM

Suggested Fix

Explicitly set the :puma_role in the deployment configuration to match the role used by the application servers. For example, add set :puma_role, :app to your deploy configuration file.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: lib/capistrano/tasks/puma_check_lingering.rake#L9

Potential issue: The `puma:check_lingering` task runs on servers with the role defined
by `:puma_role`. This variable is not set in the configuration, causing it to fall back
to the `capistrano3-puma` gem's default value of `:web`. However, the production servers
are configured with the `:app` role. As a result, the `on roles(fetch(:puma_role))`
command will not match any servers in the production environment. Capistrano silently
skips execution when no hosts match, meaning the linger check is never actually
performed. The task will complete without any errors or warnings, defeating its purpose
of verifying the server configuration.

* Fixed template error from gem
* Check linger is set but doesn't try and fix (we dont have password less sudo)
…used

This is a jump up from max 67.6% with MemoryMax=1500M for last 2 months
@ianheggie-oaf
ianheggie-oaf force-pushed the feature/2116-update-puma-service-template branch from 62f0c70 to 59b9f96 Compare August 7, 2026 12:56
@sonarqubecloud

sonarqubecloud Bot commented Aug 7, 2026

Copy link
Copy Markdown

@benrfairless benrfairless left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comments from sentry that need review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants