fix: bound ExecReload timeout and pre-pull images before the reload handler - #3
Open
mbilan1 wants to merge 1 commit into
Open
fix: bound ExecReload timeout and pre-pull images before the reload handler#3mbilan1 wants to merge 1 commit into
mbilan1 wants to merge 1 commit into
Conversation
…andler
systemd's DefaultTimeoutStartSec (90s) governs ExecReload, which runs
'tutor local start -d' = docker compose up -d: on a new image tag that is
a multi-GB inline pull + full container recreate (measured 7m19s). The
control process is killed at 90s, ansible reports 'Unable to reload
service tutor', the orphaned compose child finishes the deploy
unsupervised, and the next retry goes green off the cache it warmed.
61 of 118 globalcampus deploys failed this way since image tags became
:${CI_PIPELINE_ID}; iea/ifb/limes/ueg/polito/exasol share the pattern.
- TimeoutStartSec={{ tutor_reload_timeout }} (default 900, bounded so
genuine failures such as registry auth errors still fail red)
- 'Reload systemd daemon' handler defined before 'Reload tutor' so an
edited unit is actually re-read before the reload
- unit template + image pull also tagged tutor_config so the fix
self-delivers through the first CI playbook, and the pull runs in the
same play that writes the new tag and fires the handler
- update.yml imported after config.yml: the pull resolves tags from the
on-host config, so pulling first fetched the previous deploy's tags
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
systemctl reload tutorrunstutor local start -d(=docker compose up -d) under systemd's default 90sTimeoutStartSec— the unit template sets noTimeout*directive. On a new image tag that reload is a multi-GB inline pull + full container recreate (measured 7m19s on globalcampus staging). systemd kills the control process at 90s, ansible fails withUnable to reload service tutor, the orphaned compose child finishes the deploy unsupervised minutes later, and the retry goes green off the cache the orphan warmed — CI success is decoupled from deploy success.Numbers (globalcampus-tutor): 0/155 deploys failed before image tags became
:${CI_PIPELINE_ID}(2025-03-24), 61/118 (52%) after; 9/15 (60%) in Aug 2026. Same signature confirmed on iea (16), ifb (6), limes (4), ueg (3), polito (1), exasol production (1, 2026-08-24). Two concurrent orphaned compose runs collided on 2026-08-24 and produced a 502 window on globalcampus staging.Change
TimeoutStartSec={{ tutor_reload_timeout }}in the unit, default 900 — bounded on purpose, neverinfinity, so genuine failures (e.g. the 2026-08-18 Harborunauthorized) still fail red.Reload systemd daemonhandler defined beforeReload tutor(handlers run in definition order;state: reloadedalone does not re-read an edited unit).tutor_configso the fix self-delivers through the first CI playbook (-t tutor_config) — the only playbook a currently-failing host ever reaches — and the pull runs in the same play that writes the new tag and fires the handler.update.ymlimported afterconfig.yml: the pull resolves image refs from the on-host config, so pulling first fetched the previous deploy's tags (re-tagging alone would fix nothing).Compatibility / rollout
5451a04or tagv1.1.0(created on that same commit) are untouched until they bump.globalcampus-tutor,c40-tutor,ueg-tutor,iea-tutortrackversion: main— they pick this up on their next deploy once merged. Pin-to-tag MRs are being opened in those repos; merge them first, then this, then bump tenants one at a time (staging branch first)./etc/systemd/system/tutor.service.d/50-reload-timeout.conf(TimeoutStartSec=900). The rendered unit supersedes it once this lands; the drop-in can then be removed.ansible-lint: 0 failures / 0 warnings (production profile).Ref: Taiga
trabucchis-internal-stuffUS 1459 (extends it — legacy ansible deploy path).🤖 Generated with Claude Code