Skip to content

fix: bound ExecReload timeout and pre-pull images before the reload handler - #3

Open
mbilan1 wants to merge 1 commit into
mainfrom
fix/reload-timeout-prepull
Open

fix: bound ExecReload timeout and pre-pull images before the reload handler#3
mbilan1 wants to merge 1 commit into
mainfrom
fix/reload-timeout-prepull

Conversation

@mbilan1

@mbilan1 mbilan1 commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Problem

systemctl reload tutor runs tutor local start -d (= docker compose up -d) under systemd's default 90s TimeoutStartSec — the unit template sets no Timeout* directive. On a new image tag that reload is a multi-GB inline pull + full container recreate (measured 7m19s on globalcampus staging). systemd kills the control process at 90s, ansible fails with Unable to reload service tutor, the orphaned compose child finishes the deploy unsupervised minutes later, and the retry goes green off the cache the orphan warmed — CI success is decoupled from deploy success.

Numbers (globalcampus-tutor): 0/155 deploys failed before image tags became :${CI_PIPELINE_ID} (2025-03-24), 61/118 (52%) after; 9/15 (60%) in Aug 2026. Same signature confirmed on iea (16), ifb (6), limes (4), ueg (3), polito (1), exasol production (1, 2026-08-24). Two concurrent orphaned compose runs collided on 2026-08-24 and produced a 502 window on globalcampus staging.

Change

  • TimeoutStartSec={{ tutor_reload_timeout }} in the unit, default 900 — bounded on purpose, never infinity, so genuine failures (e.g. the 2026-08-18 Harbor unauthorized) still fail red.
  • Reload systemd daemon handler defined before Reload tutor (handlers run in definition order; state: reloaded alone does not re-read an edited unit).
  • Unit template task + image pull task additionally tagged tutor_config so the fix self-delivers through the first CI playbook (-t tutor_config) — the only playbook a currently-failing host ever reaches — and the pull runs in the same play that writes the new tag and fires the handler.
  • update.yml imported after config.yml: the pull resolves image refs from the on-host config, so pulling first fetched the previous deploy's tags (re-tagging alone would fix nothing).

Compatibility / rollout

  • Consumers pinning 5451a04 or tag v1.1.0 (created on that same commit) are untouched until they bump.
  • ⚠️ globalcampus-tutor, c40-tutor, ueg-tutor, iea-tutor track version: main — they pick this up on their next deploy once merged. Pin-to-tag MRs are being opened in those repos; merge them first, then this, then bump tenants one at a time (staging branch first).
  • Interim mitigation already live on elstaging/elearning.gchumanrights.org and iea-staging.abzt.de: drop-in /etc/systemd/system/tutor.service.d/50-reload-timeout.conf (TimeoutStartSec=900). The rendered unit supersedes it once this lands; the drop-in can then be removed.
  • ansible-lint: 0 failures / 0 warnings (production profile).

Ref: Taiga trabucchis-internal-stuff US 1459 (extends it — legacy ansible deploy path).

🤖 Generated with Claude Code

…andler

systemd's DefaultTimeoutStartSec (90s) governs ExecReload, which runs
'tutor local start -d' = docker compose up -d: on a new image tag that is
a multi-GB inline pull + full container recreate (measured 7m19s). The
control process is killed at 90s, ansible reports 'Unable to reload
service tutor', the orphaned compose child finishes the deploy
unsupervised, and the next retry goes green off the cache it warmed.
61 of 118 globalcampus deploys failed this way since image tags became
:${CI_PIPELINE_ID}; iea/ifb/limes/ueg/polito/exasol share the pattern.

- TimeoutStartSec={{ tutor_reload_timeout }} (default 900, bounded so
  genuine failures such as registry auth errors still fail red)
- 'Reload systemd daemon' handler defined before 'Reload tutor' so an
  edited unit is actually re-read before the reload
- unit template + image pull also tagged tutor_config so the fix
  self-delivers through the first CI playbook, and the pull runs in the
  same play that writes the new tag and fires the handler
- update.yml imported after config.yml: the pull resolves tags from the
  on-host config, so pulling first fetched the previous deploy's tags

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant