fix(chart): gate the maintenance cronjob on actual GLPI/DB readiness - #196
Open
danielqb wants to merge 1 commit into
Open
fix(chart): gate the maintenance cronjob on actual GLPI/DB readiness#196danielqb wants to merge 1 commit into
danielqb wants to merge 1 commit into
Conversation
… not just file existence ## Problem The maintenance CronJob (front/cron.php, schedule every N minutes) is a normal resource created in the same Sync phase as nginx/php-fpm, without waiting for the post-install glpi-db-install/glpi-db-configure/ glpi-cache-configure Jobs (which run as Helm hooks, post-install hook-weight 10/20/30) to finish. On a fresh install, this lets the first scheduled run(s) fire before GLPI is actually usable, failing with "Unable to load the GLPI configuration from the database" - self-heals via Kubernetes' OnFailure retry, but noisy (BackOff events, wasted retries) on every fresh install. ## First attempt (insufficient - documented here for the record) Initially added an initContainer that waits for `/etc/glpi/config/config_db.php` to exist before running cron.php. Verified empirically on a live kind cluster that this REDUCES but does NOT eliminate the race: `glpi-db-install.sh` runs `db:install --reconfigure`, which writes that same config_db.php file as soon as the *schema* is created (hook-weight 10) - well before db:configure/ cache:configure (weight 20/30) finish. Across 4 live re-installs with just the file-existence check, saw BackOff on roughly half of them, confirmed via captured pod state: the initContainer completed (file existed) but the "base" container still failed immediately with "Unable to load the GLPI configuration from the database". ## Actual fix Replaced the file-existence check with a loop of the exact read-only command glpi-db-install.sh itself already uses to decide whether the schema is fully installed: `php bin/console database:check_schema_integrity` (exit 0 once GLPI is genuinely ready to serve requests). This requires switching the initContainer from busybox to the real php-fpm image (needed for the php binary + GLPI codebase) and giving it the same glpi-config/glpi-secret/MARIADB_PASSWORD env wiring as the main container, since the check needs a working DB connection to run. ## Testing - helm lint --strict: 0 failures - helm template: initContainer renders with the php-fpm image and the correct env/secretKeyRef wiring - Live kind cluster (kindest/node:v1.35.0), schedule accelerated to every minute to stress-test the install-time race: 5 consecutive fresh install cycles (helm uninstall + delete pvc + fresh helm install each time), zero BackOff/failed cronjob events across all 5 - compared to a ~50% failure rate observed with the file-existence-only check across 4 separate live runs beforehand - Confirmed GLPI still serves its real login page (HTTP 200) after each install
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The maintenance CronJob (front/cron.php, schedule every N minutes) is a
normal resource created in the same Sync phase as nginx/php-fpm, without
waiting for the post-install glpi-db-install/glpi-db-configure/
glpi-cache-configure Jobs (which run as Helm hooks, post-install
hook-weight 10/20/30) to finish. On a fresh install, this lets the first
scheduled run(s) fire before GLPI is actually usable, failing with
"Unable to load the GLPI configuration from the database" - self-heals
via Kubernetes' OnFailure retry, but noisy (BackOff events, wasted
retries) on every fresh install.
First attempt (insufficient - documented here for the record)
Initially added an initContainer that waits for
/etc/glpi/config/config_db.phpto exist before running cron.php.Verified empirically on a live kind cluster that this REDUCES but does
NOT eliminate the race:
glpi-db-install.shrunsdb:install --reconfigure, which writes that same config_db.php file as soon as theschema is created (hook-weight 10) - well before db:configure/
cache:configure (weight 20/30) finish. Across 4 live re-installs with
just the file-existence check, saw BackOff on roughly half of them,
confirmed via captured pod state: the initContainer completed (file
existed) but the "base" container still failed immediately with "Unable
to load the GLPI configuration from the database".
Actual fix
Replaced the file-existence check with a loop of the exact read-only
command glpi-db-install.sh itself already uses to decide whether the
schema is fully installed:
php bin/console database:check_schema_integrity(exit 0 once GLPI is genuinely ready to serve requests). This requires
switching the initContainer from busybox to the real php-fpm image
(needed for the php binary + GLPI codebase) and giving it the same
glpi-config/glpi-secret/MARIADB_PASSWORD env wiring as the main
container, since the check needs a working DB connection to run.
Testing
correct env/secretKeyRef wiring
every minute to stress-test the install-time race: 5 consecutive
fresh install cycles (helm uninstall + delete pvc + fresh helm
install each time), zero BackOff/failed cronjob events across all 5 -
compared to a ~50% failure rate observed with the file-existence-only
check across 4 separate live runs beforehand
install