fix: make helm-v1 data migration crash-resumable

[sklarsa]

Summary

Makes the Helm-v1 data-migration init container (charts/questdb/templates/init_db_migrations_configmap.yaml, migrate_to_helm_v1.sh) idempotent and crash-resumable. Closes #260.

Stacking note: this was written on top of #247 (POSIX/alpine swap). #247 has since merged, so this targets master directly — the auto-retarget the issue anticipated. It touches only the migration ConfigMap.

The bug (#260)

The migration used a single go/no-go signal — "is there a tables.d.* marker at the volume root?" — but the migration itself destroys that signal partway through: the for loop moves the marker into db/ before the old db/ backup is relocated to db/db. If the init container is killed in between (OOMKill / eviction / node-drain → Kubernetes re-runs init containers from scratch), the re-run sees no marker → prints Nothing to move. → exits 0 → the original db/ data is stranded under a timestamped temp dir and never relocated. The timestamped temp name also made leftovers unrecognisable across runs.

The fix

Redesigned so that interrupting the script at any point and re-running it to completion converges to the correct db/ layout, with no stranded data and no leftover temp/backup dirs (no ~2× copy — only renames):

1. Durable completion sentinel db/.helm_v1_migration_done — written last, checked first, so the decision no longer depends on the marker surviving.
2. Deterministic backup name db_helm_migration_1_bak (no timestamp) — a leftover is recognisable and finishable across runs.
3. In-progress marker db/.helm_v1_migration_started — committed once db/ is claimed as the work dir, so a re-run never mistakes a half-filled work dir for a fresh pre-existing db/ and re-nests it (this is the subtle Case-A resumability trap).
4. Go/no-go now resumes on a leftover backup or an in-progress marker too, and a top-of-script recovery finishes a backup already sitting inside db/.
5. Kept the refuse-if-db/db-exists safety (don't silently clobber) and everything the old script already got right: ls -d empty-dir marker detection (with its # shellcheck disable=SC2086), null-glob + broken-symlink guards on both loops, quoting for names with spaces, hidden-file globs, and db/db nesting for a pre-existing db/.

Verification

Extracted the embedded script and drove it under /bin/dash. Crash injection wraps mv/mkdir/touch/rm to run the real op then exit 99 after the Nth mutation, then the unmodified script runs to completion and the final db/ layout is diffed against the fully-migrated expected layout (with checks for no stranded data, no leftover *_bak, and root containing only db/).

Behaviour-preservation (all exit 0, correct layout): fresh/empty volume, old→new (regular + hidden + subdir + spaces), pre-existing db/ → db/db, broken visible & hidden symlinks, empty-directory marker, and idempotent re-run (Already migrated.).

Crash-injection matrix — every point converged:

Scenario	Injection points	Result
Case B (pre-existing db/ + full mix) — single injection after op N	N = 1‥13 (every op)	13/13 converged ✓
Case B — double injection (N then M during resume)	13 × 14 grid	182/182 converged ✓
Case B — randomized multi-injection fuzz (≤4 crashes/run)	400 runs	400/400 converged ✓
Case A (old→new, no pre-existing db/) — single injection after op N	N = 1‥10 (every op)	10/10 converged ✓
Case A — randomized multi-injection fuzz (≤4 crashes/run)	500 runs	500/500 converged ✓
#260 repro — leftover backup inside db/	resume	✓ data → db/db, not stranded
#260 repro — leftover backup at root	resume	✓ data → db/db, root files → db/

Also: shellcheck -s sh clean (only the intentional, already-suppressed SC2086 on the marker glob), helm lint charts/questdb/ passes, and helm template … | grep migrate_to_helm renders; the helm-rendered script is byte-identical to the source and re-passes shellcheck.

🤖 Generated with Claude Code

[sklarsa]

Independent verification on kind (real alpine:3.23 busybox + QuestDB)

Verified this on a local kind cluster with the actual chart (the automated crash-injection matrix above runs under dash; this adds real busybox + a running QuestDB, which that matrix can't cover). All green:

Test	Init log	Pod	Check
Fresh install (empty PVC)	Nothing to move.	1/1 Running	no sentinel written (exits before migrating); QuestDB creates its own db/
Old→new migration	Migration complete!	1/1 Running	all files migrated into db/; .helm_v1_migration_done written, .helm_v1_migration_started cleaned up; QuestDB starts fine with the sentinel dotfiles present
Idempotent re-run (delete pod → init re-runs)	Already migrated.	1/1 Running	sentinel short-circuits correctly
Crash-resume (seed interrupted state: old data in leftover db_helm_migration_1_bak, root items unmoved, .helm_v1_migration_started present)	Migration complete!	1/1 Running	old data recovered to db/db/ with content intact, root items moved into db/, no leftover _bak, sentinel written

shellcheck -s sh clean and helm lint pass reproduced independently. The key thing the dash matrix couldn't check — that QuestDB tolerates the new .helm_v1_migration_done / .helm_v1_migration_started dotfiles in db/ — is confirmed: it starts 1/1 Running in every case.

LGTM.