Data migration init container is not crash-resumable (can strand data)

[sklarsa]

Summary

The Helm-v1 data-migration init container (charts/questdb/templates/init_db_migrations_configmap.yaml, migrate_to_helm_v1.sh) is not crash-resumable. If the init container is killed mid-migration (OOMKill, eviction, node drain — all of which cause Kubernetes to re-run init containers from scratch), a re-run can silently skip the migration and strand the original db/ data.

This is a pre-existing issue (the original bash script had the same flaw); it is not introduced by the alpine/POSIX swap in #247. Filing as a follow-up so it can be fixed on its own with a proper redesign.

Root cause

The script uses a single signal — "is there a tables.d.* at the volume root?" — to decide whether to migrate, but the migration itself destroys that signal partway through:

1. The old db/ (if present) is moved aside to a timestamped db_helm_migration_1_tmp_<ts> dir.
2. The for item in ./* loop moves everything at the root — including the marker — into the new db/.
3. A separate, later step does mv "db/$TEMP_DIR" "db/db".

If the container dies between (2) and (3), the re-run sees no tables.d.* at the root → prints Nothing to move. → exits 0 → the old data is stranded at db/db_helm_migration_1_tmp_<ts>/ and never relocated to db/db. QuestDB (data root = db/) then starts as if the database were empty.

Two things conspire:

• (a) the go/no-go decision depends on state the migration mutates, and
• (b) the temp dir name is timestamped, so a re-run can't recognize a leftover from a prior run.

Reproduction

Simulate the on-disk state at the crash point (marker already moved into db/, temp dir not yet relocated) and re-run the script:

db/db_helm_migration_1_tmp_20260101000000/mytable/data.d   <- original table
db/tables.d.0

Re-running prints File 'tables.d.*' not found. Nothing to move. (exit 0) and leaves mytable stranded under the timestamped temp dir instead of at db/mytable.

Proposed fix

Three pieces (details in PR):

1. Durable completion sentinel — written last, checked first, so the decision no longer depends on the marker surviving:

   [ -e db/.helm_v1_migration_done ] && { echo "Already migrated."; exit 0; }

2. Deterministic (non-timestamped) backup name so a leftover is recognizable across runs:

   TEMP_DIR="db_helm_migration_1_bak"

3. Crash-recovery step at the top that finishes an interrupted run before anything else (leftover may be at the root or already inside db/):

   for leftover in ./"$TEMP_DIR" ./db/"$TEMP_DIR"; do
     [ -d "$leftover" ] || continue
     mkdir -p db
     [ -e db/db ] && { echo "db/db already exists; cannot auto-resume" >&2; exit 1; }
     mv "$leftover" db/db
   done

Alternative (bulletproof but heavier): build the new layout in a staging dir, touch the sentinel inside it, then a single atomic mv staging db. Rejected as the default because it transiently needs ~2× the data size on the PVC.

Acceptance criteria

• Migration is idempotent and resumable: interrupting the init container at any point and re-running converges to the correct db/ layout with no stranded data.
• Existing behavior preserved for the happy paths (fresh install, old→new migration, pre-existing db/ → db/db, hidden files, broken symlinks, files with spaces).
• Crash-scenario repros added/verified under a POSIX shell (dash/busybox); shellcheck -s sh clean; helm lint passes.

Related: #247