feat(update): 0.3.5.1 → 0.4.0 in-place update flow

[helix-nine]

0.3.5.1 → 0.4.0 in-place update

Delivers the update path from StartOS 0.3.5.1 to 0.4.0 without the flaky, file-count-scaling full-rootfs rsync the 0.3.5.1 updater does today. We ride 0.3.5.1's existing (unchangeable) updater, but serve it a minimal payload — a single 0.4.0 base squashfs + kernel/initramfs + boot-config — and do the rootfs rewire in the 0.4.0 initramfs at next boot, before / is mounted.

Status: ready for review (x86_64 / aarch64). Raspberry Pi has no in-place path — it requires a USB/SD reflash (#3443). End-to-end validated on a VM via the real production flow (see the validation comment); a physical-hardware pass is still advisable.

The immovable channel (0.3.5.1, core/startos/src/update/mod.rs @ v0.3.5.1)

do_update is fixed Rust on already-shipped boxes; we can't change it. It:

1. GET {marketplace}/eos/v0/latest → version;
2. rsyncs (rsync daemon proto) {host}::{version}/{platform}/ → /media/embassy/next/ — this per-file rsync of a whole rootfs is the flaky step we're eliminating;
3. force-copies the running host's fstab / machine-id / ssh host keys into next/etc;
4. sync_boot: rsync next/boot → /boot, then chroot next update-grub2 (skipped on raspberrypi);
5. touch config/upgrade → reboot.

The serving side is frozen — and needs no change

The endpoint a 0.3.5.1 box hits is the old Haskell registry (immutable). Its pipeline (build/registry/{upload.cgi,resyncRsyncRegistry} @ v0.3.5.1) accepts an uploaded .squashfs, loop-mounts it (mount $filename /srv/rsync/$version/$arch), and rsync-serves its contents as module [$version]. So the box rsyncs the tree inside whatever squashfs we publish into next/. We conform by crafting the uploaded OTA squashfs — no registry change. Publishing uses the existing OTA upload flow, just with a differently-structured image for the 0.4.0 release.

Design (3 stages)

1. Apply-time (rides 0.3.5.1's do_update): the OTA squashfs we publish for 0.4.0/<platform> contains, as its tree, the minimal scaffold — images/<hash>.rootfs (the real 0.4.0 base squashfs, one nested blob), boot/ (0.4.0 kernel + initramfs), a .startos-migration sentinel, minimal etc/, and our own usr/sbin/update-grub2. rsync transfers ~a dozen files + one blob instead of ~40k files (the flakiness win). Then 0.3.5.1 execs our update-grub2 in the chroot, which deterministically writes a boot entry (root=PARTUUID=<embassyfs>, subvol-stripped) pointing at the 0.4.0 kernel+initramfs — no reliance on grub-mkconfig's probing (which the image build already has to sed-correct). Everything expensive happens pre-reboot; a failure safely leaves the box on 0.3.5.1.
2. Boot-time (0.4.0 initramfs startos-initramfs-module): the else-branch detects the sentinel, carries the box's real identity out of next/etc into config/overlay/etc, installs the shipped squashfs as current.rootfs (atomic images/ publish), and drops the disposable old roots. Idempotent, crash-safe.
3. Post-boot (existing Rust version/ chain, v0_3_6_alpha_0 → v0_4_0_beta_N): replays the db/package/volume migration against /media/startos/data, loading the offline migration-images baked into the base squashfs. The data partition is the only preserved state; neither stage 1 nor 2 touches it.

No rollback (per @dr-bonez — emergency recovery is a thumb-drive reflash), so the boot swap is one-way; we optimize for reliability of the forward path.

Scope checklist

• initramfs migration else-branch — reproduced boot-failure bug fixed; sentinel-gated; identity from next/etc; atomic images publish; idempotent. Shell-sim verified.
• Migration-payload assembler (assemble-migration-payload.sh) — 0.3.5.1 base squashfs from the release ISO + nested 0.4.0 base image + boot/ + sentinel + our update-grub2.
• Deterministic boot swap (x86/aarch64) (migration-update-grub) — writes grub.cfg for the 0.4.0 kernel, reuses the box's real root=, and emits boot=startos (the param that selects the 0.4.0 initramfs module — bug caught + fixed).
• Build target start-os-migration-squashfs.
• CI: dedicated startos-migration-image.yaml workflow — repackages an ISO build's <platform>.squashfs artifact onto the 0.3.5.1 release base and uploads <platform>.migration.squashfs (ops publishes it to the frozen registry). Pure repackaging, so it runs on x86_64 for all platforms.
• Changelog entry.
• Raspberry Pi — no in-place migration; Pi users reflash (USB/SD) instead. Its boot model (firmware → U-Boot → GRUB across a 4-partition GPT) can't be reconstructed on a migrated 0.3.5.1 Pi's 2-partition MBR SD without partition surgery that isn't worth the risk. Decided; 0.3.5.1 → 0.4.0 update: Raspberry Pi path (deferred from #3399) #3443 closed.
• Space: can't preflight before the frozen updater's rsync (no hook) — documented limitation; payload is one blob so peak is bounded; a near-full box safely stays on 0.3.5.1 → reflash.
• User guide (update-040.md) → OTA, deferred until hardware-validated.
• E2E validation — real 0.3.5.1 box + legacy registry; out of sandbox.

Testing boundary

End-to-end validated on a VM via the real server.update production flow — see the validation comment below. A live, set-up 0.3.5.1 box was updated through its own do_update (rsync payload → next/, sync_boot → chroot our update-grub2, reboot), and the serial console confirms it booted 0.4.0-beta.10 from the migrated squashfs with the data pool + credentials preserved. Still not covered: physical hardware, UEFI/Secure-Boot variants, and Raspberry Pi (#3443) — so a hardware pass is still advisable before flipping the user guide to OTA.

[helix-nine]

✅ End-to-end validated on a VM via the real production update flow

Replicated the actual 0.3.5.1 → 0.4.0 path as honestly as possible — the real server.update RPC running on a live, set-up 0.3.5.1 box, not an offline hack.

Setup

• Built the 0.4.0 image from this branch (so it carries the initramfs fix): startos-0.4.0-beta.10…squashfs.
• Installed + fully set up a real StartOS 0.3.5.1 VM from the release ISO (RPC install → JWE setup.execute; encrypted data pool on the data partition; master password set).
• Stood up a registry that mirrors the real one: rsyncd module [0.4.0] serving the payload's contents + an eos/v0/latest endpoint returning 0.4.0 — the exact transport do_update expects.

The flow (all real 0.3.5.1 code)

• POST server.update {marketplace-url} on the running box → do_update fetched eos/v0/latest, rsynced the payload into /media/embassy/next/, copied identity, ran sync_boot → chroot next update-grub2 (our script) → touch config/upgrade. Box reported updated: true.
• server.restart → reboot.

Serial-console proof of the migrated boot:

Linux version 7.0.12+deb13-amd64 …
Command line: … root=UUID=45e44af7-… boot=startos console=ttyS0,115200n8 console=tty0
… BTRFS device label rootfs … /dev/vda3
… loop0: detected capacity change from 0 to 3238584        # our squashfs mounted as current.rootfs
… systemd 257.13-1~deb13u1 running in system mode
StartOS v0.4.0-beta.10 [x86_64] on start.local (ttyS0)     # 0.4.0 login prompt
… EMBASSY_…_main / …_package-data                          # preserved encrypted data pool, mounted

This exercises every piece of the device side:

• update-grub — booted the 0.4.0 kernel, reused the box's real root=UUID, swapped boot=embassy→**boot=startos** (the fix that makes the initramfs module run), added the console.
• initramfs else-branch — installed the shipped squashfs as current.rootfs and booted the overlay (the reproduced old.squashfs boot-failure is gone; the box boots).
• Data partition preserved — 0.4.0 opened the encrypted LVM pool from 0.3.5.1 untouched.
• Credentials preserved — logging into the 0.4.0 box with the password set during 0.3.5.1 setup succeeds.

No kernel panics, no failed mounts, no not available to boot. Clean migration.

Still out of scope for this validation: physical hardware, UEFI/Secure-Boot variants, and Raspberry Pi (#3443). This was QEMU/KVM (SeaBIOS, single disk). The core logic is proven; a hardware pass is still wise before flipping the user-facing docs to OTA.