Deferred from #39 item 1, per that issue's acceptance criteria ("fixed or explicitly deferred with a stated production caveat in docs/persistent-volumes.md") and §8.3 of the hardening plan.
The gap
vm.Manager.Stop signals the Firecracker VMM process, so the microVM is destroyed without any guest-side shutdown and guest page-cache writes never reach the virtio-blk device. ext4 journal replay keeps the filesystem consistent; unflushed application data is lost.
This is pre-existing Stop() behavior, but the local-volume slice makes it load-bearing: the offline resize path is preflight → deleteService(prev) → Start, so every quota change and every service update is a power-pull for a persistent volume.
SendCtrlAltDel is not a sufficient fix — it is unsupported on aarch64, and the deployment example runs arm64 nodes.
First deliverable: fc-init becomes a resident PID-1 supervisor
This is the largest part of the work and the reason the item was deferred rather than scheduled. Today execService calls applyUserSpec — which drops to the target uid/gid — and then syscall.Execs the application, replacing the fc-init process image entirely. After that point fc-init does not exist: there is no process left to listen on vsock, no parent to signal or reap the application, and no root credentials with which to unmount.
So before a single line of the shutdown protocol can be written, fc-init has to be redesigned to stay resident as PID 1, retain root, fork/exec the application into its dropped-credential child, reap orphans, and forward signals. That is a rewrite of the guest entry path, not an addition to it, and it sits underneath every existing workload — volume-bearing or not.
Acceptance criteria for this deliverable on its own:
Second deliverable: the vsock quiesce channel
- add a virtio-vsock device to the Firecracker machine config in
writeVMConfig, with a host-side Unix socket in the existing VM directory;
- the resident
fc-init listens on a fixed guest port for a versioned request/response protocol. The only initial verb is quiesce: signal the application SIGTERM, wait a bounded interval for exit and reap it, sync, unmount every volume mount in reverse mount order, and reply with per-mount success or failure;
Manager.Stop gains a quiesce phase before signalling the VMM: connect, send quiesce, wait up to a configured timeout, then proceed to the existing ownership-validated SIGTERM/SIGKILL sequence regardless of outcome;- every failure mode — no vsock device, connect refused, timeout, protocol version mismatch, unmount failure — degrades to exactly today's behavior and is reported on the volume status, so an old
fc-init in an existing guest image keeps working;
- the resize path additionally treats a failed quiesce as a reason to abandon the resize while the VM is still live, rather than proceeding to a power-pull of a volume it is about to shrink.
Validation
This is the one item in the #39 set that cannot be validated without live Firecracker on both architectures, and it ships a new guest binary requiring a guest-image rollout on each. It should follow the local validation lab in #42.
Related: #19 (epic), #39.
Deferred from #39 item 1, per that issue's acceptance criteria ("fixed or explicitly deferred with a stated production caveat in
docs/persistent-volumes.md") and §8.3 of the hardening plan.The gap
vm.Manager.Stopsignals the Firecracker VMM process, so the microVM is destroyed without any guest-side shutdown and guest page-cache writes never reach the virtio-blk device. ext4 journal replay keeps the filesystem consistent; unflushed application data is lost.This is pre-existing
Stop()behavior, but the local-volume slice makes it load-bearing: the offline resize path ispreflight→deleteService(prev)→Start, so every quota change and every service update is a power-pull for a persistent volume.SendCtrlAltDelis not a sufficient fix — it is unsupported on aarch64, and the deployment example runs arm64 nodes.First deliverable: fc-init becomes a resident PID-1 supervisor
This is the largest part of the work and the reason the item was deferred rather than scheduled. Today
execServicecallsapplyUserSpec— which drops to the target uid/gid — and thensyscall.Execs the application, replacing thefc-initprocess image entirely. After that pointfc-initdoes not exist: there is no process left to listen on vsock, no parent to signal or reap the application, and no root credentials with which to unmount.So before a single line of the shutdown protocol can be written,
fc-inithas to be redesigned to stay resident as PID 1, retain root,fork/execthe application into its dropped-credential child, reap orphans, and forward signals. That is a rewrite of the guest entry path, not an addition to it, and it sits underneath every existing workload — volume-bearing or not.Acceptance criteria for this deliverable on its own:
Second deliverable: the vsock quiesce channel
writeVMConfig, with a host-side Unix socket in the existing VM directory;fc-initlistens on a fixed guest port for a versioned request/response protocol. The only initial verb isquiesce: signal the application SIGTERM, wait a bounded interval for exit and reap it,sync, unmount every volume mount in reverse mount order, and reply with per-mount success or failure;Manager.Stopgains a quiesce phase before signalling the VMM: connect, sendquiesce, wait up to a configured timeout, then proceed to the existing ownership-validated SIGTERM/SIGKILL sequence regardless of outcome;fc-initin an existing guest image keeps working;Validation
This is the one item in the #39 set that cannot be validated without live Firecracker on both architectures, and it ships a new guest binary requiring a guest-image rollout on each. It should follow the local validation lab in #42.
Related: #19 (epic), #39.