perf(storage): ext4 metadata volume for fsync-hot boltdb state (ABX-496)

[AprilNEA]

Summary

Implements internal-docs/plans/ext4-metadata-volume.md (committed in this PR, incl. measured results in §9): a 2 GiB sparse docker-meta.img attached as vdc, formatted guest-side as journaled ext4 (+fast_commit), carrying exactly the five profiled fsync-hot boltdb locations, bind-mounted over their btrfs paths before containerd/dockerd start. Bulk data (layers, blobs, volumes) stays on compressed btrfs.

Implementation surfaced a third root cause: the VZ disk attachment defaulted to synchronizationMode = .full — every guest FLUSH cost ~10 ms of host F_FULLFSYNC. A dedicated commit switches it to .fsync, aligning VZ with the custom-HV backend (whose block worker has always used plain fsync) and with the durability level Colima/OrbStack ship.

Measured (same host, same day)

Guest dd conv=fsync (avg of 20): VZ .full 10.3 ms → VZ .fsync 1.7 ms on the ext4 volume (HV reference 2.2 ms, Colima 1.0 ms).

10-run avg	ArcBox (this PR)	Colima
docker create (snapshot-prepare was ~182 ms alone)	53 ms	48 ms
docker run --rm alpine true	496 ms (was ~700)	173 ms

Remaining docker start gap (~330 ms) is in the network-endpoint/iptables portion — outside both ABX-496 root causes, to be filed separately.

Upgrade / downgrade

• Existing machines are recreated via a new block_devices drift arm (recreation rewrites the machine record only; image files untouched).
• The host declares the device via arcbox.docker_metadata_device=/dev/vdc on the cmdline: declared-but-missing hard-fails; undeclared (older daemon) skips with zero probe delay.
• Migration is per-entry, idempotent, crash-safe: copy via .partial + atomic rename, then retire the btrfs source to *.pre-ext4 (prevents a recreated blank volume from resurrecting stale state; doubles as the manual downgrade artifact). Every crash window is unit-tested.
• Full matrix in the plan doc §3.

Validation (all green)

• cargo test -p arcbox-agent --lib metadata_migrate: 8/8 crash-window tests
• cargo test -p arcbox-core vm_lifecycle: 53/53 incl. new drift + filename tests
• clippy clean (touched crates); musl cross clippy: zero findings in new files
• boot-assets#45 merged, v0.6.11 released, assets.lock bumped here
• boot_assets e2e full path (format → bind → full Docker lifecycle): VZ 122 s ok, HV 170 s ok
• Version-skew: new daemon + old agent ok; new agent + old rootfs (skip path) ok
• docker_build suite D1–D10: 11/11 checks pass (run with test(e2e): docker build e2e suite — real-world matrix, scenarios D1–D5 + D9–D10 #495's test files overlaid; the suite itself lands via test(e2e): docker build e2e suite — real-world matrix, scenarios D1–D5 + D9–D10 #495)
• Bench recorded on ABX-496 + plan doc §9

[linear-code]

ABX-496

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 101ac9bc1c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[pullfrog]

ℹ️ No critical issues — a few minor observations inline, all non-blocking.

Reviewed changes — a 2 GiB ext4 metadata volume (vdc) is added to carry the five fsync-hot boltdb locations, bind-mounted over their btrfs paths before containerd/dockerd start, plus a VZ synchronization-mode switch that removes the F_FULLFSYNC cost on every guest flush.

• Crash-safe migration state machine — metadata_migrate::prepare_entry copies source→.partial→atomic-rename, retires the btrfs source to *.pre-ext4, and stubs the mountpoint; idempotent with 8 crash-window tests.
• Guest mount/format orchestration — metadata_volume::ensure_metadata_mount waits for the declared device, formats ext4 (fast_commit, no lazy init), e2fsck -y retry on mount failure, then binds each mapping.
• Third block device + drift — create_default_machine attaches vdc and declares arcbox.docker_metadata_device= on the cmdline; a new block_devices-count drift arm recreates pre-vdc machines.
• Drop btrfs NOCOW flagging — the old disable_cow_on_metadata_dirs path is removed; the hot metadata now lives on ext4 and the rest stays on plain btrfs.
• VZ .fsync synchronization — VZDiskImageStorageDeviceAttachment switches from the default .full to .fsync, aligning VZ durability with HV/Colima.
• Supporting wiring — docker_meta_img_path(), disk-usage reporting, resource-cleanup image list (4 names), constants, docs, and assets.lock bump to v0.6.11.

ℹ️ Blank recreated metadata volume starts fresh with no warning

When the metadata image is deleted and recreated blank while a *.pre-ext4 backup still exists on btrfs, prepare_entry correctly refuses to resurrect the stale backup and starts fresh (blank_volume_after_retire_starts_fresh_not_stale). That is the right behavior, but the runtime then opens an empty database silently — no log line records that a .pre-ext4 recovery artifact was left behind and skipped.

Given the design explicitly treats the two images as a paired set (delete/move together), an operator who accidentally removed only docker-meta.img gets a fresh-but-empty metadata store against populated bulk data with nothing in the log to explain it.

Technical details

# Blank recreated metadata volume starts fresh with no warning

## Affected sites
- `guest/arcbox-agent/src/metadata_migrate.rs:82-84` — the `create_entry(&final_path, kind)` / `Prepared::Fresh` branch, reached when the volume side is blank but a retired backup exists at `<target>.pre-ext4`.

## Required outcome
- Emit a warn-level log when starting a fresh entry while a sibling `*.pre-ext4` backup exists, so the paired-image invariant violation is diagnosable.

## Suggested approach (optional)
- In the `Fresh` branch (or in the caller `metadata_volume.rs`), check for `path_with_suffix(target, RETIRED_SUFFIX)` existence and `tracing::warn!` when present.

  ^{｜ Fix all ➔ ｜ Fix 👍s ➔ ｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}

[greptile-apps]

Greptile Summary

This PR moves fsync-heavy container metadata to a dedicated ext4 volume. The main changes are:

• Adds and attaches a sparse 2 GiB metadata image.
• Formats, mounts, migrates, and bind-mounts five metadata locations.
• Adds crash-safe staged copying and source retirement.
• Detects older machine records that need the third block device.
• Uses fsync synchronization for the Virtualization.framework disk.

Confidence Score: 5/5

This looks safe to merge.

The copied files and every copied directory are synced before publication. The published volume entry is synced before the original source is retired.

No blocking issues remain in the updated migration path.

T-Rex Logs

What T-Rex did

• The initial metadata-migrate cargo invocation was captured and failed with exit code 101 due to a missing protoc, as shown in the before log.
• The actual-module harness was compiled and executed successfully, exiting with code 0 and reporting 8 passed; 0 failed in the after log.
• The harness Rust source demonstrates how the test imports the production module with a #[path] attribute and contains the assertions used for validation.

_{Ran code and verified through T-Rex}

Important Files Changed

Filename	Overview
guest/arcbox-agent/src/metadata_migrate.rs	Adds crash-safe metadata migration and now persists copied directory entries before publishing and retiring the source.
guest/arcbox-agent/src/agent/linux/metadata_volume.rs	Adds ext4 volume setup, repair, migration, and bind mounting before the container runtimes start.
app/arcbox-core/src/vm_lifecycle/boot.rs	Creates and attaches the metadata image and declares its guest device on the kernel command line.
virt/arcbox-vz/shim/Sources/ArcBoxVZShim/Devices.swift	Changes writable disk synchronization to the fsync mode used by the new metadata volume.

_{Reviews (2): Last reviewed commit: "fix(agent): make migration publish durab..." | Re-trigger Greptile}

[pullfrog]

✅ No new issues found.

Reviewed changes — the incremental delta since the prior pullfrog review (101ac9bc): a crash-safety durability fix in the migration state machine, plus documentation rewording.

• Migration publish now durable before retire — copy_dir_synced fsyncs each directory after populating it, and prepare_entry fsyncs volume_root after the publish rename (before the btrfs-side retire), with both syncs promoted from best-effort to error-propagating (?). This directly addresses the prior review's directory-entry durability gap.
• docs/daemon-lifecycle.md — reworded docker.img references to "disk images" (plural) to cover the new docker-meta.img and Rosetta counterparts.

Traced the durability fix: the publish sync targets the ext4 volume and is issued synchronously before the btrfs retire rename, so no cross-filesystem crash window can persist the retire while losing the publish — the state machine re-converges either way. The prior three inline threads (config.rs rosetta accessor, migration dentry durability, VZ .fsync blast radius) are all resolved with author replies.

  ^{｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}