Materialize guest runtime on Btrfs

[AprilNEA]

What changed

• Treat VirtioFS /arcbox/runtime as a transport-only source.
• Verify the current generation and architecture manifest, then atomically materialize runtime assets under the guest Btrfs data disk.
• Execute Docker, K3s, Sandbox/Firecracker, kernels, and FEX only through the stable Btrfs runtime link.
• Remove the direct VirtioFS execution fallback; failures are visible and retryable.
• Add generation reuse, checksum repair, atomic replacement, stale-generation GC, and fail-closed FEX registration.
• Bump the host/guest protocol contract and support locally generated manifests only in the Development profile.

Why

Direct VirtioFS execution kept the guest coupled to host-mounted resources and made the removed runtime.erofs image necessary. A single Btrfs-backed execution path is smaller, durable across source loss, and has explicit integrity checks.

Compatibility

Old hosts/agents fail immediately at the protocol boundary. Existing four-disk System machines are recreated. Development may use a locally generated manifest; Production remains pinned.

Validation

• cargo test -p arcbox-agent -p arcbox-core -p arcbox-constants
• cargo clippy -p arcbox-agent -p arcbox-core -p arcbox-constants --all-targets -- -D warnings
• ARM64 musl release build and target Clippy
• cargo fmt --all -- --check
• git diff --check

Dependency

Draft until arcbox-boot 0.8.0 and the matching boot artifact are published, then this branch must update Cargo.toml, Cargo.lock, and assets.lock.

[greptile-apps]

Greptile Summary

Materializes generation-scoped runtime assets onto the guest Btrfs disk.

• Stages host runtime binaries and manifests in versioned transport directories.
• Selects the boot generation through the kernel command line and verifies manifest/asset integrity before publication.
• Atomically exposes the active Btrfs generation to Docker, Kubernetes, sandbox, kernel, and FEX consumers.
• Adds generation reuse, repair, stale-generation cleanup, protocol enforcement, and Development-only unpinned manifests.

Confidence Score: 5/5

The PR appears safe to merge.

The previously reported generation-isolation failure is fixed: the host retains generation-keyed runtime sources, while the guest selects the same generation for both its manifest and assets before publishing the Btrfs-backed runtime link; no blocking failure remains.

Important Files Changed

Filename	Overview
guest/arcbox-agent/src/agent/linux/runtime_cache.rs	Selects generation-scoped manifest and runtime transport sources, materializes them locally, and registers Btrfs-backed FEX.
guest/arcbox-agent/src/runtime_materialize.rs	Implements verified staging, atomic generation publication, stable-link replacement, reuse, repair, and stale-generation cleanup.
app/arcbox-core/src/runtime.rs	Stages guest runtime binaries into generation-specific host directories.
app/arcbox-core/src/vm_lifecycle/boot.rs	Adds the selected runtime generation to the guest kernel command line.
app/arcbox-core/src/boot_assets/provider.rs	Removes runtime EROFS handling and enforces profile-aware manifest pinning around binary preparation.

_{Reviews (3): Last reviewed commit: "chore(runtime): adopt boot assets 0.8.0" | Re-trigger Greptile}

[pullfrog]

Important

This PR's own Dependency section says it should stay a draft until arcbox-boot 0.8.0 and the matching boot artifact are published — and at 387a8a1 the workspace still pins arcbox-boot = "0.7.0" with assets.lock on boot 0.6.13. Separately, the Compatibility claim that existing four-disk System machines are recreated has a hole on the error path.

Reviewed changes

• guest/arcbox-agent/src/runtime_materialize.rs — new ~717-line materializer: manifest planning, per-asset SHA-256 verification, staging + renameat2(RENAME_EXCHANGE) publication, stable-symlink swap, old-generation reclaim.
• guest/arcbox-agent/src/agent/linux/runtime_cache.rs — the new ensure_local_runtime() seam (memoised, spawn_blocking) that every runtime consumer now gates on.
• Removal of the VirtioFS execution fallback (detect_runtime_bin_dir, mount_runtime_image) and repointing of ARCBOX_RUNTIME_BIN_DIR at /run/arcbox/runtime/bin.
• Retirement of the fourth read-only runtime-image disk: RUNTIME_IMAGE_BLOCK_DEVICE, arcbox.runtime_image_device=, and DesiredBoot::expected_disk_count() all deleted; arcbox.runtime_generation= added.
• AGENT_PROTOCOL_VERSION / MIN_AGENT_PROTOCOL_VERSION 2 → 3, plus the new protocol_version: 0 boot-contract signal from handle_ping.
• BootAssetConfig::allow_unpinned_manifest, gated on ArcboxProfile::Development.
• Docs (docs/data-directories.md) and the FEX binfmt registration moved onto the Btrfs-backed interpreter path.

⚠️ The PR is marked ready-for-review while its own stated blocker is unresolved

The description says: "Draft until arcbox-boot 0.8.0 and the matching boot artifact are published, then this branch must update Cargo.toml, Cargo.lock, and assets.lock." None of those three updates is in this commit.

The consequence isn't cosmetic. The guest's new manifest deserializer reads a per-binary install_dir field, and guest/arcbox-agent/src/config.rs:61 depends on install_dir: "kernel" existing so /run/arcbox/runtime/kernel/vmlinux resolves for the Firecracker sandbox. The repo's own manifest fixture (app/arcbox-core/src/boot_assets/tests.rs:87-102) models a binary as {name, version, targets: {arm64: {path, sha256}}} — a per-target path, no install_dir anywhere — and the host's prepare_binaries writes flat into data_dir/runtime/bin (runtime.rs:755). So on the currently-pinned schema there is no producer for the kernel/ subdirectory the guest now requires.

I can't confirm this locally: arcbox-boot is a registry dependency, not vendored, so 0.8.0's actual schema isn't inspectable from this tree. But it means the merge-readiness question and the schema question are the same question — landing this before the bump means the change is untested against the manifest it was written for.

Technical details

Workspace Cargo.toml:215
    arcbox-boot = { version = "0.7.0", features = ["download"] }

assets.lock
    [boot]
    version = "0.6.13"
    manifest_sha256 = "7bff7252fa38a74fd1a513baffeceb80eaddb556b11eec55438503ac7e9ec6c0"

The materializer's own unit-test fixture uses "install_dir": "kernel", which is
the clearest in-repo evidence that a new schema is assumed rather than present.

⚠️ Materialization is now all-or-nothing across every manifest binary for the arch

plan_assets walks every entry in the manifest that has a target for the guest arch, and copy_verified fails the asset unless the source is a regular, executable file whose full-file SHA-256 matches the manifest exactly. Any single failure aborts materialize_runtime, which aborts ensure_local_runtime, which now blocks Docker start, Kubernetes start, and sandbox create/restore.

The prior behaviour only checked that the REQUIRED_RUNTIME_BINARIES subset existed on the VirtioFS share. This is a genuine tightening and mostly a good one — but it converts "one optional tool is missing or has a stale hash" from a degraded runtime into a total guest failure, and there's no partial-success or skip path.

The specific class to check before landing: ABX-413 notes that tgz-packaged tools are pinned by the archive sha, not the extracted binary. If any manifest entry's sha256 is an archive digest, copy_verified on the extracted file fails 100% of the time and the guest never starts a runtime at all. Worth confirming against the real 0.8.0 manifest that every entry's sha256 is the extracted-binary digest.

Technical details

// runtime_materialize.rs — copy_verified
// requires: regular file, mode & 0o111 != 0, sha256(full file) == manifest sha
// any Err propagates out of materialize_runtime -> ensure_local_runtime -> caller

// Call sites that now hard-fail:
//   runtime.rs::try_start_bundled_runtime
//   kubernetes.rs::do_start_kubernetes
//   sandbox.rs (503 on SandboxCreate / SandboxRestore)

ℹ️ Nitpicks

• runtime_cache.rs:41 — unregister_fex() runs before materialization. If materialization then fails, the guest is left with no x86_64 binfmt handler at all (the old registration is gone, the new one never happens) until a subsequent pass succeeds. Deferring the unregister to just before register_local_fex would keep the old handler working through a failed pass.
• app/arcbox-core/src/boot_assets/tests.rs:114 — only_development_config_accepts_a_locally_generated_manifest asserts the production arm is_err(), which holds only because assets.lock currently carries a manifest_sha256. Drop the pin and verify_manifest_pin warns and returns Ok, flipping the assertion. Worth constructing the production config with an explicit pin in the test rather than inheriting the lockfile's.

  ^{｜ Fix all ➔ ｜ Fix 👍s ➔ ｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

Run failed. View the logs →

  ^{｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

Run failed. View the logs →

  ^{｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

Run failed. View the logs →

  ^{｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}