Fallback: run HV and VZ Rosetta utility VMs side-by-side with Docker/OCI routing #290

Current status

This issue is now a fallback / contingency plan, not the preferred architecture.

The current preferred path is tracked by #292: validate All-in-HV + FEX64 for linux/amd64 runtime support. If #292 succeeds, ArcBox should avoid a second always-on VZ Rosetta runtime VM and keep VZ/Rosetta only as an optional amd64 build backend or compatibility escape hatch. If #292 fails, resume this issue as the dual-runtime fallback.

Fallback architecture

If FEX64 is not viable for general Docker runtime workloads, adopt the dual-stack execution model for macOS Apple Silicon:

• HV utility VM remains the fast default path for native linux/arm64 workloads.
• VZ Rosetta utility VM is retained as the Apple-only capability path for linux/amd64 workloads that require Rosetta.
• ArcBox exposes one Docker/OCI API while routing workload groups to the correct backend internally.

ArcBox daemon / Docker API
  ├─ HV utility VM
  │    └─ linux/arm64 native containers and machines
  └─ VZ Rosetta utility VM
       └─ linux/amd64 containers/machines via Apple Rosetta

Why this is fallback only

The design works around the fact that Apple exposes Linux Rosetta through VZLinuxRosettaDirectoryShare / Virtualization.framework, not through Hypervisor.framework. However, it adds significant control-plane complexity:

• workload-to-VM role registry;
• restart reconstruction across guest dockerds;
• BuildKit /session ordering and routing;
• Compose/project-level scheduling semantics;
• globally coordinated host ports, DNS/IPAM, bind mounts, volumes, and diagnostics;
• higher memory footprint when both utility VMs are running.

Because FEX64 may provide good-enough amd64 execution inside the HV guest, this complexity should not be the default until #292 has answered the FEX64 viability question.

Fallback scheduling rules

docker run

• --platform linux/arm64 → HV utility VM.
• --platform linux/amd64 + Rosetta available/enabled → VZ Rosetta utility VM.
• --platform linux/amd64 + Rosetta disabled/unavailable → HV fallback translator if configured, otherwise clear error.
• No platform:
  • image has arm64 manifest → HV;
  • image is amd64-only → VZ Rosetta.

Compose / grouped workloads

Backend choice must be made at the workload-group level, not per isolated process:

• If all services support arm64 → run the project on HV.
• If any service requires amd64/Rosetta → run the whole project on VZ Rosetta by default.
• Mixed-backend Compose should remain experimental only after DNS, port, volume, and namespace semantics are explicit.

Kubernetes / pod semantics

• Pods must not span backends.
• If any container in a pod requires amd64/Rosetta, schedule the pod on the VZ Rosetta environment.

Acceptance criteria if resumed

• linux/arm64 containers run on the HV utility VM by default.
• linux/amd64 containers run on the VZ Rosetta utility VM by default when Rosetta is available.
• Both utility VMs can run concurrently behind one ArcBox daemon/API endpoint.
• Docker/ArcBox diagnostics expose backend metadata for routing decisions.
• Host port allocation is globally coordinated across both utility VMs.
• Compose projects are scheduled as a single backend group with clear behavior for amd64-only services.

Related issues

• Evaluate FEX64 as default linux/amd64 runtime inside HV #292 — primary decision gate: evaluate FEX64 as default linux/amd64 runtime inside HV.
• Replace VZ with HV as default macOS backend #250 — strategic HV default workstream.
• ABX-288 — completed backend selection; auto policy may change if Evaluate FEX64 as default linux/amd64 runtime inside HV #292 succeeds.
• ABX-281 — completed VZ Rosetta integration; may become build-only/optional.
• ABX-116 — stale pre-ABX-281 Rosetta support ticket.
• Refactor Vmm into trait-based backend dispatch (VmBackend) + split HV/VZ into their own crates #248 — backend trait split remains useful, but should not over-optimize for always-on dual runtime VMs unless this fallback is resumed.