GPU sandbox design proposal

[YangYuS8]

This discussion follows up on #111 and #160.

I originally opened #160 as a documentation PR, but based on maintainer feedback, I’m moving the proposal here first so the design can be discussed before becoming part of the docs.

The goal is to collect feedback on the scope, component responsibilities, lifecycle limitations, deployment limitations, and open questions before starting implementation work.

[YangYuS8]

GPU Sandbox Design Proposal

Motivation

GPU support is becoming increasingly important for AI, inference, model serving, media processing, and other accelerator-dependent workloads.

For Cube Sandbox, the challenge is not only how to attach a GPU to a sandbox, but how to introduce accelerator support without weakening the existing design goals around isolation, lifecycle control, fast startup, and high-density deployment.

GPU acceleration introduces several architectural trade-offs that should be made explicit from the beginning:

• GPU virtualization and passthrough technologies are fragmented. VFIO passthrough, mediated devices, API forwarding, and vendor-specific sharing models each optimize for different goals such as performance, isolation, density, or operational simplicity.
• GPU drivers are tightly coupled with the guest kernel and user-space runtime stack. This complicates guest image management and can conflict with the lean MicroVM model.
• GPU initialization, driver loading, and device preparation can significantly increase cold-start latency, which is one of Cube Sandbox’s core metrics.
• Accelerator support should not be hardcoded only for NVIDIA GPUs. The design should leave room for future TPU, XPU, or other PCIe accelerator backends.
• GPU compatibility cannot be treated as a late runtime check only. Hardware, guest kernel, kernel modules, user-space driver libraries, and CUDA/runtime versions need to be validated as a coherent compatibility contract.

This proposal therefore treats GPU support as a layered design problem: first define a backend-agnostic accelerator abstraction, then use VFIO passthrough as the first reference implementation.

Design Goals

The initial design should aim to:

• define a clear accelerator resource model that is not tied to a single GPU backend
• provide high-level API and module abstractions for requesting, scheduling, attaching, releasing, and observing accelerator devices
• support VFIO passthrough as the first concrete implementation and performance/security baseline
• move host-side GPU preparation into the host initialization phase where possible, instead of handling it repeatedly in the MicroVM control plane
• support template-based GPU instances with device hot-plugging
• ensure that GPU MicroVM memory is fully allocated and pinned when GPU devices are attached dynamically
• keep GPU guest images separate from CPU-only guest images
• allow instances or templates to specify a compatible guest kernel for the selected GPU driver stack
• define and enforce a GPU compatibility contract across hardware, guest kernel, modules.img, kernel driver, user-space driver libraries, CUDA/runtime, and template metadata
• centralize GPU VFIO resource reservation, release, reconciliation, and monitoring in Cubelet
• define clear design boundaries for the Cubelet VFIO resource model, template/kernel compatibility, hot-plug lifecycle, and compatibility validation flow
• document lifecycle limitations explicitly instead of treating them as edge cases
• preserve future extensibility for mediated devices, API forwarding, GPU sharing, TPU, XPU, or other accelerator types

Non-goals

The initial design does not aim to:

• implement full GPU support immediately
• define a final stable public API
• support every GPU virtualization technology in the first iteration
• support live migration or GPU oversubscription
• support mediated devices or time-sliced GPU sharing as the first backend
• preserve snapshot-based lifecycle features for GPU-backed sandboxes
• place host-side GPU initialization work on the MicroVM startup path when it can be completed earlier during host initialization
• merge GPU driver components into CPU-only guest images
• make CubeShim responsible for node-level VFIO resource ownership
• rely only on guest startup checks to discover incompatible GPU environments
• optimize for all vendor-specific GPU capabilities

High-level Architecture

The proposed architecture has two layers.

The first layer is a backend-agnostic accelerator abstraction. This layer describes accelerator requests, node capabilities, device allocation, attachment lifecycle, health state, compatibility metadata, and release semantics without assuming a specific implementation such as VFIO.

The second layer is a pluggable backend implementation. In the initial phase, VFIO passthrough can serve as the reference backend because it provides strong isolation, clear ownership, and a useful baseline for performance and lifecycle behavior.

The initial GPU flow should separate host initialization, template compatibility declaration, sandbox scheduling, MicroVM creation, device attachment, and in-guest validation:

1. During host initialization, Cubelet discovers GPU devices, validates host capabilities, prepares VFIO-related host state, and builds a node-local inventory of allocatable accelerator resources.
2. GPU-capable templates declare their compatibility contract, including guest image, guest kernel, modules.img, GPU driver version, CUDA/runtime version, supported hardware labels, backend type, and memory requirements.
3. The user or higher-level API requests accelerator resources through a generic accelerator model.
4. CubeMaster schedules the sandbox to a node that can satisfy the accelerator request and match the selected template compatibility contract.
5. Cubelet revalidates the selected device and template compatibility before reserving the accelerator resource.
6. Cubelet reserves the selected accelerator resources from its node-local inventory and marks them as unavailable to other sandboxes.
7. Cubelet selects a GPU-capable instance template that matches the requested accelerator, backend, guest image, guest kernel, driver/runtime stack, and memory requirements.
8. The MicroVM is started from the selected template, with memory fully allocated and pinned when GPU devices are expected to be attached dynamically.
9. CubeShim attaches the selected backend-specific device resources to the MicroVM, preferably through a template-compatible device hot-plug path.
10. The GPU-specific guest image, guest kernel, and runtime stack initialize the accelerator inside the sandbox.
11. cube-agent performs final in-guest validation, such as confirming loaded modules, driver availability, and CUDA/runtime visibility.
12. When the sandbox exits, Cubelet uniformly manages VFIO resource release, cleanup, reconciliation, and monitoring.
13. The device returns to the node inventory as allocatable only after cleanup and validation succeed.

This structure allows VFIO passthrough to be implemented first without making VFIO concepts the only possible long-term accelerator model. It also avoids placing unnecessary host-side preparation work on the MicroVM startup path and prevents incompatible hardware/kernel/runtime combinations from reaching a partially usable runtime state.

Accelerator Abstraction

The generic accelerator model should describe what the sandbox needs, not how a particular backend satisfies it.

A minimal abstraction may include:

• accelerator class, such as GPU, TPU, XPU, or other PCIe accelerator
• vendor and device information
• capability labels, such as CUDA support, memory size, device generation, or backend type
• allocation mode, such as exclusive whole-device assignment
• health state, such as allocatable, allocated, unhealthy, disabled, or quarantined
• backend type, such as VFIO passthrough or a future mediated-device backend
• template compatibility
• guest kernel requirement
• guest image/runtime requirement
• driver/runtime compatibility profile
• memory allocation and pinning requirement
• lifecycle state, including initialized, allocatable, reserved, attached, releasing, released, unhealthy, and quarantined

The abstraction should avoid exposing VFIO-specific details in the user-facing model unless those details are required for explicit advanced configuration.

For the first iteration, exclusive whole-device allocation is the simplest model. It avoids the scheduling and isolation complexity of shared-device backends while still giving the project a concrete implementation target.

Host Initialization and Node-local Resource Preparation

GPU-related host preparation should be completed as much as possible during host initialization, rather than being repeatedly handled by the MicroVM control plane at sandbox startup time.

During host initialization, Cubelet should be responsible for:

• discovering GPU devices and their PCI addresses
• validating IOMMU, VFIO, kernel module, and driver prerequisites
• recording device model, vendor, device ID, generation, memory size, NUMA locality, and backend compatibility
• preparing node-local VFIO resource state
• detecting whether devices are already bound, reserved, unhealthy, or administratively disabled
• exposing accelerator inventory to the scheduling layer
• exposing hardware capability labels used by GPU template compatibility matching
• preparing reusable host-side state needed by GPU-capable templates
• initializing Cubelet-owned resource records before any sandbox consumes the device

This decouples host-side preparation from MicroVM lifecycle management. It can reduce sandbox startup overhead, simplify failure handling, and avoid runtime races where device preparation and MicroVM startup compete on the critical path.

Host initialization should not imply that every GPU is immediately attached to a sandbox. It should only make devices visible, validated, and manageable by Cubelet.

Pluggable Backend Model

Different accelerator backends may require different preparation, attachment, and cleanup logic. The design should make this backend boundary explicit.

A backend should be responsible for:

• validating whether a node can support the requested accelerator mode
• preparing backend-specific host state during host initialization where possible
• attaching devices to the sandbox VM
• exposing backend-specific failure information
• cleaning up devices after sandbox exit
• restoring devices to an allocatable state
• reporting backend-specific health or capability information
• contributing backend-specific compatibility metadata when needed

The generic control plane should not need to know every detail of each backend. It should operate on common accelerator states and delegate backend-specific work to the selected implementation.

This makes it possible to start with VFIO passthrough while leaving room for future GPU sharing, mediated devices, or API-forwarding approaches.

VFIO Passthrough Reference Implementation

VFIO passthrough should be treated as the first reference implementation rather than the only possible GPU model.

The VFIO backend can provide a baseline for:

• strong device isolation
• exclusive physical device ownership
• predictable performance
• clear lifecycle semantics
• security comparison against future shared or virtualized approaches

For a VFIO-based sandbox:

1. Cubelet discovers and validates GPU devices during host initialization.
2. Cubelet uniformly manages the node-local VFIO resource inventory.
3. Cubelet exposes hardware capability labels and backend compatibility metadata.
4. CubeMaster selects a node by matching the accelerator request with node hardware metadata and template compatibility metadata.
5. Cubelet revalidates the selected physical GPU and template compatibility before reservation.
6. Cubelet selects and reserves an available physical GPU for a sandbox.
7. The selected device is prepared through the VFIO backend using prevalidated host state.
8. A GPU-capable template is selected based on accelerator, image, kernel, driver, runtime, backend, and memory compatibility.
9. The MicroVM is started with fully allocated and pinned memory when GPU hot-plug is expected.
10. CubeShim attaches the selected VFIO device to the MicroVM, preferably through device hot-plugging.
11. The guest initializes the GPU using the required kernel modules and user-space libraries.
12. cube-agent performs final in-guest compatibility validation.
13. Cleanup happens only after the sandbox VM and shim process have fully exited.
14. Cubelet restores, reconciles, monitors, and marks the VFIO resource allocatable again only after cleanup succeeds.

This model intentionally favors correctness and isolation over density. It should be considered the “gold standard” baseline before introducing more complex sharing or virtualization schemes.

Detailed Boundary: Cubelet VFIO Resource Model

Cubelet should be the node-local owner of VFIO-backed GPU resources. Other components may participate in scheduling, attachment, and guest initialization, but the source of truth for physical GPU allocation on a node should remain in Cubelet.

Resource Ownership

Cubelet should own:

• physical GPU discovery
• VFIO capability validation
• node-local resource inventory
• reservation and release
• device state transitions
• health monitoring
• compatibility metadata reporting
• reconciliation after failures
• quarantine and recovery decisions
• memory overhead and pinned-memory accounting for GPU-capable MicroVMs

CubeShim should not own global VFIO allocation state. It should consume a device assignment created by Cubelet and report lifecycle information that helps Cubelet decide when a device can be safely released.

CubeMaster should not manage individual device file descriptors or VFIO state. It should schedule based on abstract resource availability and compatibility metadata reported from nodes.

Device State Model

A VFIO-backed GPU device may move through the following states:

• initialized: discovered during host initialization and known to Cubelet
• allocatable: validated and available for sandbox placement
• reserved: selected for a sandbox and no longer available to other sandboxes
• preparing: backend-specific preparation is in progress
• attached: successfully attached to a MicroVM
• releasing: sandbox teardown has started, but cleanup is not complete
• released: backend cleanup has completed and the device can return to allocatable after validation
• unhealthy: the device failed validation or health checks
• quarantined: the device is held out of allocation after an uncertain failure until manual or automatic recovery succeeds
• disabled: the device is administratively unavailable

The exact names can change during implementation, but the design should distinguish between “reserved,” “attached,” “releasing,” and “allocatable.” Treating these as the same state would make failure handling and reconciliation fragile.

State Transition Overview

The Cubelet VFIO resource model should be treated as a state machine rather than a set of independent flags. Each transition should have a single owner, explicit preconditions, and a clear failure target.

A possible transition flow is:

1. initialized -> allocatable: the device has been discovered, host prerequisites are validated, and Cubelet can advertise it to the scheduler.
2. allocatable -> reserved: Cubelet atomically reserves the device for a sandbox after scheduling selects the node.
3. reserved -> preparing: backend-specific preparation starts, such as validating VFIO state or preparing attachment metadata.
4. preparing -> attached: CubeShim successfully attaches the device to the MicroVM and reports the result back to Cubelet.
5. attached -> releasing: sandbox teardown starts, guest shutdown begins, or the sandbox is forcefully terminated.
6. releasing -> released: CubeShim has exited or reported final device release status, and backend cleanup has completed.
7. released -> allocatable: Cubelet reconciliation confirms that the device is safe to reuse.
8. any active state -> unhealthy: Cubelet detects a hard device, backend, or health-check failure.
9. any uncertain state -> quarantined: Cubelet cannot prove the device is safe to reuse after a partial failure.
10. quarantined -> allocatable: manual or automatic recovery succeeds and Cubelet revalidates the device.
11. any state -> disabled: an operator or policy removes the device from allocation.

The most important rule is that Cubelet should never transition a device back to allocatable only because the control-plane sandbox object has disappeared. Host-side reality must be reconciled first.

State Transition Rules

The state machine should enforce the following rules:

• Only Cubelet can move a physical VFIO-backed GPU into or out of allocatable.
• CubeMaster may observe summarized capacity and health state, but should not mutate node-local VFIO state directly.
• CubeShim may report attachment and teardown events, but should not decide whether a physical GPU can be reused.
• reserved must be reached before MicroVM startup or GPU hot-plug begins.
• attached should only be reached after CubeShim reports successful device attachment.
• releasing should be entered before backend cleanup starts.
• released is not the same as allocatable; Cubelet must still validate host state before reuse.
• Any failed or ambiguous transition should move the device to quarantined or unhealthy, not directly back to allocatable.
• A device in disabled, unhealthy, or quarantined should not be advertised as schedulable.

These rules keep ownership simple: Cubelet owns node-local resource truth, CubeShim owns MicroVM attachment behavior, and CubeMaster owns cluster-level placement decisions.

Failure Recovery Paths

Failure handling should be defined around the lifecycle stage where the failure occurs.

If reservation succeeds but MicroVM startup fails, Cubelet should:

• cancel the sandbox startup path
• run any backend cleanup that was already started
• verify that the device was never attached
• return the device to allocatable if the host state is clean
• move the device to quarantined if the host state is uncertain

If MicroVM startup succeeds but hot-plug fails, Cubelet should:

• wait for CubeShim to report the attachment failure
• clean up any partial VFIO attachment state
• stop or reject the sandbox startup path
• reconcile host binding and holder state
• return the device to allocatable only if cleanup is proven safe

If hot-plug succeeds but guest initialization or compatibility validation fails, Cubelet should:

• treat the sandbox startup as failed
• trigger sandbox teardown
• wait for CubeShim final release status
• run VFIO cleanup
• decide between allocatable, quarantined, or unhealthy based on post-cleanup validation

If sandbox teardown starts but CubeShim still holds VFIO file descriptors, Cubelet should:

• keep the device in releasing
• avoid exposing the device as allocatable
• retry or wait for cleanup according to timeout policy
• move the device to quarantined if release cannot be proven safe

If Cubelet restarts during an active GPU sandbox, it should reconstruct state from persisted reservations, runtime sandbox records, and host device state before advertising capacity again.

Persistence and Reconciliation

Cubelet should persist enough information to recover VFIO resource ownership after restart or partial failure.

Persisted reservation state may include:

• sandbox identifier
• device identifier and PCI address
• backend type
• selected template
• selected compatibility contract
• expected lifecycle state
• timestamp of reservation
• memory and pinned-memory requirement
• last known CubeShim or sandbox runtime identifier

After restart, Cubelet should reconcile persisted state with host reality before reporting capacity to CubeMaster.

Reconciliation should check:

• whether the sandbox still exists
• whether the MicroVM process still exists
• whether CubeShim still exists or has already exited
• whether the PCI device exists at the expected address
• whether the device is still bound to the expected driver
• whether any process still holds the device
• whether the device passes basic health validation
• whether the persisted state matches runtime reality

If persisted state and host reality disagree, Cubelet should prefer safety over availability. The device should remain unavailable until Cubelet can prove it is safe to reuse or an operator resolves the conflict.

Events Reported to CubeMaster

Cubelet should report summarized accelerator state to CubeMaster without exposing unnecessary VFIO implementation details.

Useful reportable information may include:

• total GPU count by accelerator class and capability label
• allocatable GPU count
• allocated GPU count
• unhealthy GPU count
• quarantined GPU count
• disabled GPU count
• supported backend types
• supported template or compatibility labels
• memory and pinned-memory availability for GPU-capable MicroVMs
• high-level reason codes for unavailable devices

CubeMaster can use this information for scheduling and capacity awareness, while Cubelet remains the owner of detailed node-local VFIO state.

Reservation Rules

Cubelet should reserve a device before MicroVM startup or device hot-plug begins.

Reservation should record:

• sandbox identifier
• node identifier
• selected device identifiers
• selected backend
• selected template
• selected guest image
• selected guest kernel
• selected compatibility profile
• allocation timestamp
• memory and pinned-memory requirement
• current lifecycle state

Reservation must be atomic from Cubelet’s local perspective. Once a device is reserved, no other sandbox on the same node should be able to consume it, even if VM startup has not happened yet.

Release Rules

Cubelet should mark a device allocatable again only after all release conditions are satisfied.

For the VFIO reference backend, release should require:

• guest shutdown or sandbox termination has completed
• MicroVM process has exited
• CubeShim has exited or reported that the device is no longer held
• VFIO file descriptors are no longer held by the sandbox path
• backend-specific cleanup has completed
• host binding state is restored or validated
• Cubelet reconciliation confirms that the node-local resource state is consistent

A conservative release path is preferred. A stale reservation is safer than incorrectly reassigning a device that is still busy.

Reconciliation and Failure Recovery

Cubelet should periodically reconcile its resource state with host reality.

Reconciliation should cover:

• devices reserved in Cubelet state but not attached to any running sandbox
• devices marked attached while the corresponding MicroVM no longer exists
• devices that failed hot-plug
• devices that failed guest initialization
• devices that failed in-guest compatibility validation
• devices that failed VFIO cleanup
• devices that remain busy after sandbox exit
• devices whose binding state differs from Cubelet’s expected state

Failure recovery should be conservative. If Cubelet cannot prove that a device is safe to reuse, it should move the device to quarantined or unhealthy rather than returning it to allocatable.

Possible failure cases include:

• reservation succeeds but MicroVM startup fails
• MicroVM startup succeeds but hot-plug fails
• hot-plug succeeds but guest driver initialization fails
• guest compatibility validation fails
• guest exits but CubeShim still holds the device
• cleanup fails or times out
• device health check fails after release

Each failure case should have a clear target state and retry behavior.

Monitoring and Health

Cubelet should expose node-local accelerator health information to the scheduling layer.

At minimum, health reporting should distinguish:

• allocatable
• allocated
• unhealthy
• quarantined
• disabled
• unknown

For the VFIO reference backend, health checks may include:

• device presence
• PCI address stability
• expected driver binding state
• IOMMU/VFIO readiness
• whether the device is currently held by an unexpected process
• backend cleanup success
• basic device-level health signals when available

The health model should be generic enough that future GPU sharing or non-GPU accelerator backends can reuse it.

Detailed Boundary: Template and Kernel Compatibility

GPU-backed sandboxes should be based on GPU-capable templates rather than extending CPU-only templates implicitly.

The template should represent a compatibility contract between the requested accelerator resource and the guest runtime environment.

GPU Template Contract

A GPU-capable template should explicitly bind together:

• guest image
• guest kernel
• modules.img
• GPU driver version
• user-space driver library version
• CUDA or runtime version
• supported accelerator class
• supported device vendor or capability labels
• supported backend type, such as VFIO
• memory allocation and pinning requirements
• device hot-plug expectation
• guest initialization expectation
• expected in-guest validation checks

This contract makes compatibility visible before sandbox startup. It also prevents CPU-only images from accumulating unnecessary GPU driver components.

Guest Image Separation

GPU guest images should be separated from CPU-only guest images.

This separation has several advantages:

• CPU-only images do not need to carry GPU driver components
• storage consumption is reduced for non-GPU workloads
• CPU-only sandbox startup remains leaner
• GPU images can evolve according to driver/runtime compatibility requirements
• GPU image testing can be scoped to accelerator-capable templates

GPU-capable images may be larger and more specialized. That trade-off should be explicit rather than hidden inside the default guest image.

Guest Kernel Selection

The system should support an instance or template specifying the guest kernel to use.

This is necessary because GPU driver compatibility is often tied to:

• guest kernel version
• kernel module build
• modules.img
• NVIDIA driver version
• user-space driver library version
• CUDA/runtime version
• device generation

A GPU template should not merely specify an image name. It should also specify the guest kernel and module package that are known to work with the selected GPU driver stack.

Scheduling Compatibility

CubeMaster should consider template and kernel compatibility during scheduling.

A GPU sandbox should only be placed on a node where:

• the requested accelerator class is available
• the requested hardware labels match the selected template
• the requested backend is supported
• the selected template is supported
• the guest kernel/runtime requirements can be satisfied
• the node can provide the required pinned memory
• the node can attach the selected device through the expected hot-plug path

This makes scheduling more than a “GPU count” decision. It becomes a compatibility match between device, backend, template, kernel, image, runtime, and memory requirements.

Compatibility Metadata

Compatibility metadata may include:

• template identifier
• guest image identifier
• guest kernel identifier
• modules.img identifier
• kernel driver version
• user-space driver library version
• CUDA/runtime version
• supported accelerator class
• supported vendor/device labels
• supported backend type
• required memory pinning behavior
• whether device hot-plug is required or supported
• expected in-guest validation command or probe

This metadata should remain abstract enough to support future non-NVIDIA accelerator types.

Compatibility Validation and Enforcement

Compatibility across GPU hardware, guest kernel drivers, and user-space driver/runtime stacks should be represented as an explicit compatibility contract.

The goal is to fail early with a clear compatibility error if the selected hardware, template, kernel, modules, driver, or runtime stack cannot work together. The system should avoid starting a sandbox that reaches a partially usable GPU runtime state.

Compatibility Contract

A compatibility contract should bind together:

• GPU hardware identity, such as vendor ID, device ID, model, generation, and memory size
• accelerator capability labels, such as CUDA support, backend support, or device family
• backend type, such as VFIO passthrough
• guest image identifier
• guest kernel version or identifier
• modules.img or kernel module package version
• GPU kernel driver version
• user-space driver library version
• CUDA or runtime version
• template identifier
• memory allocation and pinning requirements
• expected guest-side validation behavior

This contract can be attached to a GPU-capable template and matched against node-local hardware metadata reported by Cubelet.

Validation Stages

Compatibility should be validated in multiple stages.

During host initialization, Cubelet should:

• discover GPU hardware identity and capability labels
• validate host IOMMU/VFIO readiness
• record backend compatibility
• expose node-local accelerator metadata for scheduling
• mark devices unhealthy or disabled if host-side validation fails

During template build or registration, GPU-capable templates should declare:

• guest image
• guest kernel
• modules.img
• kernel driver version
• user-space driver library version
• CUDA/runtime version
• supported hardware labels
• supported backend type
• memory and hot-plug requirements

During scheduling, CubeMaster should:

• match the accelerator request against node hardware metadata
• match the selected template compatibility contract against node capability metadata
• reject placements where hardware, backend, template, kernel, or runtime requirements cannot be satisfied

During reservation and hot-plug, Cubelet should:

• revalidate that the selected physical device still matches the selected compatibility contract
• confirm that the selected template and guest kernel match the requested GPU stack
• confirm that required pinned memory can be provided
• fail before device attachment if compatibility or memory requirements are no longer satisfied

During guest startup, cube-agent should:

• confirm that required kernel modules are loaded
• verify that GPU device files are visible as expected
• confirm that user-space driver libraries are available
• confirm CUDA/runtime visibility when applicable
• report a clear validation failure if the in-guest environment is not usable

Enforcement Behavior

Compatibility validation should be enforced before the sandbox is considered ready.

If validation fails before scheduling, CubeMaster should reject the placement.

If validation fails during reservation or hot-plug preparation, Cubelet should reject the sandbox startup path and release or quarantine the device depending on whether the failure is safe and recoverable.

If validation fails inside the guest, cube-agent should report the failure to the control plane. Cubelet should then decide whether to stop the sandbox, mark the startup as failed, release the device, or quarantine the device if the failure leaves the hardware state uncertain.

The error should be explicit enough to identify which layer failed:

• host hardware capability mismatch
• backend incompatibility
• template incompatibility
• guest kernel mismatch
• modules.img mismatch
• kernel driver mismatch
• user-space driver/runtime mismatch
• CUDA/runtime validation failure
• pinned memory requirement failure

This staged validation model keeps compatibility checks close to the layer that owns the relevant information while still enforcing a single end-to-end compatibility contract.

Detailed Boundary: Hot-plug Lifecycle

The hot-plug lifecycle should define when a GPU device is attached, what must be true before attachment, and how failure should be handled.

Lifecycle Phases

A GPU-backed sandbox using VFIO passthrough may follow this lifecycle:

1. Host initialization
2. Template compatibility validation
3. Scheduling
4. Cubelet device reservation
5. MicroVM creation from GPU-capable template
6. Full memory allocation and pinning
7. Pre-hot-plug compatibility revalidation
8. Device hot-plug through CubeShim
9. Guest-side driver/module initialization
10. In-guest compatibility validation
11. Runtime operation
12. Guest shutdown or sandbox termination
13. CubeShim exit
14. VFIO cleanup
15. Cubelet reconciliation
16. Device release or quarantine

This lifecycle keeps resource ownership in Cubelet while allowing CubeShim to focus on MicroVM attachment behavior.

Preconditions for Hot-plug

Before GPU hot-plug begins, the following should be true:

• Cubelet has reserved the selected device
• the selected device is not allocatable to other sandboxes
• the selected template is GPU-capable
• the selected compatibility contract matches the chosen physical device
• the guest image and guest kernel are compatible with the requested GPU stack
• MicroVM memory has been fully allocated
• required memory has been pinned when required by the backend
• the node still satisfies the required host-side VFIO state
• CubeShim has received an explicit device assignment from Cubelet

If any of these conditions fail, the system should not attempt device hot-plug.

Memory Allocation and Pinning

GPU device attachment may require stronger memory guarantees than CPU-only sandboxes.

For GPU-capable MicroVMs, the design should require:

• full memory allocation before device attachment
• pinned memory when required by the backend or device model
• memory overhead accounting during scheduling
• failure before hot-plug if pinned memory cannot be provided
• clear reporting when a GPU sandbox cannot start due to memory constraints

This may increase per-sandbox memory overhead and reduce density, but it avoids runtime instability and unpredictable GPU workload performance.

Hot-plug Failure Handling

If hot-plug fails, the system should recover without leaving the device in an ambiguous state.

Possible actions include:

• stop or reject the sandbox creation path
• detach any partially attached device state when possible
• wait for CubeShim to report attachment failure
• run backend-specific cleanup
• reconcile host device state
• return the device to allocatable if safe
• quarantine the device if safety cannot be proven

Cubelet should remain the component that decides whether the device can be reused.

Teardown Ordering

Teardown should follow a strict ordering:

1. guest workload stops
2. guest OS or sandbox shutdown begins
3. MicroVM exits
4. CubeShim exits or reports final device release status
5. VFIO file descriptors are no longer held by the sandbox path
6. backend-specific cleanup completes
7. Cubelet reconciles host state
8. Cubelet releases the device to allocatable, or quarantines it if cleanup is uncertain

This ordering should be treated as a lifecycle invariant for the VFIO reference implementation.

Template and Device Hot-plug Model

GPU-backed sandboxes should preferably be based on templates rather than fully ad-hoc MicroVM construction at runtime.

The template model should allow the system to predefine GPU-compatible VM configuration, guest image, guest kernel, runtime expectations, compatibility metadata, and validation behavior. Device attachment can then happen through a controlled hot-plug path when the sandbox is created or activated.

This approach has several advantages:

• it keeps GPU instance configuration explicit and repeatable
• it avoids mixing CPU-only and GPU-specific assumptions in the same template
• it allows GPU guest image and kernel compatibility to be validated earlier
• it exposes compatibility mismatches before sandbox startup
• it reduces the amount of per-sandbox configuration logic required at runtime
• it provides a clearer boundary between template preparation and device assignment

When GPU devices are hot-plugged into a MicroVM, memory handling becomes important. GPU workloads should not rely on memory that may be swapped or lazily committed during device operation.

For GPU-capable MicroVMs, the design should require:

• full memory allocation before GPU attachment
• pinned memory when required by the backend or device model
• explicit memory overhead accounting in scheduling and allocation
• clear rejection behavior when the node cannot provide the required pinned memory

This may increase per-sandbox memory overhead, but it improves stability and avoids runtime performance surprises for GPU workloads.

Component Responsibilities

CubeMaster

CubeMaster should own the cluster-level scheduling and resource accounting model.

Its responsibilities may include:

• defining accelerator-aware scheduling inputs
• tracking node-level accelerator capacity and allocation state
• selecting nodes that satisfy accelerator requirements
• considering backend compatibility during scheduling
• considering GPU template, guest image, guest kernel, driver, and runtime compatibility during scheduling
• considering pinned-memory requirements during scheduling
• rejecting placements that cannot provide the requested accelerator resources or compatibility contract
• preserving generic accelerator metadata for future TPU or XPU support

CubeMaster should avoid embedding VFIO-specific assumptions directly into the scheduling model where a more generic accelerator abstraction is sufficient.

Cubelet

Cubelet should own node-local accelerator lifecycle management and should be the central manager for GPU VFIO resources.

Its responsibilities may include:

• discovering local accelerator devices during host initialization
• validating host-side GPU and VFIO prerequisites
• reporting allocatable accelerator inventory and hardware capability labels
• managing GPU VFIO resource reservation, release, reconciliation, monitoring, and health checks
• reserving and releasing devices for sandbox lifecycle operations
• revalidating device/template compatibility before reservation and hot-plug
• coordinating backend-specific preparation and cleanup
• preventing double allocation on a single node
• reconciling stale or failed allocations
• managing device state transitions across initialized, allocatable, reserved, preparing, attached, releasing, released, unhealthy, quarantined, and disabled states
• accounting for memory overhead and pinned memory requirements for GPU-capable MicroVMs
• deciding whether failed devices can be reused or must be quarantined

Centralizing VFIO resource management in Cubelet provides consistent reservation, release, and monitoring behavior across the node. It also reduces the risk of resource conflicts between multiple local components.

CubeShim

CubeShim should attach the selected backend-specific device resources to the sandbox VM.

For the VFIO reference backend, its responsibilities may include:

• attaching VFIO devices to the MicroVM
• supporting a template-compatible device hot-plug path
• keeping the device lifecycle tied to the actual VM process lifecycle
• reporting attachment and teardown status back to Cubelet
• ensuring teardown ordering is compatible with safe device release
• mounting required read-only resources such as modules.img when needed by the guest

CubeShim should not own VFIO resource allocation globally. It should consume a device assignment made by Cubelet and report enough lifecycle information for Cubelet to release the resource safely.

cube-agent

cube-agent runs inside the guest and should handle guest-side initialization, compatibility validation, and diagnostics.

Its responsibilities may include:

• loading required guest kernel modules during startup, such as NVIDIA modules provided through modules.img
• configuring guest-side NVIDIA runtime settings, such as persistence mode when required
• verifying that expected GPU device files are visible
• verifying that driver libraries and CUDA/runtime components are available
• reporting in-guest compatibility validation results
• providing guest-side diagnostics to confirm that the attached accelerator is usable inside the sandbox

cube-agent should not be treated as the host-side accelerator inventory source.

Guest Image, Guest Kernel, and Runtime Stack

GPU-backed sandboxes require a guest software stack that differs from CPU-only sandboxes.

GPU guest images should include the required kernel and user-space driver components for the selected GPU stack. They should be separated from CPU-only guest images to avoid increasing storage consumption and image complexity for workloads that do not need GPU support.

The guest side may need:

• guest kernel support for PCI device initialization and required GPU driver modules
• guest kernel modules delivered through the root filesystem or modules.img
• NVIDIA kernel driver components inside the guest
• NVIDIA user-space driver libraries inside the guest
• NVIDIA Container Toolkit or equivalent runtime integration if the workload expects it
• version alignment between guest kernel, guest modules, driver stack, and CUDA runtime

The system should support an instance or template specifying the guest kernel to use. This is necessary because GPU driver compatibility is often tied to specific kernel versions, module builds, and runtime libraries.

In practice, GPU-capable templates should explicitly bind together:

• guest image
• guest kernel
• modules.img
• driver version
• user-space driver library version
• CUDA or runtime version
• supported accelerator backend
• supported device class or capability labels
• memory allocation and pinning requirements
• hot-plug support expectation
• guest-side validation expectations

This makes GPU compatibility part of the template contract rather than an implicit runtime assumption.

Discovery, Scheduling, Allocation, and Release

Discovery

Node-local accelerator discovery should identify enough information for scheduling and lifecycle management.

At minimum, discovery should capture:

• PCI address
• vendor and device identifiers
• accelerator class
• device generation or capability labels
• backend compatibility
• driver binding state
• health state
• allocatable state
• template/runtime compatibility labels
• pinned-memory requirement or memory overhead model
• current sandbox association, if allocated

The inventory model should remain generic enough to support future accelerator types.

Scheduling

CubeMaster should treat accelerator requests as hard placement constraints.

A sandbox that requests accelerators should only be scheduled to a node where:

• the requested accelerator class exists
• enough allocatable devices are available
• the requested hardware labels match the selected GPU template
• the node supports the requested backend
• the required host runtime prerequisites are satisfied
• the selected compatibility contract can be satisfied
• the required guest image, guest kernel, driver, and runtime stack are compatible
• the required GPU-capable template is available
• enough memory can be allocated and pinned for the GPU-capable MicroVM

For the first implementation, exclusive whole-device assignment should be assumed.

Allocation

After placement, Cubelet should reserve selected devices before MicroVM startup and backend attachment.

Allocation metadata should include:

• sandbox identifier
• node identifier
• selected device identifiers
• selected backend
• selected template
• selected guest image
• selected guest kernel
• selected compatibility contract
• memory and pinned-memory requirement
• allocation timestamp
• lifecycle state

Reservation must happen before backend-specific attachment to avoid races with concurrent sandbox creation.

Release

Device release is not only a control-plane state transition.

A device should be marked allocatable again only after:

• the guest has stopped using it
• the VM process has exited
• CubeShim has exited or reported final device release status
• backend-specific cleanup has completed
• the host confirms the device is no longer held by the previous sandbox path
• the device is restored to an expected state
• Cubelet has reconciled and updated the node-local VFIO resource state

A conservative release path is preferred. A stale allocation is safer than reallocating a device that is still busy.

VFIO Lifecycle Pitfall

The most important VFIO-specific operational risk is teardown ordering.

For passthrough, the device may need to be rebound from the host driver to vfio-pci before it can be attached to a MicroVM. This preparation should be handled by Cubelet and the VFIO backend, preferably using host initialization state where possible.

During teardown, releasing the device too early is unsafe if CubeShim still holds VFIO file descriptors or if VFIO unpin operations are still in progress.

The VFIO backend and Cubelet should enforce the following invariant:

• prepare and validate VFIO state before device attachment
• do not mark the device free when the guest only begins exiting
• wait for CubeShim to exit or report final device release status
• confirm that no remaining holder keeps the device busy
• complete backend-specific cleanup
• only then restore and release the device through Cubelet

Violating this ordering can produce false-free devices, failed rebinds, or cross-sandbox interference on the next allocation.

Cold-start and Performance Trade-offs

GPU-backed sandboxes are expected to have different startup characteristics from CPU-only sandboxes.

Cold-start latency may increase due to:

• GPU template selection and validation
• compatibility contract matching
• pinned memory allocation
• device hot-plugging
• guest driver initialization
• kernel module loading
• CUDA or runtime initialization
• in-guest compatibility validation
• device health validation

The design should reduce avoidable runtime overhead by moving host-side preparation into the host initialization phase. However, some GPU-specific costs will remain inherent to the sandbox startup or activation path.

This trade-off should be treated as explicit product behavior. GPU-backed sandboxes may not preserve the same startup latency, high concurrency, or deployment density as CPU-only sandboxes.

The design should make it clear that GPU support targets workloads where accelerator access is more important than extreme elasticity.

Unsupported Lifecycle Features

The initial GPU design should explicitly exclude lifecycle features that conflict with physical accelerator ownership or require major additional work.

GPU-backed sandboxes should not initially support:

• app snapshots
• 1:N cloning from one source sandbox
• snapshot rollback
• general pause and resume
• live migration

These limitations should be documented as expected behavior rather than hidden as edge cases.

Deployment Limitations

The VFIO reference implementation depends on PCI passthrough.

Because PVM-based deployments cannot run guest VMs with passthrough PCI devices, PVM-based deployments are out of scope for the initial VFIO reference implementation.

The first implementation should target environments where KVM and PCI passthrough are available and operationally supported.

Extensibility for Future Backends

Although VFIO passthrough is the proposed reference implementation, the architecture should remain open to future backends.

Possible future directions include:

• mediated devices
• vendor-specific GPU sharing
• API forwarding
• time-sliced GPU access
• non-GPU PCIe accelerators
• TPU or XPU backends

To support that evolution, internal records, scheduling predicates, backend interfaces, template compatibility metadata, compatibility contracts, and node capability reporting should be accelerator-oriented rather than NVIDIA-only or VFIO-only.

Open Questions

• What is the minimal user-facing accelerator request API?
• Which concepts belong in the generic accelerator abstraction, and which should remain backend-specific?
• Where should the pluggable backend boundary be placed?
• Should backend implementations live inside Cubelet, or behind a separate node-local helper?
• Which GPU preparation steps must happen during host initialization, and which must remain on the sandbox lifecycle path?
• What node validation is required before a node can advertise GPU readiness?
• How should CubeMaster represent backend compatibility during scheduling?
• What health states are needed for accelerators?
• What exact VFIO device states should Cubelet expose internally and externally?
• How should Cubelet persist and reconcile VFIO reservation state after restarts or partial failures?
• How should GPU template compatibility be represented?
• How should the compatibility contract be stored, versioned, and validated?
• Which compatibility checks should be hard failures, and which should be warnings?
• How should guest image and guest kernel compatibility be matched with accelerator requirements?
• What is the expected versioning model for NVIDIA driver, CUDA, guest kernel, and modules.img?
• How should instances specify the required guest kernel?
• What is the required memory allocation and pinning model for GPU-capable MicroVMs?
• At which lifecycle point should memory be fully allocated and pinned?
• How should failure recovery work if device preparation succeeds but VM startup or hot-plug fails?
• How should hot-plug success or failure be reported from CubeShim back to Cubelet?
• How should in-guest compatibility validation failures be reported back to Cubelet and CubeMaster?
• What metrics should be collected to compare VFIO with future backend implementations?
• How should cold-start latency changes be measured and exposed?
• What is the right abstraction boundary to support TPU and XPU devices without overengineering the first GPU iteration?

Summary

This proposal frames GPU sandbox support as a layered accelerator design.

The first layer is a backend-agnostic accelerator abstraction for API, scheduling, allocation, lifecycle, template compatibility, compatibility validation, and health management. The second layer is a pluggable backend model, with VFIO passthrough serving as the first reference implementation and baseline for isolation, performance, and lifecycle behavior.

Compared with the previous draft, this version places more emphasis on detailed design boundaries. Cubelet should centrally own the VFIO resource state machine, including reservation, attachment state, release, reconciliation, health monitoring, failure recovery, quarantine, persistence, and node-local hardware capability reporting. GPU sandboxes should use GPU-specific templates that explicitly bind guest image, guest kernel, modules.img, driver version, user-space driver library version, runtime version, backend type, device capability labels, validation expectations, and memory requirements. GPU device attachment should preferably follow a template-based hot-plug lifecycle with full memory allocation and pinning before attachment.

Compatibility should be enforced through a staged compatibility contract across host GPU hardware, guest kernel, modules.img, kernel driver, user-space driver libraries, CUDA/runtime, and GPU-capable templates. The system should fail early with clear errors when incompatible combinations are detected instead of allowing the sandbox to reach a partially usable GPU runtime state.

The next step should be to refine these boundaries into concrete implementation-facing design items: the Cubelet VFIO resource state machine, the template/kernel/runtime compatibility contract, the compatibility validation flow, and the hot-plug lifecycle between Cubelet, CubeShim, and the guest.

[YangYuS8]

Overall the design looks solid. However, there is one aspect I’d like to clarify: How do we validate and enforce compatibility across GPU hardware, kernel drivers, and user-space driver/runtime stacks?

Thanks for raising this. I agree this should probably be discussed before I fold it back into the proposal.

My current thinking is that compatibility should be represented as an explicit GPU compatibility contract, rather than validated only after the sandbox starts.

At a high level, the contract could bind together:

• GPU hardware identity and capability labels, such as vendor ID, device ID, model/generation, memory size, and supported backend
• guest kernel version
• modules.img or kernel module package version
• GPU kernel driver version
• user-space driver library version
• CUDA/runtime version
• guest image or template identifier

Then validation could happen in multiple stages:

1. Host initialization: Cubelet discovers GPU hardware, validates IOMMU/VFIO readiness, records vendor/device IDs and capability labels, and exposes node capability metadata.
2. Template build or registration: GPU-capable templates declare their guest image, guest kernel, modules.img, driver version, CUDA/runtime version, supported GPU hardware labels, and supported backend.
3. Scheduling: CubeMaster only places a GPU sandbox on nodes whose hardware/backend metadata matches the selected template compatibility contract.
4. Sandbox startup and hot-plug: Cubelet revalidates the selected device and template before reservation and attachment.
5. Guest startup: cube-agent performs a final in-guest validation, such as checking loaded modules, driver availability, and CUDA/runtime visibility.

If any stage fails, the sandbox should fail early with a clear compatibility error instead of starting with a partially usable GPU environment.

Does this staged compatibility-contract model match what you had in mind, or would you prefer compatibility to be handled more strictly at template registration time, before scheduling is involved?

[charliecgxu]

That makes sense to me.

[YangYuS8]

That makes sense to me.

Thanks, I’ve updated the discussion body to include the compatibility validation model.

I added it as a staged compatibility contract across GPU hardware, guest kernel, modules.img, kernel driver, user-space driver libraries, CUDA/runtime, and GPU-capable templates. The proposal now also makes the validation points and failure behavior more explicit, so incompatible combinations should fail early instead of reaching a partially usable GPU runtime state.

Please take another look when convenient.

[charliecgxu]

Nice update, you can keep going with this proposal.

[YangYuS8]

Nice update, you can keep going with this proposal.

I continued refining the proposal and expanded the Cubelet VFIO resource state machine as the next implementation-facing boundary.

This update focuses on:

• explicit VFIO device state transitions
• reservation, attachment, release, and teardown rules
• failure recovery paths for startup, hot-plug, guest validation, and cleanup failures
• quarantine behavior when device safety cannot be proven
• Cubelet persistence and reconciliation after restart or partial failure
• high-level events Cubelet may report back to CubeMaster

I started with this section because it seems to be the foundation for reservation, release, reconciliation, monitoring, and failure recovery. If this direction looks good, I’ll continue with the template/kernel/runtime compatibility contract next.

[musaabhasan]

For GPU-enabled sandboxes, I would make the threat model explicit before finalizing the component split. GPU access changes the isolation boundary: the sandbox is no longer only CPU, filesystem, and network. It also includes device plugin behavior, driver surface, memory cleanup, scheduling fairness, and possible cross-workload leakage.

Design points I would want captured in the proposal:

• whether GPU access is pass-through, mediated, or partitioned with MIG/vGPU-like isolation
• how device nodes and driver capabilities are exposed to the sandbox
• whether GPU memory is reset or scrubbed between sandbox lifecycles
• how failed/aborted workloads release GPU state and quotas
• per-sandbox GPU quota, timeout, and telemetry limits
• audit events for device assignment, release, and policy denial
• clear unsupported cases, especially nested runtimes and privileged containers

For AI agent workloads, I would also add a network/filesystem policy example with GPU enabled. In practice, the risky case is not just "can it use CUDA"; it is "can an untrusted model/tool use GPU plus filesystem plus network in the same execution boundary."