Sub-10ms TTI via copy-on-write snapshot restore and pre-warmed snapshot pool

[misaelzapata]

Goal

Reach ~10 ms time-to-interactive (TTI) for workloads like Node.js, Python, and JVM runtimes by restoring from a pre-booted snapshot instead of cold-booting the kernel + init + runtime on every invocation. Target: from gocracker run <image> to a responsive process on the guest in under 10 ms on a warm host.

Current state

Snapshot/restore is already implemented in pkg/vmm:

• TakeSnapshot — pauses the VM, dumps vCPU state + device state + full guest RAM to disk.
• RestoreFromSnapshot — allocates a fresh VM, reads the memory dump back into guest RAM, replays device and vCPU state, and resumes execution.

This is functionally correct but not tuned for latency: the restore path does a full sequential read of the memory dump into a freshly-allocated mmap region. For a 256 MiB guest that is already 30–80 ms of pure I/O, which dominates TTI.

Proposed work

1. Pre-boot snapshot — already done

Boot the target image, wait until the runtime reports ready (e.g. Node's listen callback, the exec-agent handshake, or a user-supplied readiness probe), then call TakeSnapshot. Nothing new here beyond exposing a CLI verb / API endpoint that combines boot + wait + snapshot into one step.

2. Copy-on-write memory restore (~100 LOC, biggest win)

Replace the "open file, read() into mapped RAM" loop in the restore path with a single mmap(MAP_PRIVATE) over the snapshot's memory file, then register that mapping as the guest memory region via KVM_SET_USER_MEMORY_REGION.

• MAP_PRIVATE gives copy-on-write semantics: the guest sees the snapshot's memory lazily, page by page, as it touches it. Pages the guest never touches are never faulted in.
• No up-front I/O — restore becomes O(vcpu + device state), which is kilobytes, not megabytes.
• Multiple concurrent restores from the same snapshot file share the same backing pages until written (lower RSS across a fleet).

Files that need to change:

• internal/kvm/kvm.go — add a memory-region constructor that takes an existing mmap instead of allocating a fresh anonymous one.
• pkg/vmm/snapshot.go (or wherever RestoreFromSnapshot lives) — branch on "snapshot file on a local filesystem" and take the mmap path; fall back to the existing copy path for remote/streaming snapshots.
• Add a SnapshotFormat marker so older dumps without the expected page alignment still restore correctly.

Expected restore cost after this change: ~5 ms on NVMe, dominated by KVM vCPU setup.

3. Pre-warmed snapshot pool (~200 LOC)

Pure orchestration, not VMM-level:

• A new subcommand / API (gocracker pool ...) that keeps N snapshots for a given image ready on disk.
• On run, pop one from the pool, restore it (COW path from §2), and asynchronously replenish the pool in the background.
• Pool metadata: image digest, runtime fingerprint (env, cmdline, kernel), creation time, TTL.
• Invalidation when the image or kernel changes.

This is where TTI actually becomes user-visible — the pool amortizes the boot + wait cost so the user-facing path is always restore-only.

4. Benchmark / success criteria

• bench/tti_test.go — measures run → first byte of stdout across cold boot, current snapshot restore, and COW snapshot restore.
• Pass criteria: median TTI ≤ 15 ms on a CI runner with NVMe, ≤ 10 ms on a tuned host with madvise(MADV_POPULATE_READ) pre-faulting the hot set.

Effort estimate

• §2 COW restore: 1 day (most of it is testing that device state + IRQ setup survives a lazily-faulted memory region).
• §3 Pool: 1 day.
• Benchmark + docs: 0.5 day.

Total: 2–3 days for someone familiar with pkg/vmm and internal/kvm.

Risks / open questions

• Does KVM tolerate MAP_PRIVATE pages for KVM_SET_USER_MEMORY_REGION? Firecracker uses MAP_SHARED with a backing file; we may need the same for dirty-tracking on subsequent snapshots of the restored VM.
• Dirty-tracking: after a COW restore, the dirty bitmap must reflect guest-written pages only, not the original snapshot pages. Needs a short review of internal/virtio.DirtyTracker interaction with the new mmap.
• ARM64 parity: the PL011/PL031/GIC state restore path (added in 04611f4) must be exercised by the new benchmark too.

Related code

• pkg/vmm/snapshot.go
• pkg/vmm/vmm.go — RestoreFromSnapshot
• internal/kvm/kvm.go — memory region setup
• internal/virtio/ — device state serialization