Repository navigation
perf: improve default VM create-to-first-exec time below the current 3-second allowance #335
Description
Production-default image VM startup still takes about 2.67 seconds from cached public POST /vms/create through the first completed successful workload exec on the new Linux x86_64 KVM host.
Follow-up to #319 and phase 1 (#299) of the #289/#291 plan. The maintainer explicitly accepts a 3-second limit for the current work and requests this performance follow-up. Keep the remaining 13-phase plan moving; do not change RAM/CPU defaults or qualification requirements to meet a timing number.
Measured twice on clean integration source f3e4629878d637ad5fec762a2c963760d604bad7, with verified, source-matching producer assets and identical host binaries:
| Resources | Create through first completed exec |
|---|---|
| Default 12 GiB / 4 CPU | 2.6747s, 2.6642s |
| Comparison 2 GiB / 2 CPU | 2.0075s, 2.0892s |
Default boot-stage logs report kernel startup 0.94s/0.92s, remaining guest setup 0.42s/0.41s, and first exec 0.181s/0.228s. The remaining create cost is approximately 1.13s/1.11s by subtraction; this is an arithmetic remainder, not a profiled attribution. Functional checks verified immutable kernel audit registration/rules, refusal to unlock auditing, public command history, an actual unique command audit record, and awaited session cleanup. This measurement report lacks the machine-doctor output; a corrected diagnostic is pending. It is not complete exact-source qualification evidence.
Start with guest/artifacts/capsem-init boot-stage instrumentation, crates/capsem-core/src/vm/config.rs, the KVM implementation in crates/capsem-core/src/hypervisor/kvm/, and the service/process create-to-ready handoff. Profile the remaining host admission/setup and workload-launch costs before selecting an optimization. Measure unchanged inputs twice, then compare candidate samples under the same host lease and verified asset cohort.
Acceptance: reduce default cached create-to-first-completed-exec time toward the original sub-2-second goal, without weakening asset verification, audit policy, readiness, ephemeral cleanup, production defaults, or existing deadlines. Run the owning runtime/installed-package gates and complete qualification on the final source; report exact paired samples and variation.
Prepared EROFS path measurements (2026-10-12)
The prepared path is now selected for explicit image grants in ea2c7d1, and the normal pinned fixture serves its subject-bound EROFS in a4e30fa. Native tests assert the exact filesystem digest, read-only EROFS lower, no raw extracted roots, and a completed workload command on create/resume/fresh cached create. Both prior AGY/Codex-Ollama cold-unpack failure owners pass with the original deadlines.
With production 12 GiB/4 CPU, two fresh VMs per image use the same live registry and warmed service cache. Public POST /vms/create through first completed workload exec:
| Image | Two observations |
|---|---|
| dev | 7.1219s, 6.4902s |
| capsem-debug | 11.5182s, 11.2078s |
Cached resumes separately measured dev5.2237s/4.8490s and debug6.9771s/6.8922s. These are diagnostic wall-clock observations, not a controlled comparison with the original2.67s samples. The machine doctor run during testing refused the active test companions; after awaited cleanup it passed (Linux x86_64,16cores,KVM,load1.04). The three-second target remains unqualified.
Reproducer: tests/ironbank/test_prepared_image_root.py, with both pinned layouts and their prepared filesystem artifacts prefetched. Start profiling crates/capsem-service/src/container_setup.rs (full OCI pull still precedes EROFS selection), crates/capsem-assets/src/oci/pull/, and guest/artifacts/container/launch.py::published_root/verified_blob. Attribute host cache revalidation and guest filesystem verification separately, preserve integrity and readiness, and measure cached create-to-first-exec rather than replacing it with resume latency.
Activity
ebursztein commentedon Oct 12, 2026
Crypto and EROFS update on
0.7, through8bb0d6dc6:capsem-cryptoowns accelerated Rust hashes, crypto providers, verification and entropy. SHA/BLAKE comparisons run as the normalcapsem-benchcryptodimension, with no extra benchmark crate.application/vnd.capsem.rootfs.erofs.v2with the pinned manifest'sorg.capsem.rootfs.blake3annotation. OCI transport remains SHA-256. Host cache reuse and guest descriptor-held verification use BLAKE3.Paired cached public create-to-first-completed-exec observations for the same dev EROFS, with default 12 GiB / 4 CPUs and original deadlines:
The BLAKE3 diagnostic's guest initial EROFS verification took 1.8005s/1.6928s and mount took about 28ms. Kernel startup reported 1.08s in both boots; guest launch began around 1.83s/1.84s. Initial VirtioFS reads and guest startup remain on the critical path. Warm guest hash measurements (BLAKE3 0.457–0.534s for this file) cannot replace the initial read or establish boot latency.
The 3-second target is still unqualified. These measurements use a private diagnostic initrd overriding only the new sysutil verifier; they are not complete guest-cohort, installed-package or Apple proof. Source was
6b5015e09plus the subsequently committed BLAKE3/audit changes. No extraction, readiness, integrity, resources or deadlines were relaxed.Start further work at
guest/artifacts/container/launch.py::verified_blob/published_root,crates/capsem-core/src/hypervisor/kvm/virtio_fs, and servicecontainer_setup.rsstartup spans. Preserve the pinned subject, exact size, no-follow held descriptor and full hash check. Usetests/ironbank/test_prepared_image_root.pyfor root/workload acceptance, collect startup markers and paired public create-to-first-exec samples, then qualify the full runtime cohort and installed-package lanes at the phase boundary.ebursztein commentedon Oct 12, 2026
Pushed the systematic Linux performance fixes to
0.7atef4aebf1df21d8a8f0a568f104ecc052a5e36445; latest main remains an ancestor.At the unchanged production settings of 12 GiB and four CPUs:
The dominant boot defect was anonymous VM RAM without guaranteed huge-page
alignment or a huge-page hint on a host whose THP policy is
madvise.capsem-foundation::unix::guest_memorynow owns the aligned lazy mapping,reads the host PMD size (including larger ARM pages), and retains ordinary
pages as a fallback. Actual VM allocation and PCI-hole slot alignment are
guarded; a real unaligned-allocation mutation failed before restoration.
The pinned 6.18.44 guest carries the upstream FUSE
bdi->io_pagesbackport.VirtioFS uses 512-entry rings to fit one-MiB payloads plus headers, bounded
worker buffers, foundation positioned I/O and event-index notifications on
fresh/resumed queues. Speculative read-ahead remains bounded as before.
The broader audit found that checkpoint restore copied sparse holes into
physical RAM and that vhost ignored the userspace vsock event queue. Sparse
extent restoration and TRANSPORT_RESET delivery now preserve existing file
and directory handles while reconnecting sockets immediately. Page-sized
checkpoint sparsity, complete kernel ring ranges and index-cell arithmetic
have regression coverage. Bounds defects were fixed in separate security
commits (
a898e0906,b8af86f71).The checked-in lesson is
skills/dev-rust-patterns/references/guest-memory-performance.md. FocusedRust, strict clippy, formatting and 45 final tracked source/shape/skill/collector
guards pass. Raw paired evidence is recorded through stock capsem-bench:
storage runs 5–8, lifecycle boot runs 9–10 and lifecycle resume runs 11–12.
No benchmark crate was added and full image integrity checks stay enabled.
These native measurements use explicitly diagnostic asset cohorts. Full
exact-source/runtime, installed-package and Apple qualification remain open
in S01-002/027/105; this issue should remain open until that final proof.