Layered rootfs: read-only shared base + per-VM writable layer

[artemnikitin]

Problem

Every service gets its own monolithic, read-write rootfs image, even when services are byte-for-byte equivalent. The rootfs content is a function of three inputs that are currently fused at build time in firework-gitops-example:

• source_image (the Docker image)
• rootfs_size_mb (the ext4 size, applied by dd + mkfs.ext4 -F -d)
• the config overlay set (configs/<service>/, configs/<tenant>-<service>/)

Because they are fused, two services sharing a Docker image still produce two full images if they differ in size or overlays. In the current example repo, 4 of 7 services use elasticsearch:8.19.11, but only one pair (tenant-3's two data nodes) is actually identical:

Service	Image	Size	Overlay
tenant-1-elasticsearch	es:8.19.11	2048	shared + tenant
tenant-3-elasticsearch-voting-only	es:8.19.11	2048	none
tenant-3-elasticsearch-data-1	es:8.19.11	4096	none
tenant-3-elasticsearch-data-2	es:8.19.11	4096	none
tenant-2-elasticsearch	es:9.3.0	2048	shared + tenant
tenant-1-kibana	kibana:8.19.11	2048	shared
tenant-2-kibana	kibana:9.3.0	2048	shared

Cost: redundant CI build time, redundant bucket storage and upload, redundant per-node download, and redundant node disk. A host running both tenant-3 data nodes stores 2x4 GB of identical bytes.

Why the rootfs must be read-write today

This is the blocking mechanism, and it is one function. fc-init mutates the rootfs on every boot: applyUserSpec -> ensureWritablePaths -> chownPathRecursive recursively Lchowns each declared writable path in place.

Consequently writeVMConfig attaches the rootfs as:

drives := []firecrackerDrive{{DriveID: "rootfs", PathOnHost: svc.Image, IsRootDevice: true, IsReadOnly: false}}

Since the backing file is opened read-write and there is no per-VM copy, two microVMs can never share one rootfs file. anti_affinity_group is preferential, not a guarantee, so two services pointing at one path can co-schedule and corrupt the ext4.

Note that IsReadOnly already exists on the drive struct and is simply always false.

Proposal

Attach the base image read-only and compose the per-VM writable state as a separate layer:

1. Read-only base drive. IsReadOnly: true on the rootfs drive. Firecracker supports this natively, and it is what makes sharing one host file across microVMs safe.
2. Per-VM writable layer. A small scratch drive (or tmpfs) as the overlayfs upperdir, with the base as lowerdir. rootfs_size_mb then describes the writable layer, not the base — the base is sized to its content.
3. Overlay set as its own layer. Ship configs/ as a small read-only drive or via the existing config channel instead of baking it into the base.

Most of the plumbing already exists:

• writeVMConfig already builds a multi-drive list; volumes already map to /dev/vdb../dev/vdz
• there is already a host->guest channel for layout (firework.volumes64= kernel arg, base64 JSON)
• fc-init already mounts arbitrary devices at arbitrary paths, and already mounts tmpfs on /run and /tmp
• runtime.json.writable_paths already declares the paths the application writes to

Result for the example repo: 7 images collapse to 4 distinct bases (es:8.19.11, es:9.3.0, kibana:8.19.11, kibana:9.3.0), and a node hosting both tenant-3 data nodes stores one base plus two small writable layers.

Verified constraints

The guest kernel supports it. firework-deployment-example provisions the guest kernel from the Firecracker CI bucket (spec.ccfc.min/firecracker-ci/<ver>/<arch>/vmlinux-5.10.x). Those kernel configs have CONFIG_OVERLAY_FS=y and CONFIG_SQUASHFS=y on both x86_64 and aarch64. Worth re-confirming against the exact pinned 5.10.x, but the design is not kernel-blocked.

The cheap shortcut does not work. Mounting a bare tmpfs over just the writable_paths subtrees (avoiding a whole-root overlay and switch_root) fails, because docker-to-rootfs.sh appends the image's WorkingDir to writable_paths unconditionally, and for these images WorkingDir is the entire application tree (/usr/share/elasticsearch, /usr/share/kibana). A bare tmpfs there hides the application. Real overlayfs with the base as lowerdir is required, which means restructuring the fc-init init sequence around switch_root.

Main risk

writable_paths is currently a hardcoded ES/Kibana heuristic plus WorkingDir and Docker VOLUMEs. Today an image that writes outside that list silently works, because the whole rootfs is writable. With a read-only base, every such write becomes a boot failure — for arbitrary images, not just these two. Making writable_paths derived and verified rather than hardcoded should be treated as a prerequisite, not a follow-up.

Scope

This spans two repos and is not a GitOps-only change:

• core: fc-init init sequence (overlay mount + switch_root), writeVMConfig drive list, enricher image resolution
• firework-gitops-example: service schema, since source_image / rootfs_size_mb / overlays would need to be declared as separable inputs rather than fused at build time

Suggested sequencing

1. Make writable_paths derived and verified (prerequisite, valuable on its own).
2. Read-only base + overlayfs + per-VM writable layer in fc-init / writeVMConfig.
3. Separate the overlay set into its own layer and decouple rootfs_size_mb from the base.

A cheaper interim step that does not touch core: deduplicate identical builds inside firework-gitops-example's build script (build once per content key, copy for the rest). That captures the CI-side win with no runtime risk and does not foreclose any of the above.