composefs backend: zstd:chunked images fail to pull with "unexpected EOF reading tar entry"

Environment

bootc 1.16.9-1.fc44, composefs backend, sealed UKI + systemd-boot
Image: OCI, layers application/vnd.oci.image.layer.v1.tar+zstd with a zstd:chunked TOC annotation, pushed by buildah push --compression-format zstd:chunked

Reproduction

Publish a bootc image with buildah push --compression-format zstd:chunked.
Run bootc upgrade on a composefs-backend host whose origin references it.

Fails deterministically in seconds:

error: Upgrading composefs: ... Unable to pull container image docker://...:
Failed to pull config Descriptor { media_type: ImageConfig, digest: sha256:073a8cb2...,
size: 23125, ... }: unexpected EOF reading tar entry

The registry side was verified intact before filing: skopeo inspect passes, the named config blob fetches byte-exact (size and sha256 match the descriptor), and podman and skopeo consume the same image without complaint.

Cause (from source, at the composefs-rs rev v1.16.9 pins - 0fbc8533, unchanged on current main f22ecda)

crates/composefs-oci/src/layer.rs decompress_async() routes MediaType::ImageLayerZstd through a plain async_compression ZstdDecoder. A zstd:chunked blob is a multi-frame stream (including skippable frames); the decoder stops at the first frame boundary, truncating the decompressed tar, and crates/composefs-oci/src/tar.rs bails with unexpected EOF reading tar entry. Chunked support for this path is wip: zstd:chunked support via oci-client composefs/composefs-rs#137, a draft dormant since June 2025.
The error is misattributed: crates/composefs-oci/src/skopeo.rs pull() wraps ensure_config_with_layers() - the config fetch and every layer import - in a single with_context("Failed to pull config ..."), so a layer-side failure is reported against the config descriptor. That sent me down an hour of registry forensics on a blob that was byte-perfect. composefs fails with Docker v2s2 images #1703 looks like another failure class surfacing under the same wrapper.

This is not #509: that issue tracks zstd:chunked in the containers-storage/podman subsystem; this is the composefs backend's own pull path (composefs-rs), which podman never exercises.

Why this is worse than a missing feature

A sealed composefs host whose publisher switches to zstd:chunked is stranded: the component that would carry any fix (bootc) only arrives through the pull that fails. Recovery is publisher-side re-push (plain zstd pulls fine) or reinstall - nothing host-side works.

Asks

Split the error context so layer failures name the failing layer, not the config.
Until wip: zstd:chunked support via oci-client composefs/composefs-rs#137 lands: detect the chunked TOC annotation in the manifest before pulling and fail with an explicit "zstd:chunked is not supported by the composefs backend" message.
A line in docs/src/experimental-composefs.md documenting the limitation.

Recovery confirmed from the same host: re-publishing with buildah push --compression-format zstd (no chunked TOC) restored the update channel. The same bootc 1.16.9 composefs host staged the rebuilt image on the first bootc upgrade, 31 s wall for a ~1.4 GB image, and booted it clean, with no host-side action beyond the normal upgrade. That matches the stranding assessment above: the recovery lever is entirely publisher-side.

On ask 2: the release-1.16.10 branch pins the same composefs-rs rev (0fbc8533), so the explicit unsupported-format error would be the only signal a 1.16.10 host gets before hitting the same misattributed failure.

Root cause confirmed at the composefs-rs rev v1.16.9 pins, and unchanged on current main: the plain ZstdDecoder stops at the first frame boundary of a chunked blob, and the pull error context misattributes the failure to the config.

Three follow-ups, smallest first. composefs/composefs-rs#381 enables multi-frame decode so chunked images pull correctly: full pull, no TOC exploitation, the interim described in #509 ("at least doesn't barf on them (but it won't have optimized fetches)"), with decode parity to ostree-ext's Decompressor as suggested in #1703. composefs/composefs-rs#382 splits the error context so layer failures name the failing layer. #2409 adds the Known Issues line until a fixed composefs-rs reaches bootc.

Verified against a skopeo-produced zstd:chunked image: fails at the v1.16.9 pin with exactly the error above, pulls clean with the decode fix. The release-1.16.10 branch carries the same composefs-rs pin, so the limitation ships in it as well.

Thanks for the report, this is conceptually a bug in composefs-rs, so we should have a tracker there, but it makes sense to have one here until we've bumped bootc with a fix.

At least on the bootc side though, unified storage would avoid this issue.

But yeah, we are definitely missing integration tests w/ zstd:chunked here.

Filed the composefs-rs tracker: composefs/composefs-rs#383.

docs/composefs: Document zstd:chunked pull limitation

oci: Decode multi-frame compressed layers

oci: Attribute pull errors to the failing blob

Status from a retest, since this one is gated on a pin rather than on code.

All three asks are addressed upstream now. The error attribution is composefs/composefs-rs#382, the docs caveat is #2409, and the explicit-failure ask is moot because #381 makes full zstd:chunked pulls work rather than needing a pre-flight refusal.

What keeps this open is the pin. bootc v1.16.10, which Fedora 44 installs, vendors composefs-oci 0.7.0 at 0fbc8533, and a zstd:chunked pull against that rev still fails the same way. bootc main already pins v0.9.2, which carries the decoder fix and pulls the same image clean.

So from my side this closes whenever a release carries that pin. Happy to retest on a release candidate if that's useful.

oci: Attribute pull errors to the failing blob

tmt: Add test switching to a zstd:chunked image

Navigation Menu

Repository navigation

composefs backend: zstd:chunked images fail to pull with "unexpected EOF reading tar entry" #2408

Environment

Reproduction

Cause (from source, at the composefs-rs rev v1.16.9 pins - 0fbc8533, unchanged on current main f22ecda)

Why this is worse than a missing feature

Asks

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

composefs backend: zstd:chunked images fail to pull with "unexpected EOF reading tar entry" #2408

Description

Environment

Reproduction

Cause (from source, at the composefs-rs rev v1.16.9 pins - 0fbc8533, unchanged on current main f22ecda)

Why this is worse than a missing feature

Asks

Activity

andrewdunndev commented on Aug 24, 2026

andrewdunndev commented on Aug 24, 2026

cgwalters commented on Aug 24, 2026

andrewdunndev commented on Aug 24, 2026

andrewdunndev commented on Sep 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions