Skip to content

Write deletion vectors as valid Puffin files (add file magic + footer)#1094

Closed
raghav-reglobe wants to merge 1 commit into
duckdb:v1.5-variegatafrom
raghav-reglobe:fix-puffin-dv-container
Closed

Write deletion vectors as valid Puffin files (add file magic + footer)#1094
raghav-reglobe wants to merge 1 commit into
duckdb:v1.5-variegatafrom
raghav-reglobe:fix-puffin-dv-container

Conversation

@raghav-reglobe

Copy link
Copy Markdown
Contributor

Problem

IcebergDelete::WriteDeletionVectorFile writes a deletion vector by serializing the
deletion-vector-v1 blob and writing only that blob to the file, while labelling it
file_format = puffin with content_offset = 0 and file_size_bytes == blob size.

That is not a valid Puffin file. The Puffin spec
requires the layout Magic Blob₁ … Blobₙ Footer, with the 4-byte magic
0x50 0x46 0x41 0x31 (PFA1) at the start of the file and a Footer
(Magic, FooterPayload JSON, FooterPayloadSize, Flags, Magic) at the end.
The file produced today has neither the leading magic nor the footer — it is a bare blob.

It works in practice only because every reader (duckdb-iceberg included) locates the
vector via the manifest's content_offset / content_size_in_bytes and never parses the
Puffin container. A spec-compliant Puffin reader — or any tool that enumerates blobs via
the footer, validates the file standalone, or lacks the manifest offsets — cannot read
these files.

Fix

Add IcebergDeletionVectorData::ToPuffinFile, which wraps the blob in a proper container:

Magic(PFA1) | Blob (at offset 4) | Footer{ Magic | FooterPayload(JSON) | FooterPayloadSize(LE i32) | Flags(4) | Magic }

The FooterPayload is a FileMetadata describing the single deletion-vector-v1 blob.
Per the spec, for a deletion vector snapshot-id and sequence-number are -1, the blob
carries the required referenced-data-file and cardinality properties, and it is
uncompressed (no compression-codec, flags = 0).

content_offset is now 4 (the blob begins right after the leading magic). The read path
is unchanged — it is driven by the manifest's content_offset, so existing deletion-vector
files written with content_offset = 0 keep reading correctly. No migration needed.

Validation

  • Footer framing (leading/footer/trailing magic, little-endian FooterPayloadSize, flags)
    is byte-identical to an Iceberg-Java-generated Puffin file
    (empty-puffin-uncompressed.bin): PFA1 | … | <size LE> | 0x00000000 | PFA1. The blob
    metadata field order matches Iceberg-Java
    (type, fields, snapshot-id, sequence-number, offset, length, properties).
  • The deletion-vector-v1 blob is unchanged (ToBlob), so it remains byte-identical to
    the Iceberg reference vector.
  • Both a strict footer-driven reader and the manifest-offset reader recover the same blob.

Testing

test/.../delete/test_deletion_vector_is_valid_puffin.test: after a v3 DELETE, resolves
the deletion-vector file via iceberg_metadata (content = 'POSITION_DELETES') and asserts
the file begins and ends with PFA1 — i.e. it is a valid Puffin container, not a bare
blob. Skipped on Nessie (v3 fixture limitation, like the other v3 delete tests).

WriteDeletionVectorFile wrote only the deletion-vector-v1 blob to the
file and labelled it file_format=puffin, with content_offset=0 and
file_size == blob size. That is not a valid Puffin file: the Puffin spec
requires the layout `Magic Blob... Footer`, with the 4-byte magic
(0x50 0x46 0x41 0x31, "PFA1") at the start and a Footer
(Magic, FooterPayload JSON, FooterPayloadSize, Flags, Magic) at the end.

The bare blob only works because every reader locates the vector via the
manifest's content_offset/content_size_in_bytes and never parses the
Puffin container; a spec-compliant Puffin reader (or any tool that
enumerates blobs via the footer) cannot read these files.

Add IcebergDeletionVectorData::ToPuffinFile, which wraps the blob in a
proper container: leading magic + blob (at offset 4) + footer whose
FileMetadata describes the single deletion-vector-v1 blob. Per the spec,
snapshot-id and sequence-number are -1, and the blob carries the
referenced-data-file and cardinality properties; the blob is
uncompressed. content_offset is now 4. Reads are unchanged: they remain
driven by the manifest's content_offset, so existing DV files written
with content_offset=0 keep reading correctly (no migration).

Footer framing (magic, little-endian FooterPayloadSize, flags) verified
byte-identical to an Iceberg-Java-generated Puffin file; the blob is
unchanged. Adds a test asserting a written deletion-vector file begins
and ends with the Puffin magic.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant