erb format specs

README.md

@@ -10,7 +10,7 @@ Historical data providers offer **three primary storage formats**:
 - [**Era**(Beacon Chain History)](./formats/era.md) - Stores data from the genesis of the Beacon Chain onwards. Can be used by Execution layer clients for history **from The Merge onward**, including historical block data.
 - [**E2SS**(Execution State)](./formats/e2ss.md) - **State snapshots** for execution clients.
 - [**E2HS**(Execution Layer History)](./formats/e2hs.md) - **full execution layer history** for execution clients, provides data from genesis to latest, headers are accompanied by proofs of canonicalness.
-- **Erb**(Blob) - Era file equivalent for blob sidecars [ ⚠️ Under Development ].
+- [**Erb**(Blob)](./formats/erb.md) - Era file equivalent for blob data; stores blobs and KZG proofs paired with an `.era` file by filename.
 
 ## E2store Types
 No e2store type may be reused. A list of all defined E2store types can be found at [types/README.md](./types/README.md)

formats/erb.md

@@ -0,0 +1,239 @@
+# Erb files
+
+Erb files are e2store files which store blobs and their KZG proofs. For more
+information on the underlying e2store format, see
+https://github.com/status-im/nimbus-eth2/blob/stable/docs/e2store.md.
+
+The overall structure of an Erb file follows closely the structure of an
+[Era](./era.md) file, grouped per `SLOTS_PER_HISTORICAL_ROOT` slots with a
+trailing `SlotIndex` for random-access reads. The per-slot payload is the
+list of blobs and KZG proofs attached to the beacon block at that slot.
+
+In examples, we assume the mainnet configuration:
+`SLOTS_PER_HISTORICAL_ROOT == 8192` and
+`MAX_BLOB_COMMITMENTS_PER_BLOCK == 4096`.
+
+## File name
+
+```
+<config-name>-<era-number>-<short-era-root>.erb
+```
+
+`config-name`, `era-number` and `short-era-root` are computed exactly as for
+the corresponding `.era` file (see [era.md](./era.md)). The matching
+`<config-name>-<era-number>-<short-era-root>.era` and
+`<config-name>-<era-number>-<short-era-root>.erb` files share an identical
+suffix and pair by file name alone.
+
+## Structure
+
+```
+erb           := group+
+group         := Version | blobs* | other-entries* | SlotIndex(blobs)
+blobs         := CompressedBlobsAndProofs
+```
+
+```
+Version                 = { type: [0x65, 0x32], data: nil }
+CompressedBlobsAndProofs = { type: [0x0c, 0x00], data: snappyFramed(ssz(BlobsAndProofs)) }
+SlotIndex               = { type: [0x69, 0x32], data: starting-slot | offset | offset | ... | count }
+```
+
+```python
+class BlobsAndProofs(Container):
+    blobs:      List[Blob, MAX_BLOB_COMMITMENTS_PER_BLOCK]
+    kzg_proofs: List[KZGProof, MAX_BLOB_COMMITMENTS_PER_BLOCK]
+```
+
+Differences from `.era`:
+
+* the per-slot payload is `CompressedBlobsAndProofs` instead of
+  `CompressedSignedBeaconBlock`
+* there is no state record and no `slot-index(state)` - the bundled KZG
+  proofs validate every blob against `block.body.blob_kzg_commitments`,
+  which anchor verification to the matching `.era` file without a separate
+  per-file commitment
+
+There is at most one `CompressedBlobsAndProofs` entry per slot. The entry
+contains every blob attached to the beacon block at that slot, in `index`
+order, alongside its matching KZG proof. Slots with no blobs
+are omitted; the corresponding `SlotIndex` entry is `0`.
+
+`kzg_proofs[i]` is the blob KZG proof of `blobs[i]`, as verified by
+`verify_blob_kzg_proof`, for all forks. Cell proofs are never stored;
+post-Fulu writers compute blob proofs with `compute_blob_kzg_proof`.
+
+Erb files exist only for eras starting at or after `DENEB_FORK_EPOCH`.
+Fork epochs are era-aligned on all public networks; the first mainnet erb
+file is `mainnet-01053-<short-era-root>.erb`.
+
+`other-entries` is an extension point for future record types. Unknown record
+types must be skipped.
+
+## Verifying erb files
+
+Against the paired `.era` file, for every slot `s` in
+`[starting_slot, starting_slot + 8192)`:
+
+* let `block = era[s]` (if any) and `bundle = erb[s]` (absent if the
+  `SlotIndex` offset is `0`)
+* `bundle` is present if and only if `block.body.blob_kzg_commitments`
+  is non-empty
+* `len(bundle.blobs) == len(block.body.blob_kzg_commitments)`
+* `len(bundle.kzg_proofs) == len(bundle.blobs)`
+* `verify_blob_kzg_proof_batch(bundle.blobs, block.body.blob_kzg_commitments, bundle.kzg_proofs)` succeeds
+
+## Reading erb files
+
+```python
+import io
+
+import snappy  # python-snappy; supports framed snappy via stream_decompress
+
+E2S_HEADER_LEN = 8
+
+TYPE_VERSION                  = b"\x65\x32"
+TYPE_SLOT_INDEX               = b"\x69\x32"
+TYPE_COMPRESSED_BLOBS_AND_PROOFS = b"\x0c\x00"
+
+
+def read_entry_header(f):
+    hdr = f.read(E2S_HEADER_LEN)
+    if len(hdr) != E2S_HEADER_LEN:
+        return None
+    return hdr[:2], int.from_bytes(hdr[2:8], "little", signed=False)
+
+
+def read_slot_index(f):
+    """
+    Read the trailing SlotIndex record assuming f is positioned at the *end*
+    of that record. Returns (starting_slot, index_record_start, offsets) and
+    leaves f positioned at index_record_start.
+    """
+    end = f.tell()
+
+    f.seek(end - 8)
+    count = int.from_bytes(f.read(8), "little", signed=False)
+
+    record_size = E2S_HEADER_LEN + 8 + count * 8 + 8
+    start = end - record_size
+
+    f.seek(start)
+    typ, length = read_entry_header(f)
+    assert typ == TYPE_SLOT_INDEX, f"expected SlotIndex, got {typ.hex()}"
+    assert length == 8 + count * 8 + 8
+
+    starting_slot = int.from_bytes(f.read(8), "little", signed=False)
+    offsets = [
+        int.from_bytes(f.read(8), "little", signed=True) for _ in range(count)
+    ]
+    count_again = int.from_bytes(f.read(8), "little", signed=False)
+    assert count_again == count
+
+    f.seek(start)
+    return starting_slot, start, offsets
+
+
+def snappy_frame_decompress(data):
+    src, dst = io.BytesIO(data), io.BytesIO()
+    snappy.stream_decompress(src, dst)
+    return dst.getvalue()
+
+
+def read_blobs_at_slot(f, slot, starting_slot, index_record_start, offsets):
+    """
+    Random-access read. Returns the SSZ-encoded
+    BlobsAndProofs bytes (fork-defined
+    element type), or None if the slot has no entry.
+    """
+    rel = slot - starting_slot
+    if rel < 0 or rel >= len(offsets):
+        return None
+    rel_offset = offsets[rel]
+    if rel_offset == 0:
+        return None
+
+    f.seek(index_record_start + rel_offset)
+    hdr = read_entry_header(f)
+    assert hdr is not None
+    typ, length = hdr
+    assert typ == TYPE_COMPRESSED_BLOBS_AND_PROOFS, (
+        f"expected CompressedBlobsAndProofs, got {typ.hex()}"
+    )
+    return snappy_frame_decompress(f.read(length))
+
+
+def read_erb_file(name):
+    """Walk an erb file backwards, group by group, printing a summary."""
+    with open(name, "rb") as f:
+        f.seek(0, 2)
+        total_groups = 0
+
+        while f.tell() > E2S_HEADER_LEN:
+            starting_slot, index_start, offsets = read_slot_index(f)
+
+            non_zero = [o for o in offsets if o != 0]
+            print(
+                "Group starting slot:", starting_slot,
+                "| index at:", index_start,
+                "| slots indexed:", len(offsets),
+                "| populated:", len(non_zero),
+            )
+
+            for rel, off in enumerate(offsets):
+                if off != 0:
+                    blob_ssz = read_blobs_at_slot(
+                        f, starting_slot + rel,
+                        starting_slot, index_start, offsets,
+                    )
+                    print(
+                        "  first populated slot:", starting_slot + rel,
+                        "ssz bytes:", len(blob_ssz),
+                    )
+                    break
+
+            # The Version record sits at the very front of this group;
+            # stepping over its header lands us at the end of the previous
+            # group, where its trailing SlotIndex is.
+            if non_zero:
+                prev_group_end = (index_start + non_zero[0]) - E2S_HEADER_LEN
+            else:
+                prev_group_end = index_start - E2S_HEADER_LEN
+
+            f.seek(prev_group_end)
+            total_groups += 1
+
+        print("Total groups in file:", total_groups)
+```
+
+## FAQ
+
+### Why one entry per slot rather than one entry per blob?
+
+A single offset per slot keeps the `SlotIndex` semantics identical to `.era`
+and matches how blobs are already grouped by block in the consensus
+protocol. Within a slot's entry the blobs are stored in `index` order, so
+individual blobs are still addressable after one decompression.
+
+### Why no accumulator entry?
+
+Each `CompressedBlobsAndProofs` entry carries the matching `kzg_proofs`
+that anchor each blob to its commitment in the corresponding
+beacon block. Combined with the matching `.era` file, this is sufficient to
+verify every blob without an extra per-file commitment.
+
+### Why blob KZG proofs rather than cell proofs?
+
+They are compact, fork-uniform, and verifiable in a single
+`verify_blob_kzg_proof_batch` call against the paired `.era` commitments.
+Consumers needing cells can recompute them from the blobs.
+
+### Why share the `.era` file's short root in the name?
+
+So a downloader can fetch `mainnet-00269-<root>.era` and
+`mainnet-00269-<root>.erb` together by suffix and know they describe the
+same canonical history. The cost is that erb generation requires either the
+matching era file or the beacon state at end-of-era to compute the short
+root - a self-contained alternative (e.g. the last beacon block root from
+the last populated sidecar entry, like `e2hs` does) would avoid that
+dependency at the cost of full file-name pairing.

types/0x0c00.md

@@ -0,0 +1,22 @@
+# CompressedBlobsAndProofs
+
+```
+type: [0x0c, 0x00]
+data: snappyFramed(ssz(BlobsAndProofs))
+```
+
+`CompressedBlobsAndProofs` entries carry the blobs and matching KZG proofs
+attached to a single beacon block, snappy-framed SSZ encoded.
+
+```
+class BlobsAndProofs(Container):
+    blobs:      List[Blob, MAX_BLOB_COMMITMENTS_PER_BLOCK]
+    kzg_proofs: List[KZGProof, MAX_BLOB_COMMITMENTS_PER_BLOCK]
+```
+
+`blobs[i]` is the blob at index `i` in the corresponding block's
+`body.blob_kzg_commitments`. `kzg_proofs[i]` is its KZG proof, suitable for
+`verify_blob_kzg_proof_batch` against the block's commitments.
+
+`kzg_proofs[i]` is the blob KZG proof for all forks; cell proofs are never
+stored.

types/README.md

@@ -16,6 +16,7 @@ This is a sorted list by type number
 | [0x0900](0x0900.md)  | CompressedStorage | [E2SS](../formats/e2ss.md)  |   |
 | [0x0a00](0x0a00.md)  | CompressedSlimReceipts | [Ere](../formats/ere.md)  |   |
 | [0x0b00](0x0b00.md)  | Proof | [Ere](../formats/ere.md)  |   |
+| [0x0c00](0x0c00.md)  | CompressedBlobsAndProofs | [Erb](../formats/erb.md)  |   |
 | [0x6532](0x6532.md)  | Version | ALL  |   |
 | [0x6632](0x6632.md)  | BlockIndex | [Era1](../formats/era1.md), [E2HS](../formats/e2hs.ms)  |   |
 | [0x6732](0x6732.md)  | DynamicBlockIndex | [Ere](../formats/ere.md)  |   |