feat(ir): virtualize MP3 files through the Rust IR, as the MP3 profile (§17)

conformance/virtualize/HARNESS.md

@@ -146,14 +146,14 @@ with the first's:
   files, and SAFE directories and Zarr v2, OME-Zarr and N5 stores, it runs the frozen Python profiles in
   `conformance/virtualize/reference/refkv_reference/` (what `refkv.virtualize`
   shipped before these formats moved to the IR, unchanged but for their import
-  paths); for every other profile, the shipped code. MRC and VSI were never
-  implemented outside the core, so they have no reference: their inputs
-  (`no-ref` in the corpus) compare `py` with `web` alone.
+  paths); for every other profile, the shipped code. MRC, VSI and MP3 were
+  never implemented outside the core, so they have no reference: their
+  inputs (`no-ref` in the corpus) compare `py` with `web` alone.
 - `py`: `python -m refkv.virtualize --allow-private-hosts`.
 - `web`: the browser code under Node, `js/conformance/virtualize.ts
   --allow-private-hosts`.
 
-For TIFF (NDPI included), ND2, CZI, DICOM, NIfTI, MRC, IMS, VSI and SAFE files, and SAFE directories and Zarr v2, OME-Zarr and N5 stores, `py` and `web` are one implementation: the Rust core
+For TIFF (NDPI included), ND2, CZI, DICOM, NIfTI, MRC, IMS, VSI, MP3 and SAFE files, and SAFE directories and Zarr v2, OME-Zarr and N5 stores, `py` and `web` are one implementation: the Rust core
 `rust/refkv-ir` (its parsers, schemas, read planner, projections and mirror),
 built natively for Python and as wasm32 for the browser, each with a thin host
 (the I/O, and the archive writer). Their `refkv_source` is the IR's mirror,

conformance/virtualize/compare.py

@@ -456,7 +456,7 @@ def check(item: tuple, impls: dict, out_dir: Path) -> dict:
     name, url = item[:2]
     if len(item) > 2 and item[2] == "py-only":  # past the browser's limit: every implementation but web
         impls = {k: v for k, v in impls.items() if k != "web"}
-    if len(item) > 2 and item[2] == "no-ref":  # a profile the frozen reference never had (MRC, VSI): py and web alone
+    if len(item) > 2 and item[2] == "no-ref":  # a profile the frozen reference never had (MRC, VSI, MP3): py and web alone
         impls = {k: v for k, v in impls.items() if k != "ref"}
     results = {n: run(impl, url, out_dir / f"{name}.{n}.out") for n, impl in impls.items()}
     ref_name = next(iter(impls))
@@ -575,7 +575,7 @@ def corpus(proxy: Proxy, fixtures: Path, quick: bool, local_only: bool, large: b
     stores = store_fixtures(fixtures)
     files = [p for p in sorted(p for s in FILE_SUFFIXES for p in fixtures.rglob(f"*{s}"))
              if not any(p.is_relative_to(s) for s in stores)]
-    # a profile the frozen reference never had (MRC): its files compare py with web alone
+    # a profile the frozen reference never had (MRC, VSI, MP3): its files compare py with web alone
     items = [(f"fixture-{p.stem}", proxy.local(p.relative_to(fixtures).as_posix()),
               *(["no-ref"] if FILE_SUFFIXES[p.suffix]["reference"] is None else []))
              for p in files]

conformance/virtualize/corpus_mp3.txt

@@ -0,0 +1,5 @@
+# url|name|no-ref of the public MP3 files in the corpus (the frozen reference has no MP3 profile, so each
+# compares py with web alone): a LibriVox recording on the Internet Archive (public domain): chapter 20
+# of "Public Opinion" by Walter Lippmann, MPEG-2 mono at 22.05 kHz, 64 kbps, with an Info frame and tags
+# (archive.org redirects downloads to a mirror)
+https://archive.org/download/public_opinion_1312_librivox/publicopinion_20_lippmann_64kb.mp3|librivox_lippmann_20|no-ref

demo/index.html

@@ -58,7 +58,7 @@
   <h1>Image files to Zarr in the browser</h1>
   <p class="lede">Paste the URL of an image file, a Zarr or N5 store, or a .vzip archive.
     <!-- profiles:demo-lede -->
-    TIFF, OME-TIFF, Aperio SVS, Hamamatsu NDPI, Nikon ND2, DICOM, NIfTI-1 and NIfTI-2, Imaris IMS, Zeiss CZI, MRC2014, Olympus VSI and Akoya QPTIFF files, N5, Zarr v2 and OME-Zarr 0.4 stores, and Sentinel-2 SAFE (Level-1C, Level-2A) products are supported;
+    TIFF, OME-TIFF, Aperio SVS, Hamamatsu NDPI, Nikon ND2, DICOM, NIfTI-1 and NIfTI-2, Imaris IMS, Zeiss CZI, MRC2014, Olympus VSI, Akoya QPTIFF and MP3 (MPEG Layer III audio) files, N5, Zarr v2 and OME-Zarr 0.4 stores, and Sentinel-2 SAFE (Level-1C, Level-2A) products are supported;
     <!-- /profiles -->
     the format is detected from the file's
     first bytes, or the store's metadata. A service worker reads the input's structure with a few range

docs/virtualize/README.md

@@ -33,14 +33,17 @@ who each one is for, what you get, and how to run it.
 | MRC | MRC2014 maps, image stacks, volumes and volume stacks (`.mrc`, `.map`) | [mrc](mrc/README.md) | [§14](../../spec/virtualize/mrc.md#part-2-the-profile) |
 | VSI | Olympus cellSens whole-slide images (`.vsi` with its tile files) | [vsi](vsi/README.md) | [§15](../../spec/virtualize/vsi.md#part-2-the-profile) |
 | QPTIFF | Akoya (PerkinElmer) Vectra multiplex scans and inForm component data | [qptiff](qptiff/README.md) | [§16](../../spec/virtualize/qptiff.md#part-2-the-profile) |
+| MP3 | MP3 audio files, as an array of samples (`.mp3`) | [mp3](mp3/README.md) | [§17](../../spec/virtualize/mp3.md#part-2-the-profile) |
 <!-- /profiles -->
 
 A format that reads a file takes a URL, or a local path. One that reads a
 store takes a URL ending in `/` that answers the S3 `ListObjectsV2` operation
 anonymously, such as a public S3 bucket. SAFE reads either: a product's
 `.SAFE` directory as a store, or its `.SAFE.zip` as a file. VSI is named by
 its `.vsi` file and reads the tile files beside it by name, without listing
-the directory.
+the directory. MP3 is the one format that is not an image: its archive holds
+a plain Zarr array of audio samples, decoded from the file's frames when a
+chunk is read ([mp3](mp3/README.md)).
 
 SAFE and CZI have no user-story page yet; their conventions describe the
 output. A Sentinel-2 product becomes a GeoZarr hierarchy (one group per
@@ -235,6 +238,11 @@ each format's metadata is:
   metadata tree as JSON, the list of stacks and the file's size; each
   image keeps its stack's properties (scale, origin, channels, creation
   time, magnification) and its tile file's headers.
+- **MP3:** the IR mirror on `refkv_source`: the ID3v2 tags, the Xing or
+  Info frame, every audio frame, and the ID3v1 and APE tags, so that the
+  file is rebuilt from the archive. The root keeps the stream's parameters
+  (version, sample rate, channels, frames, bitrate), the info frame's
+  counts and LAME fields, and the ID3v1 tag's fields.
 - **N5, Zarr v2 and OME-Zarr:** each node's own attributes and the layout
   members its `zarr.json` does not reproduce, on the node, as
   `{"attributes": ..., "metadata": ...}` (with `"unversioned"` for an

docs/virtualize/mp3/README.md

@@ -0,0 +1,161 @@
+# MP3
+
+refkv turns a file into a Zarr dataset by writing a small archive of
+references to the file's bytes, without copying its content
+([overview](../README.md)). This page is about MP3 files: the one format
+refkv virtualizes that is not an image.
+
+## Who this is for
+
+MP3 (`.mp3`: MPEG-1, MPEG-2 and MPEG-2.5 Layer III audio) is how a great
+deal of recorded sound is stored and shared: speech corpora, field
+recordings, podcasts, audiobooks, music. Someone who wants to read a
+minute of a long recording lazily over HTTP as a NumPy array, put a
+collection of recordings next to other arrays in a Zarr store, or run an
+array pipeline over audio without first decoding every file to PCM on disk
+has to convert it today.
+
+An MP3 file is a sequence of frames, each decoding to 1152 (MPEG-1) or 576
+samples per channel, with tags before and after them. refkv reads the file
+once, finds every frame, and writes an archive in which every Zarr chunk is
+a run of 256 frames, referenced together with the few frames before it that
+a decoder needs (MP3 frames borrow bits from their predecessors). The
+archive's `mp3` codec decodes a chunk when it is read, so the samples are
+never stored: they stay in the file as MPEG frames.
+
+## What you get
+
+A group at the archive root with one array, `samples`.
+
+- **Shape:** `[samples, channels]`, with the dimension names `time` and
+  `channel`: one row per sample, one column per channel (1 or 2).
+- **Data type:** `int16`, the decoder's 16-bit output. The array holds
+  every sample the frames decode to, the encoder's delay and padding
+  included (their counts are in the metadata, for a reader that wants to
+  trim them, as players do).
+- **Chunks:** 256 frames each (294912 samples for MPEG-1, 147456 for
+  MPEG-2 and 2.5; 6.7 seconds at 44.1 kHz, 6.7 at 22.05), the last one
+  shorter. Reading a chunk reads its frames and a few before them (the
+  bit reservoir) in one request, and decodes them in one pass.
+- **Codecs:** `mp3`, which refkv's `zarr` module registers and which
+  decodes with miniaudio (the `audio` extra). No other codec.
+- **Metadata:** under `refkv_virtualized.mp3` on the root: the stream's
+  version, sample rate, channel count and mode, frame and sample counts,
+  duration and bitrate (constant or variable), whether frames carry CRCs;
+  the Xing or Info frame's frame and byte counts, encoder (`LAME3.99r`),
+  delay and padding; the ID3v2 tags' versions and sizes; the ID3v1 tag's
+  title, artist, album, year, comment, track and genre; an APE tag's
+  version and item count. The whole file (the tags, the info frame, every
+  frame and any bytes between them) is described on `refkv_source` by the
+  Rust core's IR mirror, from which it is rebuilt byte for byte.
+- **No `ome` metadata:** the hierarchy is not an image, so OME-Zarr
+  viewers do not open it; zarr-python, xarray and any Zarr v3 reader with
+  the codec do.
+
+**What is not decoded.** The ID3v2 tag's frames (title, cover art, lyrics)
+and an APE tag's items are kept whole in the mirror, not translated; the
+samples a decoder gives are not bit-exact across decoders (the standards
+allow a difference of a few units); a stream whose frames borrow bits from
+frames that are not there (two files joined) decodes as the reader's
+decoder has it. [spec/virtualize/mp3.md §7](../../../spec/virtualize/mp3.md#7-what-the-layout-does-not-express)
+lists them.
+
+## Try it
+
+A LibriVox recording from the Internet Archive (chapter 20 of Walter
+Lippmann's *Public Opinion*, public domain, 3.9 MB), from
+[corpus_mp3.txt](../../../conformance/virtualize/corpus_mp3.txt):
+
+```bash
+uv run python -m refkv.virtualize https://archive.org/download/public_opinion_1312_librivox/publicopinion_20_lippmann_64kb.mp3 lippmann_20.vzip
+```
+
+```
+{"format": "mp3", "version": "MPEG-2", "sampleRate": 22050, "channels": 1, "frames": 18576, "samples": 10699776, "duration": 485.250612244898, "bitrate": 64, "chunkFrames": 256, "chunks": 73, "info": "Info", "tags": ["id3v2", "id3v1"], "strayBytes": 0, ...}
+```
+
+The archive is 17 KB: 73 chunk references and the mirror's table of 18576
+frames. Read the recording:
+
+```python
+import zarr
+import refkv.zarr
+
+g = zarr.open_group(refkv.zarr.open("lippmann_20.vzip"), mode="r")
+m = g.attrs["refkv_virtualized"]["mp3"]
+print({k: m[k] for k in ("version", "sample_rate", "channels", "duration")})
+print(m["info"])
+print(m["id3v1"])
+a = g["samples"]
+print(a.shape, a.dtype, a.chunks)
+print(a[22050 * 60:22050 * 60 + 8, 0])
+```
+
+```
+{'version': 'MPEG-2', 'sample_rate': 22050, 'channels': 1, 'duration': 485.250612244898}
+{'tag': 'Info', 'frames': 18576, 'bytes': 3882212, 'quality': 57, 'encoder': 'LAME3.99r', 'delay': 576, 'padding': 1128}
+{'title': '20 - A New Image', 'artist': 'Walter Lippmann', 'album': 'Public Opinion', 'year': '', 'comment': 'http://archive.org/details/p', 'track': 20, 'genre': 101}
+(10699776, 1) int16 (147456, 1)
+[11 -2  6  2  1 15 20 19]
+```
+
+The last line reads the samples one minute in: one request for the chunk's
+frames (about 54 KB), decoded on the fly. The same samples, as
+`ffmpeg -i publicopinion_20_lippmann_64kb.mp3 -f s16le -` gives them after
+its gapless trimming, start `delay + 529` samples later in its output.
+
+**In the browser.** The service worker serves a virtualized MP3 as a Zarr
+hierarchy like any other; the demo page does not display audio.
+
+## Supported and not supported
+
+Supported:
+
+- MPEG-1, MPEG-2 and MPEG-2.5 Layer III streams, mono and stereo (joint,
+  dual channel), constant and variable bitrates, with or without CRCs,
+  with or without the bit reservoir.
+- Xing, Info and VBRI frames (read, not counted as audio), LAME tags,
+  ID3v2 tags (several, with footers), ID3v1 and APE tags.
+- Junk between frames, a truncated last frame and bytes after the last
+  frame: skipped, kept in the mirror, counted in `stray_bytes`.
+
+Rejected:
+
+- Layer I and Layer II streams (`MPEG-2 Layer II (the frame at 0) is not
+  supported: only Layer III is`), and free-format bitrates.
+- A stream whose frames change version, sample rate or channel count
+  (`the frame at 2016 changes the sample rate from 16000 to 22050`): two
+  files concatenated, for instance.
+- A file with no audio frames, and an ID3v2 tag that runs past the file
+  or whose size is not syncsafe.
+
+## Performance
+
+- Virtualizing reads the whole file (an MP3 has no index), in one or a few
+  large range requests after one request for the size. The 3.9 MB file
+  above takes under a second over HTTP.
+- Reading a chunk is one request of the chunk's frames and their lead-in
+  (two frames, plus the fewest frames holding 511 bytes of main data for
+  MPEG-1 or 255 for MPEG-2: about 3 to 8 frames, a few percent of the
+  chunk), then decoding them.
+- The archive has one entry per chunk, and the mirror's table has a row
+  per frame, folded into one run when the frames are all of one length
+  (the file above: 17 KB in all), a few bytes per frame otherwise.
+
+## How it's verified
+
+`js/test/mp3/verify.py` virtualizes the synthetic MP3 files (including
+inputs to reject) with the browser code, reads them back through
+`python/src/refkv`, zarr-python and the `mp3` codec, and compares every
+sample with miniaudio's decoding of the whole stream (exactly) and with
+ffmpeg's (within a few units, where the generator stored it).
+`python/tests/ir/test_mp3.py` does the same through Python, with chunks of
+4 frames so that every file has several chunks and lead-ins. `compare.py`
+checks that the Python and browser outputs are equivalent on those files
+and on the public file of
+[corpus_mp3.txt](../../../conformance/virtualize/corpus_mp3.txt).
+
+- Profile: [spec/virtualize/mp3.md](../../../spec/virtualize/mp3.md#part-2-the-profile)
+- Rust core: [rust/refkv-ir/src/mp3.rs](../../../rust/refkv-ir/src/mp3.rs)
+- Codec: [python/src/refkv/zarr/codecs.py](../../../python/src/refkv/zarr/codecs.py)
+- Fixtures: [fixtures/mp3/](../../../fixtures/mp3/)