Skip to content

feat(ir)!: add the refkv.stream codec, port MP3 onto it, and virtualize MP4 files as the MP4 profile (§19) - #99

Open
d-v-b wants to merge 2 commits into
mainfrom
claude/stream-codec
Open

feat(ir)!: add the refkv.stream codec, port MP3 onto it, and virtualize MP4 files as the MP4 profile (§19)#99
d-v-b wants to merge 2 commits into
mainfrom
claude/stream-codec

Conversation

@d-v-b

@d-v-b d-v-b commented Oct 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

This adds refkv.stream (conventions §10), one Zarr codec for audio and video streams whose frames depend on earlier frames, and moves MP3 onto it. It also adds the MP4 profile: H.264 video tracks become Y, U and V planes with presentation times, and AAC and MP3 tracks become float32 samples, all read through the moov box alone.

What the implementation changed in the design

  • Positions, not skip and keep. Each chunk's header gives every coded frame its own output position (−1 for lead-in frames). The codec tags each packet with its index and places each output by the frame it came from, so a decoder's reordering or a dropped frame cannot shift samples.
  • Bounded-memory decoders only. ffmpeg's fixed-point MP3 decoder carries rounding error forward, so chunks differ from a whole-stream decode by one unit. miniaudio silently drops a lead-in frame whose reservoir is missing, so its output cannot be attributed frame by frame. ffmpeg's float decoder (through PyAV) is exact on every fixture, so MP3 is now float32 and miniaudio is gone. This is a breaking change for MP3 archives.
  • AAC noise substitution is never exact. Its bands come from a random generator whose state runs forever. With it off, a one-frame lead-in is exact; with it on, a chunk is repeatable but its noise bands differ from a continuous decode. The spec says so.
  • Varint headers. A fixed 8 bytes per frame doubled the MP3 example's archive. Varints halve that; the remaining cost is documented.

Verified: the Rust core's tests; pytest (2149 passed); the Node tests (638); just sections; js/test/mp3/verify.py and js/test/mp4/verify.py (0 failures; every video plane, time and audio sample equal to ffmpeg's decoding, except AAC's noise bands); just cli-test; just cli-identical --only mp (558 conversions identical). A picture of the public Prelinger film, read over HTTP through the archive, equals ffmpeg's.

Known costs, not changed

  • The 46 MB Prelinger MP4's archive is 2.4 MB. Two thirds of it is the three planes' identical references, with one range per picture because audio is interleaved.
  • Reading one picture decodes its whole chunk, up to 256 pictures plus the lead-in, about 2 seconds for that file over HTTP.
  • python/src/refkv/zarr/codecs.py imports PyAV inside _av(), deliberately, like the existing imagecodecs import for an optional extra.
Stack bookkeeping

Based on main after #91 (MP3) and #92 (User-Agent); independent of other open PRs. Main was merged in after #88 (FITS) took §18, so the MP4 profile is §19. The MP4 and MP3 fixture generators need ffmpeg (and lame), so just fixtures skips them like NDPI's; just fixtures-of mp4 regenerates.

🤖 Generated with Claude Code

d-v-b and others added 2 commits October 23, 2026 22:04
@d-v-b @claude
feat(ir)!: add the refkv.stream codec, port MP3 onto it, and virtuali…
a22533d 
…ze MP4 files as the MP4 profile (§18)

Add `refkv.stream` (conventions §10), one array-to-bytes codec for audio and video streams
whose frames depend on earlier ones: a chunk is a varint header giving each coded frame's size
and output position (-1 for the lead-in frames a decoder needs first), then the frames, decoded
in one pass by ffmpeg's decoders through PyAV, each packet tagged with its index so that every
output is placed by the frame it came from, in whatever order the decoder emits it. The `media`
extra (PyAV) replaces `audio` (miniaudio).

MP3 (breaking): the array is float32 and its codec `refkv.stream` with format mp3. ffmpeg's
fixed-point MP3 decoder carries rounding error from frame to frame and miniaudio drops a lead-in
frame whose reservoir is missing, so neither can decode a chunk exactly; the float decoder does.

MP4 (spec/virtualize/mp4.md, §18): an MP4 or QuickTime file is read through its moov box alone
(never the media). Every box is an IR struct with its fields and tables, and every sample a data
element; each H.264 track (8-bit 4:2:0, closed groups) becomes Y, U and V planes and presentation
times, each AAC-LC or MP3 track float32 samples, under tracks/<id>; other tracks are listed with
the reason. A video chunk decodes from the sync sample before it, an AAC chunk from the frame
before it. Synthetic fixtures (fixtures/mp4, 17 accepted, 8 rejected, written by ffmpeg) are
checked against ffmpeg's decoding by python/tests/ir/test_mp4.py and js/test/mp4/verify.py, value
for value but for AAC's noise-substituted bands; corpus_mp4.txt lists a Prelinger film.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@d-v-b @claude
Merge origin/main (the FITS profile, §18) into claude/stream-codec; t…
b3b26ef 
…he MP4 profile becomes §19

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@d-v-b d-v-b changed the title feat(ir)!: add the refkv.stream codec, port MP3 onto it, and virtualize MP4 files as the MP4 profile (§18) feat(ir)!: add the refkv.stream codec, port MP3 onto it, and virtualize MP4 files as the MP4 profile (§19) Oct 24, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@d-v-b