MP4 export loses video configuration changes in HLS

[chriswiggins]

Hey @Vanilagy - we're loving using Mediabunny to browser-side convert HLS playlists into downloadable MP4s on demand. Works great!

Due to where the HLS is generated from (CCTV systems), sometimes the video changes resolution and frame rate during playback - for example, from 1920×1080 to 640×480, then back again. The playlist provides a new init segment for the changed video. We're mostly handling all of that generation Gstreamer-side. As far as we know, we're generating these correctly.

The resulting Mediabunny MP4 plays in VLC, but QuickTime shows black video. macOS’s decoder stops at the first resolution change. Looking at the MP4, it contains only the original video decoder configuration, even though the later frames use a different one.

Could Mediabunny preserve these configuration changes when copying HLS into MP4, i.e by writing multiple sample descriptions and selecting the right one at each change? I was going to throw some tokens at the issue, but wanted your steer and direction first as it might end up touching a few surfaces as the implementation should probably work both ways (i.e mp4 back to hls)

Let me know what you think!

[Vanilagy]

Hi, thank you for the kind words!

Multiple sample descriptions are not supported in ISOBMFF and I'm not sure if I'll add them. FFmpeg only added support for this in July last year, so also very late. It adds additional complexity to the muxer, and more importantly, it warrants fundamental public API changes to the demuxer to be able to communicate to the consumer a custom video decoder description per packet.

I'm not fully ruling this out because the per-packet decoder information is something I considered adding while building the HLS demuxer (to properly surface the changes from discontinuities). If you would like, you can send me a portion of your HLS playlist with a discontinuity in it and I'll take a look. But it's not a major priority for Mediabunny right now.

Alternatives I could recommend: You could try transmuxing to Matroska instead; the input video is likely in Annex B format and this format carries all of its information in-band. Matroska can carry Annex B (unlike MP4), so all of the metadata will be retained. Although I couldn't even tell you off the top of my head if Chromium's VideoDecoder supports changing parameters like this or just errors mid-stream. Alternatively, you can try doing a transcode conversion to normalize every frame to the same resolution.