Repository navigation
FFmpeg core: no hang on video gaps, decoder detection, more formats - #19753
Merged
Merged
Conversation
Playing a file whose video track has a gap of more than about two seconds (frames dropped while recording, or any variable frame rate source) froze RetroArch at that point. The main thread waits for the next frame, and the decode thread only decodes a video packet once audio has caught up to it, so across the gap it decodes audio that nobody takes until the audio queue is full, then blocks too. Wait for the next frame as before, but give up once the decoder has a second or more of audio buffered ahead, which only happens across a gap: hold the frame on screen, keep playing audio, and pick the next frame up when the decoder reaches it. The interpolation factor is clamped, since a held frame would push it outside [0, 1]. ffmpeg_sweep_test plays a clip with a 4 s hole in its video track and checks audio and video both carry on past it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tpe3orhELpViSQpZcsTTjr
avcodec_find_decoder() returns one decoder per codec id, whichever the
build registered first, so which decoder played a file depended on
registration order and on what that platform's FFmpeg includes. A
platform's hardware video decoder (h264_nvv4l2, h264_v4l2m2m,
h264_rkmpp, h264_cuvid, h264_mediacodec, ...) was only used if the
build moved it to the front (the L4T FFmpeg patches do, to make every
application use nvv4l2), and a default decoder that can't work there
failed the file even when another decoder for the codec was built in.
For every audio, video and subtitle stream, collect the
non-experimental decoders for its codec and try them in order:
1. video only: platform hardware decoders (AV_CODEC_CAP_HARDWARE),
unless disabled by the new core option 'Use platform hardware
decoders' (ffmpeg_hw_wrappers, default enabled);
2. the default decoder (still libdav1d first for AV1), with the
existing hwaccel set-up for video;
3. any other software decoder.
For audio and video, open each and decode from the start until a
frame comes out, then rewind; keep the first whose frame this core can
use (video: a format swscale accepts, or the hwaccel format it set up,
so a wrapper that only returns DRM-PRIME surfaces is skipped; audio: a
sample format swresample converts). Subtitles, inputs that can't be
rewound, and codecs with a single decoder take the first decoder that
opens. The hwaccel configuration now comes from the decoder actually
being opened. The chosen decoder is logged, and a missing one names
the codec.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tpe3orhELpViSQpZcsTTjr
The player claimed a fixed list of 28 extensions, and the frontend's media type table only routed those to it, so common files (.ogv, .mka, .opus, .aac, .ac3, .aiff, .wv, .ape, .mts, ...) didn't open with the built-in player at all. - The core keeps its list and adds 31 more extensions, each only if this build has the demuxer and, where the extension implies a codec (.ogv Theora, .ape APE, .wv WavPack, .ac3 AC-3, ...), the decoder. RetroArch's platforms ship very different FFmpeg builds, so valid_extensions is built once, at runtime. - libretro_ffmpeg_ffmpeg_core_supports_extension() exposes the same check; msg_hash.c lists the extra extensions as MOVIE_OTHER / MUSIC_OTHER (after everything else, so existing claims win) and path_is_media_type() only routes them to the player when it passes. MPV builds route them unconditionally. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tpe3orhELpViSQpZcsTTjr
… has one owner The codec contexts are freed at unload whatever the FFmpeg version, not only closed, and a hardware device is handed to the codec context that uses it rather than referenced from it, so freeing that context closes the device - every decoder the probing opens and drops included. ffmpeg_sweep_test runs its ASan pass with leak detection: every load and unload in it must give back what it took.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three changes to the built-in media player (
cores/libretro-ffmpeg):Gaps in the video track no longer freeze RetroArch. A recording that dropped frames, or any variable-frame-rate file, has stretches with no video. Across a gap of more than about two seconds, the main thread waited for a frame. Meanwhile the decode thread, which only decodes video once audio has caught up, filled the audio queue and blocked too. The core now holds the frame on screen and keeps playing audio until the next frame arrives.
Decoder detection instead of first-registered.
avcodec_find_decoder()returns whichever decoder the build registered first. A platform's hardware decoder (h264_nvv4l2, v4l2m2m, rkmpp, cuvid, mediacodec, ...) was used only if the build reordered its list, as the L4T FFmpeg patches do. A default decoder that can't work there failed the file even when another one was built in. The core now tries:It keeps the first one that decodes a frame this core can use.
More formats, only when the build can open them. It adds 31 extensions (.ogv, .opus, .mka, .aac, .ac3, .aiff, .wv, .ape, .mts, .ivf, ...), each claimed only if this FFmpeg build has the demuxer and, where the extension implies a codec, the decoder. RetroArch's platforms ship very different FFmpeg builds, so this is checked at runtime.
Commits and the issues they address
Testing
samples/cores/ffmpeg_sweepgains a clip with a 4 s hole in its video track. It checks that audio and video both carry on past the hole.h264_nvv4l2by detection, without the reordered decoder list. I'll confirm before merge.Linux-samples-cores) all pass locally:ffmpeg_sweep(plain, ASan + UBSan, TSan);ffmpeg_finished_wait(ASan, TSan);ffmpeg_audio_pipe(ASan, TSan).Notes
ffmpeg_hw_wrappers("Use platform hardware decoders", default enabled). It applies to video only; the existingffmpeg_hw_decoder(hwaccel) option is unchanged.