Skip to content

FFmpeg core: no hang on video gaps, decoder detection, more formats - #19753

Merged
LibretroAdmin merged 4 commits into
libretro:masterfrom
GavinDarkglider:ffmpeg-core
Oct 8, 2026
Merged

LibretroAdmin merged 4 commits into
libretro:masterfrom
GavinDarkglider:ffmpeg-core

Conversation

@GavinDarkglider

@GavinDarkglider GavinDarkglider commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Three changes to the built-in media player (cores/libretro-ffmpeg):

  1. Gaps in the video track no longer freeze RetroArch. A recording that dropped frames, or any variable-frame-rate file, has stretches with no video. Across a gap of more than about two seconds, the main thread waited for a frame. Meanwhile the decode thread, which only decodes video once audio has caught up, filled the audio queue and blocked too. The core now holds the frame on screen and keeps playing audio until the next frame arrives.

  2. Decoder detection instead of first-registered. avcodec_find_decoder() returns whichever decoder the build registered first. A platform's hardware decoder (h264_nvv4l2, v4l2m2m, rkmpp, cuvid, mediacodec, ...) was used only if the build reordered its list, as the L4T FFmpeg patches do. A default decoder that can't work there failed the file even when another one was built in. The core now tries:

    • platform hardware decoders, behind a new option (on by default);
    • the default decoder with the existing hwaccel path (libdav1d still first for AV1, as in Support More Video Files #19583);
    • the other software decoders.

    It keeps the first one that decodes a frame this core can use.

  3. More formats, only when the build can open them. It adds 31 extensions (.ogv, .opus, .mka, .aac, .ac3, .aiff, .wv, .ape, .mts, .ivf, ...), each claimed only if this FFmpeg build has the demuxer and, where the extension implies a codec, the decoder. RetroArch's platforms ship very different FFmpeg builds, so this is checked at runtime.

Commits and the issues they address

# Commit Issues
1 ffmpeg core: don't hang on gaps in the video track no existing report; found playing back recordings with dropped frames (the companion recording PR)
2 ffmpeg core: detect working decoders instead of taking the first #18702 (generalises the AV1 fallback from #19583 to every codec and platform); hardware decoders without patching FFmpeg's registration order
3 ffmpeg core: claim more media formats, but only those the build can open .ogv and others previously failed with "path is not set"

Testing

  • Gap hang regression test. samples/cores/ffmpeg_sweep gains a clip with a 4 s hole in its video track. It checks that audio and video both carry on past the hole.
    • On master the test hangs until the watchdog kills it.
    • With this PR, all 7.5 s of audio play and 30 new frames follow the gap.
  • Gap hang in a full RetroArch build (gl, Xvfb), playing a 12 s file with a 5 s hole:
    • master: froze at the hole and had to be killed;
    • this PR: played through, and the 600-frame run exited normally after 11 s.
  • Decoder detection on x86 with FFmpeg 6.1 and platform hardware decoders enabled:
    • h264 tries h264_v4l2m2m and h264_cuvid, both fail to open, then plays with h264;
    • mpeg4 does the same through mpeg4_v4l2m2m and mpeg4_cuvid;
    • AV1 plays with libdav1d.
    • Not yet run on a Switch: there it should pick h264_nvv4l2 by detection, without the reordered decoder list. I'll confirm before merge.
  • CI lanes (Linux-samples-cores) all pass locally:
    • ffmpeg_sweep (plain, ASan + UBSan, TSan);
    • the standalone core link;
    • ffmpeg_finished_wait (ASan, TSan);
    • ffmpeg_audio_pipe (ASan, TSan).
  • C89: the core stays C89. It has no new library dependencies.

Notes

  • New core option: ffmpeg_hw_wrappers ("Use platform hardware decoders", default enabled). It applies to video only; the existing ffmpeg_hw_decoder (hwaccel) option is unchanged.
  • Probing decodes from the start until the first frame, then rewinds. Subtitles, unseekable inputs and codecs with a single decoder take the first decoder that opens, as before.

GavinDarkglider and others added 4 commits October 8, 2026 17:28
Playing a file whose video track has a gap of more than about two
seconds (frames dropped while recording, or any variable frame rate
source) froze RetroArch at that point. The main thread waits for the
next frame, and the decode thread only decodes a video packet once
audio has caught up to it, so across the gap it decodes audio that
nobody takes until the audio queue is full, then blocks too.

Wait for the next frame as before, but give up once the decoder has a
second or more of audio buffered ahead, which only happens across a
gap: hold the frame on screen, keep playing audio, and pick the next
frame up when the decoder reaches it. The interpolation factor is
clamped, since a held frame would push it outside [0, 1].

ffmpeg_sweep_test plays a clip with a 4 s hole in its video track and
checks audio and video both carry on past it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tpe3orhELpViSQpZcsTTjr
avcodec_find_decoder() returns one decoder per codec id, whichever the
build registered first, so which decoder played a file depended on
registration order and on what that platform's FFmpeg includes. A
platform's hardware video decoder (h264_nvv4l2, h264_v4l2m2m,
h264_rkmpp, h264_cuvid, h264_mediacodec, ...) was only used if the
build moved it to the front (the L4T FFmpeg patches do, to make every
application use nvv4l2), and a default decoder that can't work there
failed the file even when another decoder for the codec was built in.

For every audio, video and subtitle stream, collect the
non-experimental decoders for its codec and try them in order:
  1. video only: platform hardware decoders (AV_CODEC_CAP_HARDWARE),
     unless disabled by the new core option 'Use platform hardware
     decoders' (ffmpeg_hw_wrappers, default enabled);
  2. the default decoder (still libdav1d first for AV1), with the
     existing hwaccel set-up for video;
  3. any other software decoder.
For audio and video, open each and decode from the start until a
frame comes out, then rewind; keep the first whose frame this core can
use (video: a format swscale accepts, or the hwaccel format it set up,
so a wrapper that only returns DRM-PRIME surfaces is skipped; audio: a
sample format swresample converts). Subtitles, inputs that can't be
rewound, and codecs with a single decoder take the first decoder that
opens. The hwaccel configuration now comes from the decoder actually
being opened. The chosen decoder is logged, and a missing one names
the codec.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tpe3orhELpViSQpZcsTTjr
The player claimed a fixed list of 28 extensions, and the frontend's
media type table only routed those to it, so common files (.ogv, .mka,
.opus, .aac, .ac3, .aiff, .wv, .ape, .mts, ...) didn't open with the
built-in player at all.

- The core keeps its list and adds 31 more extensions, each only if
  this build has the demuxer and, where the extension implies a codec
  (.ogv Theora, .ape APE, .wv WavPack, .ac3 AC-3, ...), the decoder.
  RetroArch's platforms ship very different FFmpeg builds, so
  valid_extensions is built once, at runtime.
- libretro_ffmpeg_ffmpeg_core_supports_extension() exposes the same
  check; msg_hash.c lists the extra extensions as MOVIE_OTHER /
  MUSIC_OTHER (after everything else, so existing claims win) and
  path_is_media_type() only routes them to the player when it passes.
  MPV builds route them unconditionally.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tpe3orhELpViSQpZcsTTjr
… has one owner

The codec contexts are freed at unload whatever the FFmpeg version, not
only closed, and a hardware device is handed to the codec context that
uses it rather than referenced from it, so freeing that context closes
the device - every decoder the probing opens and drops included.

ffmpeg_sweep_test runs its ASan pass with leak detection: every load and
unload in it must give back what it took.
@LibretroAdmin
LibretroAdmin merged commit 2c7d46d into libretro:master Oct 8, 2026
108 of 112 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants