Skip to content

video: Reduce allocations/copies when decoding frames - #24135

Open
moulins wants to merge 7 commits into
ruffle-rs:masterfrom
moulins:video-less-alloc
Open

video: Reduce allocations/copies when decoding frames#24135
moulins wants to merge 7 commits into
ruffle-rs:masterfrom
moulins:video-less-alloc

Conversation

@moulins

@moulins moulins commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Description

Currently, the VideoDecoder::decode_frame method returns owned Bitmap<'static>s, forcing decoders to allocate a fresh buffer for each decoded frame.

This PR changes the API to instead provide a short-lived buffer through a callback, and updates the decoders to take advantage of this by reusing buffers more aggressively.

Testing

All visual/video tests pass on native, but I still need to do a visual check with my human eyes, and properly test the web decoder.

Checklist

  • I, a human, have self-reviewed this PR and fully understand the changes within.
  • I have made or updated tests where possible.
  • All of my commits are properly scoped, compile successfully, and pass all tests.
  • This PR does not make sense to split up into smaller PRs.
  • An LLM was involved in the authoring of this code.

@moulins moulins added T-perf Type: Performance Improvements A-video labels Jul 6, 2026
@torokati44

Copy link
Copy Markdown
Member

This might even have to evolve a bit further in the future, to use a buffer pool of some sorts.
The reason is, that most video codecs fit the basic "encoded frame in, decoded frame immediately out" paradigm, but not H.264, with its bidirectional predictions. So, ideally, we'd have to make the FLV/MP4 demuxers able to feed a few encoded H.264 frames ahead of the playhead, and store the decoded frames until it's time to present them. This would make us match Flash more closely, and also unlock the OpenH264 bump (and with it, CI on Windows ARM), because the decoding delay becomes irrelevant.

@moulins

moulins commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Tested various video SWF files; everything seems to be in working order.

@moulins moulins added the waiting-on-review Waiting on review from a Ruffle team member label Jul 8, 2026
Comment thread render/src/bitmap.rs
}
}

#[inline]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh, somehow this method and the one below are considered uncovered despite their callers being covered?

@Lord-McSweeney Lord-McSweeney left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm completely unfamiliar with this area but the code LGTM

@moulins
moulins force-pushed the video-less-alloc branch from 147948f to 26cc366 Compare July 14, 2026 19:03
@torokati44

Copy link
Copy Markdown
Member

I've been meaning to review this a couple times but ... uh ... life 🫥

moulins added 7 commits July 25, 2026 13:46
Outline the entirety of the 'wrong-size' path into a cold function, and
re-slice instead of allocating a new `Vec` when an `&[u8]` is too big.
This will allow decoders to reuse internal buffers to hold decoded frames.
…coded frames

This saves two allocations by `decode_frame` call.
This saves one allocation per `decode_frame` call.
This saves one allocation per `decode_frame` call.
- Update the last frame in-place in `decode_v1` instead of working on a copy
  (this will affect output for corrupted streams, which is probably OK)
- Build the final RGB frame into the preexisting `tile` sratch buffer instead of
  creating a fresh `Vec`

These changes save two allocations per `decode_frame` call.
This saves several allocations and copies for each `decode_frame` call.
@moulins
moulins force-pushed the video-less-alloc branch from 26cc366 to 1438c01 Compare July 25, 2026 11:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-video T-perf Type: Performance Improvements waiting-on-review Waiting on review from a Ruffle team member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants