Pipelined rendering adds 1 extra latency frame

[otcova]

bevy_framepace is an amazing drop in plugin that automatically lowers input latency from 2~4 frames to 0~1 frame.

Unless you use pipelined rendering (default bevy feature "multi_threaded"), then its between 1~2 frames.

How to easily mesure it?

1. Add feature "multi_threaded" in Cargo.toml [dev-dependencies].
2. Set FPS to 1 (this way we can easily count the frames). This can be done at examples/demo.rs, replacing Limiter::from_framerate(30.0) with Limiter::from_framerate(1.0)
3. Run the demo (cargo run --examples demo), and move the mouse. You will observe one of the following:

0~1 latency: The ball teleports to the current cursor position
https://github.com/user-attachments/assets/4f279c09-95af-4d2a-825c-378af844e43b

1~2 latency: The ball teleports to previous (1 second) cursor position
https://github.com/user-attachments/assets/54456446-09fb-42da-9cf5-d29b3cbe6301

Why does this happen with pipelined rendering?

1. VSync:
   Typically vsync uses a queue of frames (a FIFO), that stores between 1~3 frames before they are shown on screen.

   • Consequence: This adds 1 (time to produce the frame) + 1~3 (queue size) = 2~4 frames of input latency
   • Why is it done this way: Queuing frames protects against lag spikes, ensuring there is always something new to show on every screen refresh.
   • The problem: For many (or most) applications the queue is too big (or not needed). Even at the worst case: sometimes not having the next frame ready in time (and hence, repeating the previous frame), is better than increasing average input latency.
   • How is this avoided:

     • Increasing the FPS: By rendering much faster than the refresh rate, the latency in milliseconds is reduced. However, this isn't always an option (due to hardware limits, battery life, or fan noise), and even at high FPS, 2~4 frames of latency can be noticeable in competitive games and some GUIs.

     • Delaying frame production: By pacing the frames to match the screen refresh rate, we can control the number of frames in the queue. bevy_framepace tries to maintain it between 0 and 1.

2. bevy_framepace:

   • Why 1~2 frames of input latency with pipelined rendering?: Currently, the plugin sleeps in the render world. With pipelined rendering, Render N runs in parallel with App N+1. This means the sleep happens after App N+1 has already polled its inputs.

   Theres and extra frame of latency because the app world polled inputs early and is now waiting on the render world to finish sleeping.

3. Ideal scenario:
   Same concept and logic than bevy_framepace, but sleeping just before raw input.

   • The problem: Bevy does not have any schedule to run code just before raw input: First is just after raw input, Last is just before extract, and in the render world is to late since it runs in parallel with the raw input polling.

4. Sleeping in a subapp:
   As a workaround, we can create a FramepaceSubapp that runs after the render extract.
   The Extract schedule does not run in parallel with the main world. By sleeping in the Extract of this new FramepaceSubapp, we can stall the raw input polling of the next frame without stalling the render of the current frame.

   To verify that it works with 0~1 frames of latency, I've cloned this bevy_framepace repo and moved the sleep system from RenderSystems::Cleanup to the Subapp:

   let mut sub_app = SubApp::new();
   sub_app.set_extract(|main_world, _| main_world.run_system_once(framerate_limiter).unwrap());
   app.insert_sub_app(FramepaceSubapp, sub_app);

   (https://github.com/otcova/bevy_framepace_experiment)

Why isn't Subapp the solution (yet)?

Bevy does not provide an API to order the subapps execution.

Currently in bevy 0.19, the execution order is at the mercy of HashMap::<AppLabel, _>::iter.
This means that even changing any subapp struct name changes its AppLabel, which changes its Hash, and the order in the HashMap.

Then, how does my prototype work? I renamed the subapp struct name until i got the correct order. This is fragile:

• If a user adds another subapp, the HashMap will grow and the order might change.
• If bevy changes the name of its render subapp, the Hash will change, and the order might change.

What are the next steps?

1. Verify the correctness of what I've presented.
2. Create an Issue/Feature in bevy to request either:
   • API to order subapps execution.
   • A main world system schedule that runs just before polling the raw input.