DLSS: PathTracing - Reflex - Latency

[SveSop]

I suppose this is not a "bug" but maybe something could be improved in some way. This is also most likely depending on VKD3D improvements, but atleast i found it a bit interesting for testing purposes.

I used this demo: https://github.com/NVIDIA-RTX/RTXPT/releases/tag/v.1.5.0

This needs wine with HAGS support, so either wine-proton or some other means like custom wine with this:
https://gist.github.com/Saancreed/0aa3f4188846146698f5da3dfa0dfcbd?permalink_comment_id=5504491#gistcomment-5504491

I also used VKD3D + HansKristian-Work/vkd3d-proton#2348
and DXVK-NVAPI (Experimental) + #276

This is required to show the "stats report" show on these two screenshots:

This is from wine/Linux

Same demo running in windows 11

The performance increase for windows is appx 30%, and the frame "timings" is a lot better. I am aware that these patches are WIP, so it might not paint a correct picture of what happens... But i think the FPS is still valid.

One other thing is that on windows, when using REFLEX as a "fps limiter", it actually does limit the fps to the set value with OR without FG (2x for me) enabled... however when running this using wine, the FPS is 2x what the limit it set to. I suppose something is going on internally in the driver when it comes to "detecting" if FG is enabled when using REFLEX to limit fps when using windows perhaps? I logged the NVAPI calls on windows, and the Us setting for 20 fps is the same from the call both for windows and dxvk-nvapi, so it has to happen somewhat internally i think.

Further if you limit the FPS to something low'ish (eg. 20), the latency is more or less through the roof when panning around the scene using the keyboard. Eg. pressing A to pan left for 4-5 seconds and releasing the button will keep panning for several seconds after letting go. This is also something that do not happen in windows, as the action is immediate.

This could possibly be a wine issue, where wine ends up queuing up keystrokes (repeats) when the fps is limited using some mechanic in VKD3D perhaps? Not sure.

@Saancreed do you have any ideas here to this? I found this demo quite nice for testing up-to-date DLSS with this without depending on troubleshooting this through steam. I was hoping the out-of-order rendering issue one observes in CP2077 could be recreated with this, but so far i do not see the same behavior here even when setting the camera on "animation" where it pans around the room on itself.

[Saancreed]

Huh, you're right, the FPS cap is indeed double of what I have set.

I'm tempted to file this as Reflex FPS limiter (as implemented in the driver and exposed via the VK_NV_ll2 extension) bug, and therefore something for NV people to look into. It's unlikely to be a vkd3d-proton or dxvk-nvapi issue because the same is true for Vulkan applications (using a sample size of 1 🙃):

… unless, of course, dxvk-nvapi or its vkreflex layer is indeed doing something stupid.

Can't say much about frame timings though.

Further if you limit the FPS to something low'ish (eg. 20), the latency is more or less through the roof when panning around the scene using the keyboard. Eg. pressing A to pan left for 4-5 seconds and releasing the button will keep panning for several seconds after letting go. This is also something that do not happen in windows, as the action is immediate.

This could possibly be a wine issue, where wine ends up queuing up keystrokes (repeats) when the fps is limited using some mechanic in VKD3D perhaps? Not sure.

I'm not sure, and fwiw I cannot reproduce it, running winex11 on kwin_wayland so not exactly an uncommon setup (if we ignore PRIME laptop memes for a moment).

I was hoping the out-of-order rendering issue one observes in CP2077 could be recreated with this, but so far i do not see the same behavior here even when setting the camera on "animation" where it pans around the room on itself.

Didn't we already establish it to be a game bug? I haven't seen this in any other game with DLFG and I've checked about 10 of them.

[SveSop]

I'm tempted to file this as Reflex FPS limiter (as implemented in the driver and exposed via the VK_NV_ll2 extension) bug, and therefore something for NV people to look into. It's unlikely to be a vkd3d-proton or dxvk-nvapi issue because the same is true for Vulkan applications (using a sample size of 1 🙃):

… unless, of course, dxvk-nvapi or its vkreflex layer is indeed doing something stupid.

I am leaning towards this being some sort of internal driver issue. The setting used by NvAPI_D3D_SetSleepMode when setting minimumIntervalUs is used the same in windows, but there it works just fine. I believe this would indicate that the "driver" should be aware of FG usage and limit FPS accordingly.

Further if you limit the FPS to something low'ish (eg. 20), the latency is more or less through the roof when panning around the scene using the keyboard. Eg. pressing A to pan left for 4-5 seconds and releasing the button will keep panning for several seconds after letting go. This is also something that do not happen in windows, as the action is immediate.

This could possibly be a wine issue, where wine ends up queuing up keystrokes (repeats) when the fps is limited using some mechanic in VKD3D perhaps? Not sure.

I'm not sure, and fwiw I cannot reproduce it, running winex11 on kwin_wayland so not exactly an uncommon setup (if we ignore PRIME laptop memes for a moment).

Perhaps some weird gnome issue? To actually limit fps to 20 fps, FG must be disabled in that demo tho, as it is hardly noticeable for 40 fps (reflex 20 fps limit x 2 (FrameGen enabled)).

Somewhat off-topic:

Didn't we already establish it to be a game bug? I haven't seen this in any other game with DLFG and I've checked about 10 of them.

I suppose various timing issues break more easily when using wine vs windows which probably has a lot more quirks to workaround something like this. What does seem a bit weird is that SOMETIMES it works just fine (ref. to the CP2077 issue thread - and i can also confirm this).

In windows NvAPI_D3D12_SetNvShaderExtnSlotSpaceLocalThread is used possibly for SER https://developer.nvidia.com/blog/improve-shader-performance-and-in-game-frame-rates-with-shader-execution-reordering/
I am not sure if that is ONLY a performance thing, and that if it is NOT available it should work perfectly well without hickups, but as i have commented on before - Games tend to "assume" a lot when it comes to driversetup, meaning even tho the call is returning something else than NVAPI_OK, the game might ignore this... which i suppose would be considered as a bug.

[Saancreed]

I am leaning towards this being some sort of internal driver issue. The setting used by NvAPI_D3D_SetSleepMode when setting minimumIntervalUs is used the same in windows, but there it works just fine. I believe this would indicate that the "driver" should be aware of FG usage and limit FPS accordingly.

👍

Or, maybe the problem is the other way around and VK_NV_low_latency2 is aware of frame gen happening and changes its behavior, while D3D12 and legacy Vulkan Reflex (NvLowLatencyVk.dll / VK_NV_low_latency) as supported on Windows aren't. I'd expect that from the driver's PoV the interpolated frame presents aren't much different from the application presenting rendered frames; I think both are done entirely on the Windows side of winevulkan. Changing the minimum interval internally to half the original value seems bizarre in that case.

(What if this is the reason why Streamline never migrated to VK_NV_ll2 despite having over a year of time to do so? 🤔)

Have you reported this to Nvidia, either via forums, or email, or nvbug system? If not, might be a good idea to do just that.

Perhaps some weird gnome issue? To actually limit fps to 20 fps, FG must be disabled in that demo tho, as it is hardly noticeable for 40 fps (reflex 20 fps limit x 2 (FrameGen enabled)).

Yeah, I also had FG off when trying to repro this. Perhaps it would be worth trying out if Gamescope run directly on the TTY with DRM backend also has that issue? It will surely be less of a hassle to test than installing Plasma if you don't want an entire DE, and I also can't reproduce it in Gamescope on TTY.

I suppose various timing issues break more easily when using wine vs windows which probably has a lot more quirks to workaround something like this. What does seem a bit weird is that SOMETIMES it works just fine (ref. to the CP2077 issue thread - and i can also confirm this).

Exactly the reason why it's a nightmare to debug. I, for one, am very much not tempted to look into it despite still having CP2077 in my backlog to play in the future.

In windows NvAPI_D3D12_SetNvShaderExtnSlotSpaceLocalThread is used possibly for SER https://developer.nvidia.com/blog/improve-shader-performance-and-in-game-frame-rates-with-shader-execution-reordering/
I am not sure if that is ONLY a performance thing, and that if it is NOT available it should work perfectly well without hickups, but as i have commented on before - Games tend to "assume" a lot when it comes to driversetup, meaning even tho the call is returning something else than NVAPI_OK, the game might ignore this... which i suppose would be considered as a bug.

It's supposed to be a performance thing. Furthermore, SER is only useful for ray tracing, so there should be no difference if RT is disabled. If the out-of-order frames issue is still present even without RT then the cause lies elsewhere.

(But yes, quite off-topic indeed.)

[SveSop]

Or, maybe the problem is the other way around and VK_NV_low_latency2 is aware of frame gen happening and changes its behavior, while D3D12 and legacy Vulkan Reflex (NvLowLatencyVk.dll / VK_NV_low_latency) as supported on Windows aren't. I'd expect that from the driver's PoV the interpolated frame presents aren't much different from the application presenting rendered frames; I think both are done entirely on the Windows side of winevulkan. Changing the minimum interval internally to half the original value seems bizarre in that case.

(What if this is the reason why Streamline never migrated to VK_NV_ll2 despite having over a year of time to do so? 🤔)

Hmm.. possibly. This demo in particular has 2 methods of fps limitation - 1 is reflex, the other is.. well.. "normal"? Using Reflex as stated above has this 2x fps w/FG enabled only when using wine, the other method seems to have this 2x fps w/FG enabled both in windows and wine. So i suppose this is some sort of a different fps limiter.

Games like CP2077 and Indy also seems to do this 2x fps w/FG enabled both in windows and wine afaik. (Suppose it will be 3x or 4x if using a 50xx card)

Have you reported this to Nvidia, either via forums, or email, or nvbug system? If not, might be a good idea to do just that.

No.

In windows NvAPI_D3D12_SetNvShaderExtnSlotSpaceLocalThread is used possibly for SER https://developer.nvidia.com/blog/improve-shader-performance-and-in-game-frame-rates-with-shader-execution-reordering/
I am not sure if that is ONLY a performance thing, and that if it is NOT available it should work perfectly well without hickups, but as i have commented on before - Games tend to "assume" a lot when it comes to driversetup, meaning even tho the call is returning something else than NVAPI_OK, the game might ignore this... which i suppose would be considered as a bug.

It's supposed to be a performance thing. Furthermore, SER is only useful for ray tracing, so there should be no difference if RT is disabled. If the out-of-order frames issue is still present even without RT then the cause lies elsewhere.

(But yes, quite off-topic indeed.)

It does seem to not be a 100% surefire way to reproduce this for "everyone", but for me atleast this seems to be happening with CP2077.

Starting the game with RayTracing enabled = Borked with this out-of-order rendering thingy.

Starting the game with RayTracing disabled = Working just fine. Enabling RT will also work just fine UNTIL you use eg. quicktravel or load a savegame or something that makes the scene change. Then it borks again, and the only way to get back to normal is to disable RT -> completely reload the game -> enable it again.

DLSS +/- FG does not really seem to change much when this is done. Disabling Ray Reconstruction DOES sometimes "fix it".

[jp7677]

Slightly related info regarding the fps limits https://forums.developer.nvidia.com/t/smooth-motion-with-40-series-has-really-bad-input-lag/343915/7