Repository navigation
NVIDIA DLSS, DLAA and Reflex (NGX + Vulkan, Linux and Windows, opt-in) - #841
Conversation
…Linux, opt-in build) OoT scales its scene up by 4/3 with a network of its own (fd029a317a9a4b94). With the new Insert menu row NVIDIA DLSS on (or RX_VK_DLSS=1), NVIDIA's DLSS draws that output instead, through NGX on Vulkan: - the scene camera comes from the draws into the scene's colour target (the vertex stage's uniform buffer 3: the view-projection at 0x30, the projection at 0x70, the target's size at 0x120), the depth from their depth target - shaders/dlss_prep.comp makes DLSS's inputs: the network's input colour (LDR, the square root of the scene's), the depth, and motion vectors from the depth reprojected through this camera and the one DLSS last drew with (camera motion only: what moves by itself can ghost) - shaders/dlss_out.comp squares DLSS's output into the network's output Anything unexpected leaves the network to run, the reason in the log. NVIDIA's SDK is not part of Hookshot: scripts/linux/ngx-sdk.sh fetches its headers and libnvsdk_ngx.a (and the DLSS runtime, libnvidia-ngx-dlss.so) from NVIDIA's DLSS repository at a pinned release, checked by SHA-256, and a build with HOOKSHOT_NGX_SDK=<that folder> compiles vk_dlss.cpp with it. Without it the menu row says the build has no DLSS.
…S draws DLSS resolves detail finer than a pixel only from frames whose camera moves by a different fraction of a pixel each time. nx2/oot_jitter.py adds C after the loads of sead's perspective projection's field of view and ratio (its matrix update and the game's three inlined copies of it): for cameras of the picture's ratio it writes the projection's centre offset (+0xb4, +0xb8), which the update reads next, from host/rx_backend.c hos_dlss_jitter (Halton 2, 3 over 16 frames, scaled by the scene's size the renderer reports as rx_option 6 and 7). The renderer reports 0 unless DLSS is drawing, so the game runs as shipped otherwise. The renderer reads the offset back from the projection in the camera's uniform buffer, hands it to DLSS, and takes it off the motion vectors. RX_VK_DLSS_JITTER=0: none.
The test runs nx2/build.py's Vulkan job construction in a namespace of its own; that code now reads NGX_SDK (NVIDIA DLSS's SDK folder, empty when there is none), which the namespace lacked.
…ice of the NVIDIA DLSS row The row is now Off / DLSS / DLAA (RX_VK_DLSS=0 / 1 / 2). DLAA runs DLSS at a scale of 1 on the scene as drawn, with the 3D held at full scale whatever the Resolution row says (rx_option 2 and 3), and dlss_out.comp scales its picture to the game's upscale output with a linear filter. (The game reduces that output to 1080p for its presentation pass anyway: DLSS at 100% resolution is close to it, and cheaper.)
… On + Boost) No Streamline: the swapchain is made with latency mode on when the device has VK_NV_low_latency2 and present ids, its sleep mode follows the new NVIDIA Reflex row (RX_VK_REFLEX=0 / 1 / 2 over it; =-: the extensions not enabled), and each frame the driver gets its markers with the present's id: simulation start and input after vkLatencySleepNV, simulation end and render submit start at the game's present, render submit end before the present's submit, present start and end around vkQueuePresentKHR. The sleep follows the present on the game's presentation thread. Not used with frame generation (its own display thread). Every 300 frames the log has the driver's timings (vkGetLatencyTimingsNV), averaged, to compare On and Off.
The renderer's option 3 is now the fixed path's scale in 1/10000: DLAA's full scale is 10000 there (and option 2, dynamic resolution, off as before).
tests/test_present_semaphores.py compiles the swapchain's setup, from its image query to the log line, with stubs of its own: the call to Reflex sat inside that span. It now follows the log line, where the swapchain is the device's in full.
…K and the like) With NotRexed#849 the output is 2048x1152, 2560x1440 or 3840x2160 and the Resolution row scales the 3D against it, so the upscale's ratio is no longer 4/3. The sizes already follow the dispatch's input and output, and the feature is made again for a new output size or a render size outside its mode's range; the check on the ratio now takes up to 3x (DLSS's Ultra Performance), and a render size equal to the output's runs as DLAA.
… at the output's size) With the 3D drawn at the picture's size (DLAA, or the Resolution row's top step since NotRexed#849) the game runs the same network as a pass at one size (9dc98871eb7fc101, 2560x1440 in and out) instead of its 4/3 upscale, and DLSS was never asked. That pass is replaced as well: render size equal to the output, DLSS's DLAA mode. Checked at 2560x1440: 2560x1440 -> 2560x1440, mode DLAA.
scripts/ngx_sdk.py takes over from scripts/linux/ngx-sdk.sh, which is now a thin wrapper around it: one file table per platform at the pinned NVIDIA/DLSS release (v310.9.16), each file checked against its SHA-256 before it is kept. On Windows it fetches LICENSE.txt, the headers, x64/nvsdk_ngx_s.lib and rel/nvngx_dlss.dll; --game copies the runtime into <game>/ngx. It prints NVIDIA's license notice first and downloads nothing until the license is accepted (--accept-nvidia-license, or a typed yes). Nothing of NVIDIA's is committed or bundled. tests/test_ngx_sdk.py checks the table, the hash checks and the license gate without the network.
nx2/build.py takes HOOKSHOT_NGX_SDK on Windows too, when the folder has lib/nvsdk_ngx_s.lib (NVIDIA's static loader, built for Microsoft's static runtime, /MT), and links it into hookshot_vk.dll with -ladvapi32 (its registry reads). With llvm-mingw, Microsoft's C++ runtime it names is left out (/nodefaultlib:libcpmt) and two objects stand in for what it needs of it: host/mingw_msvc_shims.c (the /GS cookie and handler and _fltused, as for the static FFmpeg) and the new gpu/vk/vk_ngx_msvc_rt.cpp (__GSHandlerCheck_EH on the system's __CxxFrameHandler3, thread-safe statics on an SRW lock and a condition variable, Microsoft's operator new and delete, std::_Xbad_alloc and std::_Xlength_error). Neither is compiled without the SDK, nor for the MSVC build (HOOKSHOT_MSVC=1), which has Microsoft's runtime itself. Cross-built with llvm-mingw: the DLL links, and imports only KERNEL32, USER32, GDI32, ADVAPI32 and api-ms-win-crt-*. Without the SDK it is the same objects as before, with the same imports and exports.
On Windows, vk_dlss.cpp looks for nvngx_dlss.dll in RX_VK_DLSS_DIR, the game folder's ngx\ and %LOCALAPPDATA%\Hookshot\ngx (Linux keeps the game folder's ngx/ and $XDG_DATA_HOME/hookshot/ngx), and its log names the file of the platform it runs on. The paths NGX gets are converted from the ANSI code page (MultiByteToWideChar, CP_ACP), as GetModuleFileNameA and getenv give them, so a game folder with letters outside ASCII is still found. An empty RX_VK_DLSS_DIR is ignored. A comment says why NGX's core, _nvngx.dll, is never put beside the game: it is the driver's.
A new tick, "NVIDIA DLSS (downloads NVIDIA's SDK under NVIDIA's license)", off by default. Ticking it shows NVIDIA's license and stays ticked only when it is accepted. The build then passes HOOKSHOT_NGX_SDK (the builder's ngx-sdk folder) through Runner.cs and runs make_game.py with --fetch-ngx-sdk --accept-nvidia-license, which fetches NVIDIA's SDK with scripts/ngx_sdk.py before anything else (only what is missing or not the file expected). Without the tick, HOOKSHOT_NGX_SDK is not passed on. An update of a folder built with DLSS builds it with DLSS again. make_game.py places DLSS's runtime in the game folder's ngx folder whenever the build had the SDK (nvngx_dlss.dll on Windows, libnvidia-ngx-dlss.so on Linux), never beside the exe. tests/test_make_game_ngx.py checks both. The builder's C# is not compiled here (no .NET on this PC).
windows-builder.yml gets a job of its own, vulkan-dlss, beside the Vulkan back end's: it fetches NVIDIA's DLSS SDK with scripts/ngx_sdk.py (the new --build-only: the headers and nvsdk_ngx_s.lib, not the 59 MB runtime), builds hookshot_vk.dll with it through llvm-mingw, checks the build said it was with NVIDIA DLSS, and fails when the DLL imports anything but KERNEL32, USER32, GDI32, ADVAPI32 and api-ms-win-crt-*. Nothing is uploaded, and the release job does not need it. actionlint 1.7.12 (with shellcheck) passes.
readme.md gets an NVIDIA DLSS section under Build from source: the Builder's tick, the command line on Windows (scripts/ngx_sdk.py, HOOKSHOT_NGX_SDK, where nvngx_dlss.dll goes and where the game looks), Linux, NVIDIA's license, and never to share a game folder built with NVIDIA's files or to put _nvngx.dll beside the game. Credits name NVIDIA DLSS. CONTRIBUTING.md says how to build with the SDK, that nothing of NVIDIA's is ever committed, bundled or uploaded, and what to update when the pinned release changes.
|
Thank you to everyone on this collaboration branch! It's marked work-in-progress, overlaps #850, and its script fetches NVIDIA SDK files from raw.githubusercontent.com/NVIDIA/DLSS at build time. Closing to keep the queue clear; please reopen when it's ready for review as a focused PR. |
The Graphics tab's three scales read top to bottom now: Output resolution, the Upscaler (Game's own, NVIDIA DLSS, NVIDIA DLAA: the NVIDIA DLSS row's choice), Render resolution, Window scaling filter (the Upscaling row), Sharpness. NVIDIA Reflex moves up beside Frame pacing. With DLSS the Render resolution row steps through DLSS's modes, Auto, Quality (67%), Balanced (58%), Performance (50%) and Ultra Performance (33%), each a twelfth of the output (two new values, 5 and 6; 1 to 4 keep their shares). With DLAA it shows Native (DLAA) and does not step. Another Upscaler moves the resolution to the nearest of its steps. DLSS and DLAA are offered only where they can run: a build with NVIDIA's SDK, NGX's extensions on the device, and NGX's answer (asked once when the menu opens: dlss_probe). The help says why not (not in this build, an NVIDIA RTX GPU, a newer driver, DLSS's runtime missing), and dlss_mode is 0 there, so a saved DLAA does not hold the 3D at full scale on a GPU without DLSS. The saved choice is kept. Reflex is offered only with VK_NV_low_latency2. Quality presets leave the Upscaler alone; with DLSS their resolution becomes DLSS's nearest mode. Saved options still load: resolution and upscaling keep their keys, the Upscaler is saved as dlss and the NVIDIA DLSS row's nvidia_dlss is read as it. tests/test_upscaler_rows checks the steps, the names and the keys.
DLSS stood in only for the 4/3 network (fd029a317a9a4b94) and the full-size pass: the 3D at 75% (Auto's usual size) and DLAA. At the other fixed shares the game runs a network of its own for each (2048x1152: 67% 6676619641320ba7, 58% f10782f9e58b6a7c, 50% 72ba090c70fe2614, 33% 0775215f8da720e6), so DLSS at 67% or 50% left the game's upscale running, with the scene camera taken and nothing said. Those are now found by their shape (a cooperative-matrix network sampling one texture into one image) and taken only when the texture is this frame's scene and the image the B10G11R11 picture; any other such network runs untouched, without a log line. The log names a network the first time DLSS takes it.
|
Re-opening, it was tested on Windows, will update with results shortly |
vk_dlss.cpp did not include vk_work.h, so its dispatches and NGX's own commands did not move work_epoch, and full_barrier left out the barriers around the evaluation (and the game's next one): NGX read its inputs and dlss_out.comp read DLSS's output without a barrier. vk_work.h is now included last, and note_work() follows the feature's creation and the evaluation.
Runner.Info gained a fourth parameter (ngxSdk) and the reflection call still passed three, so the test threw and CI's source regressions failed. It passes null: a build without NVIDIA DLSS, as before.
|
Notes from a parallel DLSS test on v2.0.7 (Streamline 2.14.16, RTX 5080, driver 621.14), where I ported this PR's frustum-offset jitter ( Two things I saw with the jitter on, in case they show here too:
Unrelated but very visible under DLSS: the blocky light halos and the checkerboard smoke go away when the few NEAREST samplers are forced linear, see #1117. |
…as (steady shadow maps) The jitter went into sead's perspective projection itself (its frustum offset at +0xb4 and +0xb8), so the game's CPU work saw it as well as the GPU. The shadow cascades are fitted to the view's frustum: with the camera still and DLAA at 3840x2160, the cascade's projection offset moved by 0.17 shadow texels a frame on average (up to 0.32-0.44) with the jitter, against 0.0035 without it, so the shadow map was rasterised at a different sub-texel origin every frame. The jitter now goes on where the game fills each view's uniform buffers (main+0x6bc520): after the loads of the view's projection (its rows at +0x1a8), on P[0][2] and P[1][2] in the registers, so the buffers get the jittered projection and the view-projection made from it, and nothing in the game's memory changes. hos_dlss_jitter_clip takes the place of hos_dlss_jitter_cam (no per-camera bookkeeping is needed: nothing is written, so nothing has to be put back when DLSS stops); gpu/vk/dlss_math.h dlss_jitter_clip picks the projections (perspective, 16:9 or the window's ratio). Measured headless (Kokiri Forest spawn, still camera, DLAA 3840x2160): the cascade offset's step per frame 0.163-0.175 texels before, 0.0034-0.0036 after (0.0034-0.0037 with the jitter off), three sessions each; DLSS still gets the jitter (the scene camera's P[0][2] and P[1][2] in its uniform buffer, the log's per-frame jitter) and the picture's detail is the same as before. On screen at that spot the shadows' share of the frame-to-frame change is small: in one session switching between the two every 64-96 frames, neither was steadier in every segment; jitter on (either way) changes ~15% more pixels from frame to frame than jitter off. Frame rate unchanged (39.0-39.4 fps, jitter on or off). tests/test_oot_jitter.py builds the sites' C and checks only the named lanes move; tests/test_dlss_math.cpp checks which projections get the jitter.
|
Thanks for the detailed notes, and good to hear the jitter holds up in your build. Shadows: your guess was right. Our jitter went into the game's camera projection itself, so the CPU side saw it too, and the shadow cascades (fitted to the view frustum) followed it. With a still camera at DLAA 3840x2160, the shadow map's origin moved by about 0.17 shadow texels every frame (up to 0.4), against 0.003 with the jitter off. The jitter now goes only into the GPU's copy of the projection: it's added in registers where the game fills each view's uniform buffers, and nothing in the game's memory changes. DLSS gets the same jitter and picture detail is unchanged. The cascades, culling and everything else on the CPU see the real camera, and the cascade movement is back to the jitter-off level. This is pushed now (4ee204a). In my tests (RTX 3090, Kokiri Forest's dappled light) the on-screen difference was small. If you can, please check whether it's visibly calmer on your side, ideally somewhere with crisp sunlit shadows. fps: I couldn't reproduce the jitter cost. At DLAA 3840x2160 I get 38.5–39.4 fps with the jitter on or off, before and after this change, so it may be specific to the Streamline path or the GPU. Here the shadow maps are re-rendered every frame either way. #1117: thanks for the pointer, I'll try it together with DLSS. |
The five Graphics rows read once at the start (GPU device, Output
resolution, Handheld profile, Ultrawide, Robust buffers) are marked as
taking a restart (Row::restart), and the menu keeps the value each had at
the start (gpu/vk/menu_restart.h). Their values no longer say "(restart)"
whatever runs: "(after restart)" follows a value only while it is not the
one the game runs with, and while any is, the Graphics tab says so at its
foot ("Restart needed: Output resolution, Ultrawide", one line, the last
ones as "and N more" when they do not fit). A row changed back is no
longer pending.
Closing the menu with something pending asks first, on a page of its own:
"Restart to apply Output resolution and Ultrawide?", "Progress since your
last save is lost.", [Later] (chosen at first) and [Restart now], by keys,
the gamepad and the mouse (vk_menu_hit.h ask). Later closes the menu with
the settings saved for the next start, as before, and is not asked again
until one of those rows changes. Restart now saves the options and the
pipeline cache and closes the window as the player would (WM_APP + 9:
host/display.c, host/posix/display_sdl.c): rx_close_asked, the log, and at
the very end of that close host/hos_relaunch.c starts the game again.
Linux: the process becomes the game again (execve of /proc/self/exe,
argv[0] on the PATH if that is gone) with the arguments and the
environment it started with, from /proc/self/cmdline and
/proc/self/environ, so a launcher or a stream waiting for it keeps it. The
game is never run from inside the AppImage (hookshot-launch takes its
variables away), so /proc/self/exe is the game. Windows: CreateProcessW of
GetModuleFileNameW with GetCommandLineW, then the exit.
tests/test_menu_restart.py checks the pending rules (a change, a change
back, Later and a change after it), the names, the relaunch command (made-up
files and its own /proc/self ones), and the path from the menu to the
window's close; test_menu_hit.cpp the question's buttons. The screensaver
fixture's window procedure closes through hos_window_closed_exit.
Each row's help was one long text describing every choice. A row now has a short help that stays the same whatever its value (what the row is), and a hint for the value chosen (Row::hint), drawn on a line of its own under it, dimmer and set off by a green bar: Window scaling filter says "How the finished picture is scaled to the window, when its size differs from the Output resolution.", then "The plain copy.", "Sharper." or "Sharper still; can ring on edges." Every row of every tab is rewritten so; what the old texts said moves to the value it is about. The help that depended on the GPU or the other rows is a hint now: hint(v, -1) is the one to show, hint(v, 0), hint(v, 1)... every one a value can have. The Upscaler's says why DLSS and DLAA are not offered (not in this build, RX_VK_DLSS=0, no RTX GPU, an old driver, no runtime), Reflex's that it is not available, the Render resolution's follows the Upscaler shown, the filter's says when nothing is to be scaled. measure_help keeps the room of the tallest help and hint together, so the panel does not move as a value changes. NVIDIA DLSS and DLAA, DLSS's Render resolution modes and Sharpness while DLSS or DLAA runs say to set Sharpness to around 80%: both look softer than the game's own upscaler. The readme's menu section says so too, and how the help reads. test_upscaler_rows.py checks the Upscaler's reasons, the 80%, and that measure_help counts the values' lines.
|
Update: merged main again (016d582, with the start-up screen), plus three changes.
Checked on Linux:
The menu agent's earlier run also cross-built The description now says that DLSS's paths are wide but the game itself doesn't yet start from a folder outside the ANSI code page. That will be its own PR. |
Main's vk_draw.cpp per-thread objects are functions now (TLS_OBJECT: dw is dw()); dlss_stand_in's two uses of dw.imgs follow.
… in UTF-8 on Windows From the Windows tests on 842121f: the Builder saved the tick only when Build was pressed, so ticking it, accepting NVIDIA's license and closing the Builder forgot it. DlssChanged now saves the settings either way. NGX writes its paths in the ANSI code page, so in hookshot_log.txt its 'ngx:' lines showed a folder's non-ASCII letters as replacement characters while ours (UTF-8) showed them right. ngx_log converts a line that isn't valid UTF-8 from the ANSI code page. The readme says NGX starts on demand (the Insert menu's first opening, or the first upscale with DLSS), so a log from the title screen has no NGX lines yet.
|
Thanks for the thorough Windows run on 842121f: 11 of 14 now checked. The description's checklist is updated with your results. Fixed from your findings (pushed as d378bd0):
Folders outside the ANSI code page (the game reading its own path as CI: this push also merges the latest main (the v2.0.9 hotfix). Main had changed Also in the description: newer NGX releases offer DLSS Frame Generation on Linux too, so it's noted as a possible follow-up rather than Windows-only. |
Main's NotRexed#1169 (Rumble strength, option 9) and this branch both add to vk_replay.cpp's option table: both kept. Its tests/test_rumble_menu.py learns the other named option ids in vk_api.h (HOS_OPT_DLSS_JITTER_*) and gets stand-ins for the parts of set_row the Rumble strength row never reaches here (the Upscaler's rows, the restart offer).
From the Windows retest of d378bd0: after Restart now, hookshot_log.1.txt and .2.txt were missing and the run before was overwritten. The new process started while the old one still had hookshot_log.txt open; Windows does not rename an open file, so its move to hookshot_log.1.txt failed and the new run's freopen truncated it. The old process now passes its id (HOS_RESTART_OF) and the new one waits for it to end, up to 10 s, before it moves the logs and opens its own.
|
Thanks for the retest on d378bd0. Pushed as de67a56:
Also merged main again (rumble strength #1169: both kept in the option table). |
Brings in the 1920x1080 docked output (NotRexed#1144), Windows' UTF-8 code page (NotRexed#1191) and the saves guide (NotRexed#1190). Conflicts, resolved for both sides: - gpu/vk/vk_menu.cpp: the Output resolution row keeps this branch's place among the scaling rows, its short help and its restart flag, and takes main's four sizes (kLastOutput) and smallest-to-largest order (step_output_choice, after row_fixed and the Upscaler's steps in set_row). Its value no longer says "restart" (the restart rows show "(after restart)" and offer Restart now); 2048x1152 stays "(game's)". 1920x1080 has its own per-value hint. menu_init reads HOS_OUTPUT_RES through output_choice, then restart_init. DLSS's modes and DLAA at 1920x1080 need nothing else: their sizes are twelfths of the output (1280x720, 1120x630, 960x540, 640x360), and vk_dlss.cpp takes the output's size from the picture, not from a list. - gpu/vk/vk_menu.h: output_res lists 3 1920x1080 and still says it is read once, at the start. - readme.md: the Output resolution row with 1920x1080 in this branch's table; the scaling notes take main's 1920x1080 sentence and HOS_OUTPUT_RES values. - tests/run_all.py: both test_menu_restart.py and test_output_resolution.py. Also: - gpu/vk/vk_dlss.cpp: ngx_log no longer converts NGX's lines from the ANSI code page. With the manifest's UTF-8 code page it is UTF-8 (tests/test_utf8_paths.py forbids CP_ACP in gpu/vk). - tests/test_output_resolution.py: the labels without "restart", the menu's pieces it compiles taken around the Render resolution's hint and the Upscaler's rows (upscaler_rows.h, an Upscaler stand-in), DLSS's mode sizes and DLAA at 1920x1080 checked, and hos_wide_out defined (hos_rt_pool_size reads it; it did not link without it). - tests/test_rumble_menu.py: a stand-in for step_output_choice, which set_row now calls.
Brings in the microphone input (NotRexed#937), the builder's re-lift after a failed build, and main's test fixes. Conflicts, resolved for both sides: - gpu/vk/vk_menu.cpp: - Row: main's preset_exempt goes after this branch's hint and restart. Every row main marked (Fullscreen, GPU device, FPS counter, Microphone, the mouse rows, Invert camera X and Y, Rumble strength) now spells out the fields before it (no hint: nullptr; no restart: false), so a bare "true" is no longer read as a hint or a restart flag. The GPU device row is both restart-only and preset-exempt. - set_row: main's preset_exempt check replaces the list of rows, after this branch's Render resolution snap to the new Upscaler. Main's mic_chosen comes before this branch's restart note. - The Microphone row in this branch's help style: a short static help and per-value hints (hn_mic) for Off, Default (with Windows' microphone permission), a microphone, and a chosen one that is unplugged. It applies at once, so it has no restart flag. It uses the host's mic_* calls, not an option id. - menu_tick: both dlss_probe and refresh_mics while the menu is open. - tests/test_output_resolution.py: main's hos_wide_out line. The merge before this one had defined it too. Also: - tests/test_rumble_menu.py: main's step_output_choice stand-in and preset_exempt flag take the place of this branch's, and the fixture's Rumble row passes no hint and no restart before preset_exempt.
|
Test report from an RTX 4080 SUPER (Ada) at a 3840x2160 output, which covers the "with jitter, and DLAA" part of the quality item. Setup. RTX 4080 SUPER, driver 616.92, Windows 11, Ryzen 7 5800X3D, 64 GB. Built from de67a56 with llvm-mingw 20261104 and
Quality, still camera, 3840x2160 output. Same spawn view, screenshots at presents 2000-2002 (
Merge with current main. The branch conflicts in
After that, Not checked here: DLSS together with frame generation, Reflex latency, fullscreen on the display itself (the runs were in a background window), the update offer, and non-NVIDIA GPUs. |
A Graphics row under the Upscaler, in builds with DLSS, chooses the network DLSS and DLAA draw with, by NVIDIA's
preset: K (the transformer, the default and the one tested most here), M (the second-generation transformer, NVIDIA's
pick for Performance), L (the same generation as tuned for Ultra Performance) or NVIDIA's default (preset 0: the runtime
picks per mode, which can change with a runtime update). The SDK's runtime, 310.9.16, takes M ("Using App hint Preset M"
in NGX's log for every mode).
- It has nothing to choose while the game's own upscaler runs, or while RX_VK_DLSS_PRESET sets the preset for the run
(that switch still wins, now also as 0 for NVIDIA's default, and in either case).
- A change makes DLSS's feature again with the new preset, as a new size does: no restart. Saved as dlss_model; the
Quality preset leaves it alone.
- Its help: one line for the row, one for each value (the game's own and RX_VK_DLSS_PRESET have their own).
- The log's feature line says the preset asked for, or "0 (NVIDIA's default)".
- upscaler_rows.h: the presets, RX_VK_DLSS_PRESET over them, and when the row is shown and fixed, checked by
tests/test_upscaler_rows.cpp; tests/test_upscaler_rows.py checks the menu and vk_dlss.cpp use them.
- readme: the row, its values and RX_VK_DLSS_PRESET.
…e it draws Every 600 of the game's frames the log has one line, with or without DLSS: "vk: status: 600 frames; VRAM 7.1 of 8.0 GB" (the video memory in use of what the driver allows the game, asked of it then through VK_EXT_memory_budget; without that extension it says so once, and the line gives the game's own allocations). While DLSS or DLAA draws, its details go on the same line instead of a line of their own: the jitter it was last given, the actor motion's draws, frames and CPU time since the last line, the size of its images and buffers, and its feature's when NGX says it (NGX_DLSS_GET_STATS). DLSS's own 600-frame line is gone. Every 60 frames the use is compared with the allowance: the first time it is over, "vk: over the driver's video-memory allowance (8.3 of 8.0 GB)", with ": DLAA may stutter here; Quality or Balanced uses less" while DLAA is chosen (DLSS: a lower Render resolution mode). Again only after it was back under. The work a frame is a counter. - gpu/vk/status_line.h: the line's text, the warning's once and again, its advice; tests/test_status_line.cpp checks them, and tests/test_status_line.py that the renderer uses them. - vk_vma.cpp: vram_usage_now (VMA asks the driver again for a new frame index, shared with system_heap_usage). - readme: the status line where the log is described, and an 8 GB note in the DLSS section.
…s picture With the menu opened during the start-up screen, the Window scaling filter read "Not needed (window = output)" in a 1280x720 window with a 2048x1152 output: the start-up screen is presented at the window's own size, and present_image took every present's size as the picture's. Now only the game's frames (and the made ones) note the sizes; until the first one they are not known, and the row counts as needed (upscaler_rows.h filter_needed, already so). Checked by tests/test_upscaler_rows.py.
A tester saw DLSS's minimum frame rate fall far below the game's own upscaler's at a 3840x2160 output, and once a 1.5 s stall. Making DLSS's feature again in play would do that, so the log now says each time why: - "vk: NVIDIA DLSS: feature <n> made (<why>; frame <f>, <ms> of CPU): ...". The reason is one of: the first; again after it was let go; the output's size changed; another preset from the DLSS model row; or the render size left the feature's range (the range is now in the line). - The 120-frame release names the last frame DLSS was asked for and the frame it let go at. - The status line adds the slowest frame since the last line (present to present). While DLSS draws it also adds the frames it drew, the frames a network of the upscale's shape and formats ran without it (its input not this frame's scene, which used to pass without a word), and how many features have been made.
Brings in motion controls (NotRexed#1114), 5.1 surround (NotRexed#1182), the pipeline cache over 2 GB (NotRexed#1226), the queued pipeline build run at once (NotRexed#1129), the floating-point fixes (NotRexed#917), the builder's cleanup stop (NotRexed#1218) and the render target pool's 512 MB minimum. Conflicts, resolved for both sides: - gpu/vk/vk_api.h: main's motion rows took option ids 18 to 22, which DLSS's camera jitter used (18 to 20). DLSS's now start at 24 (HOS_OPT_DLSS_JITTER_W, _H, _PHASES); every use is by name (host/rx_backend.c, vk_replay.cpp). - gpu/vk/vk_replay.cpp: the option table answers both the motion rows and DLSS's jitter. - gpu/vk/vk_menu.cpp: the motion rows in this branch's Row shape and help style: - Motion controls gets a short help and a hint for each value (hn_gyro). Which pads have a gyroscope moved to On's hint. - The sensitivity and invert rows pass no hint and no restart before preset_exempt, as main had them. - tests/test_rumble_menu.py: main's reading of every HOS_OPT_* enum (it covers DLSS's jitter ids too). Also: - tests/test_output_resolution.py: the render target pool is now 512 MB at least (main's nn_applet.c), so the test expects 512, not 305. Main's own copy still expects 305 and fails.
|
Thanks to everyone who tested this: the Windows reports on the RTX 5080, which caught the restart log loss, the shadow flicker and the start-up quirk; the RTX 3090 runs on Linux; and @itaskaev2's detailed 4080 SUPER report at 3840x2160. That one confirmed the jitter and shadows on a third card generation, measured DLSS at about 1.6 ms of GPU and DLAA at about 4 ms, and showed 2.5x less shimmer. The last commits before the merge:
@itaskaev2, about the one 1.5 s stall in your DLSS Quality +
A 1.5 s slowest frame next to a "made" or "let go" line means DLSS made its feature again; frames left to the game's network point to a shader compile on the fallback path instead. Either way, a new issue is the best place for it now that this is merged. Still untested, if anyone has the setup: DLSS together with frame generation, Reflex latency, true fullscreen, the DLSS update offer, and 8 GB cards. |
|
PR-841-VALIDATION-REPORT-EN.md codex found 2 issues in the code, attaching report |
NVIDIA DLSS for Hookshot: Super Resolution, DLAA and Reflex, through NVIDIA's NGX library and Vulkan directly (no Streamline), on Linux and Windows. Opt-in at build time and off by default: a normal build, CI's release builds and the Builder without its NVIDIA DLSS tick are unchanged and fetch nothing.
No Streamline: everything uses NVIDIA's NGX library from the DLSS SDK directly, plus Vulkan's own
VK_NV_low_latency2for Reflex, so it works on native Linux as well as Windows.NVIDIA's files. Nothing of NVIDIA's is committed, bundled or uploaded. Only someone who opts in fetches them, for their own build:
scripts/ngx_sdk.pydownloads them from NVIDIA's own repository (NVIDIA/DLSS on GitHub) at one pinned commit, checks every file against a SHA-256 in the script, and refuses to download until NVIDIA's license is accepted (--accept-nvidia-license, or the Builder's tick, which shows the license first). The only automated fetch is thevulkan-dlssCI workflow, which downloads the headers and the .lib to check that the Windows link still works: it runs by hand (workflow_dispatch), and on a push or pull request only when a file DLSS's build depends on changes (paths:filters in its own workflow file, so the other jobs are untouched). It accepts NVIDIA's license for that CI run only, and uploads and caches nothing.What's in it
DLSS Super Resolution
ngx/. DLSS then replaces OoT's gameplay upscale network (fd029a31) every frame, preset K. Input and output sizes follow the game's: with Output resolution (2048x1152, 2560x1440, 3840x2160) and a full-size 3D at 4K #849 the output is 2048x1152, 2560x1440 or 3840x2160 and the Resolution row scales the 3D against it; the feature is made again when the output size changes or the render size leaves its mode's range. Colours and brightness match the native path.RX_VK_DLSS_DEBUG=motionshows them.Camera jitter
nx2/oot_jitter.pypatches the function that fills each view's uniform buffers (three loads of the view's projection) to add a Halton sub-pixel offset to P[0][2]/P[1][2] in registers, only while DLSS draws. Nothing in the game's memory changes: the GPU gets the jittered projection, and the CPU (shadow cascade fitting, culling) keeps the real camera. The host asks the renderer for the scene size and the sequence's length throughrx_option(HOS_OPT_DLSS_JITTER_W,_H,_PHASES). The renderer reads the offset back from P[0][2]/P[1][2], passes it to DLSS and removes it from the motion vectors. Verified in game: the offsets go through the sequence as expected, with no faults.The sequence is 8 × (output / render)² frames long, 8 (DLAA) to 64, as NVIDIA's guide recommends. The sites are only lifted into a build with NVIDIA's SDK (make_game.py sets
OOT_DLSS_JITTER=1for the lift, part of its cache key).dlss_jitter_clip(indlss_math.h, tested) picks the projections that get it: perspective, at 16:9 or the window's ratio; orthographic and 1:1 projections (the shadow maps, the per-character projector) never do.Shadows (reported by a tester): the first version put the jitter into sead's projection itself, so the shadow cascades, fitted to the view frustum, followed it and their origin moved by about 0.17 shadow texels a frame with a still camera at DLAA 3840x2160 (up to 0.4), against 0.003 with the jitter off. With the jitter only in the uniform buffers, it is back to the jitter-off level; picture detail is unchanged and fps is the same (38.5–39.4 at DLAA 3840x2160, jitter on or off).
DLAA
DLAA is the NVIDIA DLAA choice of the Upscaler row (
RX_VK_DLSS=2; see Settings). DLAA runs DLSS at a scale of 1 on the scene as drawn. At full scale the game doesn't run its 4/3 upscale but the same network as a same-size pass (9dc98871eb7fc101); DLSS replaces that pass too, which also covers the Resolution row's top step with DLSS on. The 3D is held at full scale whatever the Resolution row says, anddlss_out.compscales the result to the game's upscale output.Reflex
A new NVIDIA Reflex row: Off / On / On + Boost (
RX_VK_REFLEX=0/1/2).vkLatencySleepNV, simulation end and render-submit start at the game's present, render-submit end before the present's submit, and present start/end aroundvkQueuePresentKHR.VK_NV_low_latency2andVK_KHR_present_idare still enabled whenever the device has them, so the row works without a restart. Enabled alone they do nothing: nothing of them is used until the row is on.RX_VK_REFLEX=-leaves them off.RX_VK_REFLEX_LOG=1(opt-in) prints the driver's own timings (vkGetLatencyTimingsNV) every 300 frames while Reflex is on.Status
scripts/ngx_sdk.pyelsewhereSince the review
vk_dlss.cppnow includesvk_work.hand counts NGX's own commands (note_work()after the feature's creation and the evaluation), sofull_barrier's dedupe no longer leaves them out.Runner.Info's newngxSdkargument.error_codeoverload: a folder that cannot be made no longer ends the game. Windows paths are wide all the way.descriptors_disturbed()on every way out once commands are recorded;failedandlast_frameare atomics;make_pipelinesno longer leaks; a 60 s timeout on each SDK download.RX_VK_DLSS_JITTER_SIGN,RX_VK_DLSS_SQUARE) and the camera dump are gone, stale comments fixed, the jitter's options named (HOS_OPT_DLSS_JITTER_*).gpu/vk/dlss_math.hand tested on the CPU (tests/test_dlss_math.py); the menu's rules are tested by behaviour intest_upscaler_rows.cpp.vulkan-dlssCI job is in its own workflow: manual, or on DLSS's files only.ngx-sdk/is git-ignored.host/rx_backend.cnow includesgpu/vk/dlss_math.h, so the host objects' cache key hashes it and the Linux AppImage carries it.vk_main.cpp,vk_replay.cpp,nx2/build.pyandtest_shader_stamp_cache.py, all resolved keeping both sides.vk_menu.cpp: Insert menu: free the mouse while it's open, and make the menu clickable #946 moved the Insert menu's value changes intoset_row(keys, clicks, the wheel and slider drags all go through it), and the Upscaler's rules (rows that cannot change here stay as they are, the Upscaler and Render resolution step throughupscaler_rows.h, another Upscaler snaps the Render resolution) moved intoset_rowwith it, so the mouse follows them too.Checked again on Linux after these changes (RTX 3090): builds with and without the SDK (without it: no NGX in
hookshot_vk.so, nongxfolder, no jitter sites in the lifted code), the Windowshookshot_vk.dllcross-built with the SDK through llvm-mingw (imports only Windows' own DLLs and the UCRT),tests/run_all.pygreen, actionlint clean. In game: DLSS draws the upscale correctly, the jitter steps through its sequence, Reflex On runs with its markers and timings, and with Reflex Off the log has no Reflex line at all.Moving objects
OoT has no velocity buffer: its scene pass writes a single B10G11R11 colour target plus depth, and its upscale network samples only the colour, so DLSS had only the camera's motion and anything that moves by itself ghosted.
Skinned draws (Link, the other characters and animals) are ordinary indexed vertex draws with a recognisable layout: position, bone weights and bone indices, with the bone palette in a uniform buffer.
vk_draw.cpppicks those that draw into the scene, and at the upscale each is drawn again into the motion image, each vertex skinned with this frame's bones and the previous frame's. The vertex and index data are read from the game's own buffers, and the poses are copied on the GPU into two buffers that take turns, so nothing is read back to the CPU. A pixel takes an actor's motion only where its depth matches the scene's; anything else keeps the camera motion.RX_VK_DLSS_ACTORS=0turns it off, andRX_VK_DLSS_DEBUG=motionshows the actors' vectors.Measured by running Link past the camera at a 2048x1152 output (headless, frame clock, same save), the mean error over Link against the game's own upscale drops by 40% (16.0 → 9.7, lower in every frame); over the whole frame 12.5 → 7.9. The runs differ by a few pixels in Link's position, so the numbers are indicative. A control with the camera turning and Link standing is unchanged (13.78 vs 13.81). Cost: about 0.04 ms of GPU and 0.07 ms of CPU a frame.
Still on the camera's motion: particles (Navi), water, the floor mist, foliage moved in its vertex shader and other mesh-shader geometry, Link's face and hair (drawn with another layout), rigid objects without bones, and an actor's first frame on screen.
Two test switches come with it:
RX_VK_DLSS_TRACElogs one frame's draws into the scene (how the above was found), andRX_VK_SHOT_GO=<file>takes a screenshot at every present while that file exists.Settings
DLSS is a choice in the Graphics tab's Upscaler row (Game's own, NVIDIA DLSS, NVIDIA DLAA), which replaces the separate NVIDIA DLSS row. The scaling rows read in the picture's order: Output resolution, Upscaler, Render resolution, Window scaling filter (formerly Upscaling), Sharpness. NVIDIA Reflex sits with Frame pacing.
With DLSS, Render resolution shows DLSS's modes: Auto, Quality (67%), Balanced (58%), Performance (50%) and Ultra Performance (33%). With DLAA it shows Native. DLSS now replaces the game's upscale at every one of these shares, not only at 75%: at the other shares the game runs a different upscale network, which DLSS previously left running.
DLSS and DLAA are offered only where they can run (an RTX GPU, the runtime present), and the help says why when they are not. In a build without NVIDIA's SDK, which includes every official build, the Upscaler row is not shown at all, and Render resolution is the game's own row. The Window scaling filter's help says it matters only when the window's size differs from the Output resolution; while they match, the row shows "Not needed (window = output)" and keeps the saved value. Reflex likewise needs VK_NV_low_latency2. Quality presets leave the Upscaler as chosen (with DLSS, their resolution becomes DLSS's nearest mode), and
RX_VK_DLSS=0/1/2still overrides the row. NVIDIA DLSS and DLAA look softer than the game's own upscaler, so their help says to set Sharpness to around 80%.The menu's help is now a short line on what a row is, plus a separate line for the value chosen (dimmer, with a bar), so the details sit with the value they describe; the panel keeps its height as values change.
The five settings read only at start (GPU device, Output resolution, Handheld profile, Ultrawide, Robust buffers) show "(after restart)" only while they differ from what runs, the Graphics tab lists them at its foot, and closing the menu asks "Restart to apply …?" with [Later] (keeps the change for the next start, asked again only after another change) or [Restart now]. Restart now saves the options and the pipeline cache, closes the game the normal way (the start-up screen's close path), then starts it again with the same arguments and environment: on Linux by re-executing in the same process (a launcher or stream waiting on it keeps it), on Windows with
CreateProcessW. Tested headless: Output resolution 2048x1152 → 2560x1440 across a restart, options and pipeline cache written, saves untouched. Tests:tests/test_menu_restart.py, plus additions totest_menu_hitandtest_upscaler_rows.Measurements
RTX 3090, Linux, driver 615.78, 2560x1440 output, Kokiri Forest start spot, 45 s per run after a 15 s settle, each run alone on the GPU. fps is the median of the renderer's own present counts, with p10–p90 in brackets.
DLSS costs about 2.3 ms a frame over the game's own network, and DLAA about 3 ms. DLAA draws 1.78x the pixels but is only 0.7 ms slower than DLSS, because the frame here is mostly bound by the CPU and replay rather than by the GPU.
Reflex, in a 1280x720 window with DLSS off, using the driver's own
vkGetLatencyTimingsNVreports of simulation start to present end:Reflex saves about 1 ms, close to run-to-run noise: at this CPU-bound frame rate the GPU queue isn't deep enough to add much delay. Boost made no difference. The Off run had a few hitches; with one run per mode, it isn't clear whether Reflex removes those.
Windows
DLSS Super Resolution, DLAA and the camera jitter now build for Windows as well; Reflex needed no change. NVIDIA's SDK is still never committed or bundled: each build downloads it for itself, only after NVIDIA's license is accepted.
scripts/ngx_sdk.pyreplacesscripts/linux/ngx-sdk.sh, which stays as a thin wrapper. It has one file table per platform at the pinned NVIDIA/DLSS commit 9981c635 (v310.9.16), checks every file against its SHA-256, and needs--accept-nvidia-licenseor a typed yes. On Windows it fetches LICENSE.txt, the headers,x64/nvsdk_ngx_s.libandrel/nvngx_dlss.dll.The llvm-mingw build links NVIDIA's static
nvsdk_ngx_s.lib(built for MSVC's static runtime, /MT) intohookshot_vk.dllwith/nodefaultlib:libcpmtand-ladvapi32. What it needs of MSVC's runtime comes fromgpu/vk/vk_ngx_msvc_rt.cppandhost/mingw_msvc_shims.c, compiled only for mingw with the SDK. The DLL still imports only Windows' own DLLs and the UCRT. Without the SDK, the build is unchanged (same objects, imports, exports and section sizes).At run time NGX is given
RX_VK_DLSS_DIR(when set),<game>\ngxand%LOCALAPPDATA%\Hookshot\ngx, and uses the newestnvngx_dlss.dllit finds across them. DLSS's own paths are wide all the way (GetModuleFileNameW,_wgetenv). The game itself does not yet start from a folder whose name has characters outside the system's ANSI code page; that is a separate fix, not part of this PR. NGX's core,_nvngx.dll, is the driver's and is never put beside the game.NGX's parameter object is read and set through the SDK's C functions (
NVSDK_NGX_Parameter_GetI/GetUI/SetUI), not its C++ virtual methods: the object comes from NVIDIA's MSVC-built library, and llvm-mingw's Itanium vtable order differs, so the virtual calls only worked because the overloads they landed on had the same size.The Hookshot Builder has an opt-in tick, "NVIDIA DLSS (downloads NVIDIA's SDK under NVIDIA's license)", off by default. Ticking it shows the license; the build then fetches the SDK into
%LOCALAPPDATA%\HookshotBuilder\ngx-sdk, passesHOOKSHOT_NGX_SDKthrough, andmake_game.pyputsnvngx_dlss.dllin<game>\ngx. Updating a folder built with DLSS keeps DLSS.A new CI workflow,
vulkan-dlss, fetches the headers and .lib only, buildshookshot_vk.dllwith them through llvm-mingw, and checks the link and the imports. It runs by hand, and on pushes and pull requests only when DLSS's files change (gpu/vk/vk_dlss*,vk_reflex*,vk_ngx_msvc_rt.cpp,host/mingw_msvc_shims.c,upscaler_rows.h,dlss_math.h, the DLSS shaders,nx2/oot_jitter.py,nx2/build.py,nx2/make_game.py,scripts/ngx_sdk.pyand the workflow itself). It accepts NVIDIA's license for that run only and uploads nothing.The Builder's C# is compiled by CI (the Compile Windows Builder step: 0 errors), and its builder workers test now passes
Runner.Info's newngxSdkargument.NGX's log (
RX_VK_DLSS_LOG=1) has a few lines that look like errors and are not: "failed to load NGXCore: 126" (twice) is NVIDIA's library looking for its core beside the game before it loads the driver's; "Feature … failed to load from cache" is NGX finding none of its own downloaded models. The readme says so. The two Windows warnings at init, "passed vkGetInstance/DeviceProcAddr to NGX SDK, but snippet is out of date" and the same forNVSDK_NGX_FeatureCommonInfo, come from NGX probing every feature snippet it knows at init (denoiser, inpainting, DLISR, slow-mo and so on, as its log shows on Linux), some of them older ones of the driver's. Our own DLSS snippet is 310.9.16, the SDK's. Hookshot uses the current init (NVSDK_NGX_VULKAN_Init_with_ProjectIDwith both procedure addresses and the feature info), and both are needed: the procedure addresses because the renderer loads Vulkan itself, and the feature info for the search paths and the logging. So the warnings are left as they are.Never share a game folder built with NVIDIA's files. The readme credits NVIDIA DLSS; NVIDIA's files are under NVIDIA's RTX SDKs license.
Known limits of the shims: they don't check /GS cookies in the exception-handler path (the same as the existing
__GSHandlerCheck), and running out of memory inside NGX ends the process with a message instead of throwing.Windows checks (12 of 14 done on Windows 11, RTX 5080, driver 621.14; last run on 842121f)
dotnet build builder/HookshotBuilder.slncompiles: a full rebuild gives 0 warnings, 0 errors (.NET SDK 10.0.403, net8.0-windows); CI's Compile Windows Builder step also compiles itnvsdk_ngx_s.libandnvngx_dlss.dll, all hash-checked; the license notice, fetch lines and "with NVIDIA DLSS" in the Builder's log;ngx\nvngx_dlss.dllin the game folder with the right SHA-256; no_nvngx.dllanywhere. (bin\needs a packaged Builder to check: a Builder run from the source tree has none)ngxfolder or NVIDIA files, no NGX code inhookshot_vk.dll, and the menu has no Upscaler row (NVIDIA Reflex is still there: it needs no SDK)make_game.py --updatewithHOOKSHOT_NGX_SDK) rebuilt a DLSS folder "with NVIDIA DLSS" in 7.5 min, putnvngx_dlss.dllback and kept the saves; the launcher's Update button itself only shows when a newer release existsnvngx_dlss.dllanywhere, the log names the folders it looked in, then "the game's upscale runs (DLSS not available)", and DLSS isn't offered; a non-NVIDIA GPU or a missing NGX core is still to tryHookshot-DLSS Zélda, NGX loaded…Hookshot-DLSS Zélda\ngx/nvngx_dlss.dlland started. A name outside the ANSI code page (Japanese on an English Windows) stops the game itself before DLSS runs: that is the host's path handling, fixed in a separate PR%LOCALAPPDATA%\Hookshot\ngx\nvngx_dlss.dllis used when the game folder'sngxhas none;RX_VK_DLSS_DIRwins over both (tested with the same version in each place; with different versions NGX takes the newest)RX_VK_DLSS_LOG=1: NGX's own lines appear in our log (253 to 557 per run: NGX's core from the driver's DriverStore, "called from module hookshot_vk.dll", our snippet and version, "Using App hint Preset K"); its lines are now written as UTF-8 like ourstests/test_ngx_sdk.pyandtests/test_make_game_ngx.pypass, andmake_game.pywithHOOKSHOT_NGX_SDKbuilds with DLSSHOOKSHOT_MSVC=1(clang, MSVC target) withHOOKSHOT_NGX_SDK:hookshot_vk.dllbuilds and links (291 s, MSVC 14.51, Windows SDK 10.0.26100), imports only ADVAPI32, GDI32, KERNEL32 and USER32vulkan-dlsspasses on windows-2022: green on the PR at d378bd0 (and every step was also run locally on Windows with CI's pinned llvm-mingw: 187 files, 0 failed, imports check passed)Note for testers: NGX starts on demand (the Insert menu's first opening, or the first upscale with DLSS), so a log from the title screen has no NGX lines yet.
How to build it
python -I scripts/ngx_sdk.py --accept-nvidia-license <sdk folder> --game <game folder>(or the Builder's NVIDIA DLSS tick on Windows;scripts/linux/ngx-sdk.shstill works). It fetches NVIDIA's DLSS SDK headers, library and runtime from NVIDIA/DLSS at one pinned commit, checks each file's SHA-256 and copies the runtime into the game folder. NVIDIA's files are under NVIDIA's own license (LICENSE.txt is fetched beside them) and are never committed here.HOOKSHOT_NGX_SDK=build-linux/ngx-sdk. Without that variable the build is the same as before: no Upscaler row, and no camera jitter sites in the lifted code.RX_VK_DLSS=1/2), and Reflex in its row (RX_VK_REFLEX).Tested on Linux with an RTX 3090, driver 615.78.08. Windows: tested on Windows 11 with an RTX 5080 (driver 621.14), see the checklist.