[notquiteog]

What

AMD FSR 3.1 upscaling and FSR 3.1 frame generation, split out of #1335 as asked. Nothing else from #1335 is in it.

• Upscaling: a fourth value on the existing Upscaler row, AMD FSR 3.1, after DLAA (dlss=3 in hookshot_options.ini; 1 and 2 keep meaning DLSS and DLAA). It takes the gameplay upscale network's dispatch: the same dispatch DLSS takes, found the same way.
• Frame generation: with Frame generation at 2x, FSR 3.1's frame interpolation makes the frame in between. It uses the scene's depth and motion. Where it cannot run, the renderer's own interpolation runs as before, and it always handles Auto and 3x and above (see below).
• Built from AMD's open shader sources. FidelityFX SDK v1.1.4 (commit c6efa6bf, MIT) is vendored unmodified in third_party/fidelityfx-sdk and compiled into gpu/vk/shaders/fsr3_*_spv.h. Every pass is a dispatch of ours. FSR 3.1 needs no AMD runtime and no DLL: nothing of AMD's is linked or loaded at run time, so it is in every build on every platform. AMD's runtime DLLs are only for FSR 4. FSR 4 is not in this PR (FSR 4.1 is AMD FSR 4.1 as a second upscaler engine, on #841's frame chain #1008's).
• No NVIDIA changes: no DLSS, DLSS-G, DLSS 5, NGX or Streamline code. The only change to vk_dlss.cpp is in dlss_mode(). It used to clamp the row's value into 0..2, so 3 would have turned into DLAA. Now any value other than 1 or 2 means "not DLSS".
• No release or packaging changes and no licence acceptance in CI. The only workflow edit adds glslang-tools to source-regressions.yml's apt line, because tests/test_fsr3_shaders.py compiles every vendored pass again and checks the checked-in headers.

Supersedes #1313: its commit is this PR's first commit, unchanged apart from one context line in tests/run_all.py after the rebase.

Commits:

1. build(vk): the vendored passes, scripts/fsr3_shaders.py, the 28 headers and their test (build(vk): vendor AMD's FSR 3.1 passes and the script that compiles them #1313)
2. feat(vk): FSR 3.1 upscaling (vk_fsr3.*, vk_fsr3_inputs.*, fsr3_math.h, shaders/fsr3_prep.comp), the jitter in every lift
3. feat(vk): FSR 3.1 frame interpolation (vk_fsr3fi.*, the chain in vk_framegen.cpp)
4. feat(menu): the Upscaler row's FSR 3.1 value, the Frame generation hint, and readme.md

Why

Players have asked for FSR 3.1 frame generation, and #1335 was too big to review and test in one piece.

Inputs: what comes from #841 and what does not yet

All of FSR's inputs are made in one file, gpu/vk/vk_fsr3_inputs.cpp, behind one function, prepare(), which returns one Frame struct. Moving FSR 3.1 onto #1008's shared temporal inputs means changing that file alone.

• Jitter: NVIDIA DLSS, DLAA and Reflex (NGX + Vulkan, Linux and Windows, opt-in) #841's, the one jitter in the game. nx2/oot_jitter.py with hos_dlss_jitter_clip. The fork's own jitter (a viewport offset in vk_draw.cpp) is not here.
  • nx2/lift.py now installs the jitter in every standalone lift, not only in builds with NVIDIA's SDK. make_game.py no longer sets OOT_DLSS_JITTER, and oot_jitter.wanted() is gone. The lift key covers nx2/, so the first build after this PR lifts the game again. tests/test_make_game_ngx.py no longer checks the old switch.
  • The jitter stays zero unless DLSS or FSR 3.1 draws. rx_option(HOS_OPT_DLSS_JITTER_W/_H/_PHASES) returns dlss_jitter_size(), or FSR's fsr3_inputs::jitter_size() (the scene's size while FSR has drawn in the last 30 frames). The phase count is dlss_jitter_phases(), the host's own sequence length, and FSR is given the same count. RX_VK_FSR_JITTER=0 turns it off for a run.
  • FSR reads the offset back from the camera's uniform buffer (P[0][2], P[1][2]), as vk_dlss.cpp does. It uses dlss_math.h's jitter_to_pixels and unjitter to convert the offset and take it off the motion.
  • NVIDIA DLSS, DLAA and Reflex (NGX + Vulkan, Linux and Windows, opt-in) #841's names (HOS_OPT_DLSS_JITTER_*, hos_dlss_jitter_clip) are kept, so its interfaces do not churn under AMD FSR 4.1 as a second upscaler engine, on #841's frame chain #1008.
• Camera (near, far, field of view): from the projection, with no hook in the game. nx2/oot_camera.py and host/hos_oot_camera.c are not here.
  • The camera is taken from the same uniform buffer and with the same checks as dlss_note_scene_draw.
  • fsr3_math.h derives FSR's device_to_view_depth factors and tan(fov/2) from the projection at +0x70. This was checked in a run: p22 -1.0002, p23 -0.20002 gives near 0.1, far 1000.5 and a 45.00 degree field of view.
  • The scene's depth is 0 at the near plane (not reversed). The passes are compiled for reversed depth, so FSR is given 1 - d.
  • The fork's camera hook published near 1.0 and far 20000 (sead's projection object, not the one the scene is drawn with) and gave FSR reversed-infinite factors. Neither matches the depth FSR was given.
  • tests/test_fsr3_math.cpp checks the factors against FSR's own view-distance formula, for both depth directions and for finite and infinite far planes.
• Motion: the camera's only, for now.
  • shaders/fsr3_prep.comp is the fork's camera-motion pass, reworked to dlss_prep.comp's math: reproject the depth from this frame's jittered clip space to the previous frame's unjittered one, and take this frame's jitter off. The same pass also writes FSR's colour and depth copies.
  • NVIDIA DLSS, DLAA and Reflex (NGX + Vulkan, Linux and Windows, opt-in) #841's actor motion (the skinned draws drawn again with last frame's bones) lives inside vk_dlss.cpp behind HOOKSHOT_NGX, so a build without NVIDIA's SDK has none of it. Moving it out from behind the gate is AMD FSR 4.1 as a second upscaler engine, on #841's frame chain #1008's refactor (vk_temporal.cpp), and this PR does not duplicate it.
  • Until AMD FSR 4.1 as a second upscaler engine, on #841's frame chain #1008 lands, things that move by themselves can trail with FSR 3.1, as the row's help says. When it lands, vk_fsr3_inputs.cpp's prepare() takes AMD FSR 4.1 as a second upscaler engine, on #841's frame chain #1008's inputs and FSR gets the actor motion.
• Colour: the network's input as it samples it (its square-root encoding), clamped to 0..1, with NaN set to 0. FSR's output is squared back into the network's output by dlss_out.comp, the same pass DLSS's picture goes through.

Left out of the fork's version:

• the viewport jitter
• the camera hook
• upscale_seam.h, upscale_contract.h, upscale_format.h and their tests (the stand-in takes the dispatch the way dlss_stand_in does)
• FSR's own RCAS (the Sharpness row already sharpens every upscaler's picture)
• the diagnostics (RX_VK_FSR_STAGE, the VP hunt, the MRT census)

Fixed in the port:

• The fork's FSR output was blitted into the network's output without being squared back.
• The fork's FSR files did not include vk_work.h, so full_barrier could leave out a barrier after their work.
• The fork's camera-motion image never had its first layout transition recorded.
• When the size changed, the fork destroyed images that frames still in flight could be using. They are now freed a few presents later.

Frame generation

The chain in vk_framegen.cpp is: FSR 3.1, then the renderer's own blend for any present FSR does not take (no inputs for that present, a reset, a GPU that cannot run the passes).

• FSR 3.1's interpolation has no phase input: it only makes the midpoint. So it is used at 2x only. Auto and 3x to 8x use the renderer's own.
• A present FSR did not see breaks the sequence it keeps its previous frame by, so the next present is a reset. It never interpolates against a stale frame, and each frame's depth and motion are used by one present only.
• The Frame generation row's 2x hint says when FSR 3.1 makes the frame. RX_VK_FSR_FG=0 turns FSR's interpolation off for a run (=1 forces it on over the row).

GPUs and drivers

FSR 3.1 is offered where the device has what the passes need. The Upscaler row is shown on such a GPU even in a build without NVIDIA's SDK. These are on top of the renderer's own floor (Vulkan 1.3: dynamic rendering, synchronization2, timeline semaphores).

• Upscaler (fsr3::supported()):
  • VK_KHR_push_descriptor with maxPushDescriptors of at least 15
  • shaderStorageImageExtendedFormats and shaderStorageImageWriteWithoutFormat
  • storage and sampled images (optimal tiling) in R8_UNORM, R16_SFLOAT, R8G8B8A8_UNORM, R16G16B16A16_SFLOAT, R16G16_SFLOAT, R32_SFLOAT, R32_UINT, R32G32B32A32_SFLOAT and R16_SNORM
  • subgroup basic and quad operations in compute are used where present; without them the two pyramids run as their wave-less variants
• Frame interpolation (fsr3fi::capable()):
  • push descriptors for the widest pass (19)
  • shaderStorageImageReadWithoutFormat, shaderStorageImageWriteWithoutFormat and shaderStorageImageExtendedFormats
  • subgroup basic, arithmetic and quad operations in compute
  • storage images in R8_UINT, R8G8_UNORM, R16G16_SINT, R16G16_SFLOAT, R32_UINT, R32_SFLOAT, R16G16B16A16_SFLOAT, and the presented R8G8B8A8_UNORM

The log says which way it went: vk: fsr3: AMD FSR 3.1 can run on this GPU, or vk: fsr3: not on this GPU: <why>, and the same for fsr3fi. Tested only on an NVIDIA RTX 5060 Ti with driver 615.71.09 on Linux.

Testing

Linux, RTX 5060 Ti, driver 615.71.09. D=<deps> is build-linux/deps.

Builds (both link):

• EMSDK=$D/emsdk RX_CPM=$D/cpm OOT_WORK=... RX_MARCH=x86-64-v3 RX_VK_ONLY=1 python -I nx2/build.py --standalone --jobs 8: compiled, 0 failed, linked hookshot_vk.so (no NVIDIA SDK)
• the same with HOOKSHOT_NGX_SDK=<ngx-sdk>: compiled, 0 failed, linked ("with NVIDIA DLSS")
• each of commits 2 and 3 on its own: the renderer builds and the tests touching it pass

Tests:

• After rebasing onto current main (0828863): the renderer builds and links without NVIDIA's SDK, and tests/run_all.py
  (with the CPM headers found, so the SPIR-V and Vulkan header tests run too) gives 202 passed, 0 failed, 5 skipped.
• tests/run_all.py on main (a116390): 180 passed, 0 failed, 19 skipped
• tests/run_all.py on this branch: 182 passed, 0 failed, the same 19 skipped. The two new ones are test_fsr3_shaders.py and test_fsr3_math.py.
• On the earlier base 75a676e: 166 / 168 passed, 0 failed, 17 skipped.

Whole game: make_game.py --update <a copy of the game folder> --jobs 10 built the whole game from this branch in 10 minutes. The lift logged dlss jitter: 3 of 3 sites in a build without NVIDIA's SDK. A run of that build (FSR 3.1 + 2x, pacing on) gave a game fps of 58.4 and 117.2 frames shown a second, with FSR 3.1 making 600 of 600 frames and no refusal lines.

In game:

• Setup: the Kokiri Forest scene with Saria, about 1800 draws a frame, 1536x864 to 2048x1152. HOS_INPUT_FRAME_CLOCK=1 HOS_REALTIME_STEP=0 RX_VK_PRECOMPILE=0, the same HOS_INPUT_SCRIPT in every arm, and a stop at 6000 presents. The game's fps is the mean of the vk: N presents, X fps lines between presents 4800 and 6000.
• The runs were windowed on Xvfb, not HOS_HEADLESS: frame generation is off without a window (framegen_on() checks !dev.offscreen).
• No build ran during an arm. Arms run while another process was using the GPU were discarded and run again.
• The arms in the table ran on this branch before its rebase from 75a676e onto a116390. The FSR code is the same; the rebase only resolved conflicts with frame generation Auto, the menu and run_all.py. The whole-game run above is on the rebased branch.

Arm	Game fps (2 runs)	Shown fps	Frames shown 7-9 ms apart
Game's own upscaler	59.7, 59.3
FSR 3.1	59.4, 58.5
FSR 3.1 + FG 2x, pacing on (FSR makes 600 of 600)	58.4, 59.7	116.6, 119.7	1994 of 2332, 2287 of 2394
FSR 3.1 + renderer's own 2x, pacing on	59.3, 59.9	115.8, 120.0	1902 of 2318, 2333 of 2400
FSR 3.1 + FG 2x, pacing off	55.6, 57.4	109, 115
FSR 3.1 + renderer's own 2x, pacing off	58.4, 58.2	118, 117

• The game holds its 60 fps cap with FSR, so the cost is clearer in RX_VK_PASS_TIMING=1: FSR 3.1's seven passes take 0.60 ms a frame, against the game's network's 0.078 ms. Frame interpolation's 17 passes take 1.17 ms a frame (0.61 ms of that is the optical-flow search).
• Late frames (more than 2 ms after their time): none in any FSR arm, and 3 in one run of the renderer's own interpolation.
• With pacing off, the game ran 1 to 3 fps slower with FSR 3.1's interpolation than with the renderer's own (55.6 and 57.4 against 58.4 and 58.2; the lowest line was 45.3). With pacing on, the two were within run-to-run spread.
• With pacing off, both interpolators show about half the frames less than 7 ms after the frame before. That is the display thread showing them as they come; it is not FSR's doing.
• Log lines in every FSR arm:
  • vk: fsr3 1536x864 -> 2048x1152 with assembled depth and camera motion (network fd029a317a9a4b94 ...)
  • vk: fsr3 inputs: ... near 0.1000 far 1000.53 fov 45.00 deg, depth 0 at the near plane: reversed for FSR
  • every 600 frames, vk: fsr3: N frames drawn ... this one's jitter +0.312 -0.352 px of 14 phases (the jitter changes every frame)
  • in the 2x arms, vk: fsr3fi: 600 of the last 600 made frames by FSR 3.1
  • no refusal lines once the scene was reached

Visual check:

• RX_VK_SHOTS + RX_VK_SHOT_MADE=1 (a shot every 150 presents, 6000 presents): FSR's frames and FSR's made frames look right, with correct colour and no garbage, and the renderer's own flash check wrote no white_* or flash_* file.
• RX_VK_DUMP=5400: the HUD pass after the upscale (step 266) draws over FSR's picture at 2048x1152 and looks right.

Not covered

• Windows: not built or run here.
• AMD and Intel GPUs: not run. The checks above decide where FSR 3.1 is offered; whether they pass on those GPUs is untested.
• DLSS with FSR 3.1 frame interpolation (DLSS upscaling, FSR making the frame) is wired, but the NGX build was only linked, not run.
• Actor motion for FSR 3.1: waits for AMD FSR 4.1 as a second upscaler engine, on #841's frame chain #1008.
• FSR 3.1's interpolation at Auto or at 3x and above: the renderer's own does those (midpoint only).
• The made frame is interpolated from the presented frame, HUD included. The game has no HUD-less colour to give FSR, so the HUD can smear when the scene moves fast.
• Long play: the runs are 6000 presents in one place.

🤖 Generated with Claude Code