diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index eef9a591..866e1bda 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,7 +29,7 @@ game data, leave it out. | | What | |---|---| -| **Test on other hardware** | The project has been tested on one Windows PC (Ryzen 5 7600X, RTX 4060 Ti) and one Linux PC so far. Reports from AMD and Intel cards, other NVIDIA drivers, other distributions and SteamOS are useful even when everything works. | +| **Test on other hardware** | NVIDIA cards are the most tested; the README's [status table](readme.md#project-status) says what works where. Reports from AMD and Intel cards, other NVIDIA drivers, other distributions and SteamOS are useful even when everything works. | | **Play further** | The tested area is the start of the game. Reaching new areas, cutscenes, menus and items finds problems the developer has not seen. | | **Report what is broken** | Use the [issue forms](https://github.com/NotRexed/Hookshot/issues/new/choose): game problem, build problem or idea. Include the version, your hardware, where you were and `hookshot_log.txt`. | | **Send a fix** | Fork, fix, test, open a pull request. Small, focused changes are easiest to review and merge. | @@ -64,13 +64,15 @@ about 10 GB of RAM. folder. Without it the build falls back to emsdk's clang at `C:\emsdk` with Visual Studio's C++ tools and the Windows SDK (`HOOKSHOT_MSVC=1` forces that route). -3. Copy the pieces a clone does not carry from the `tools` folder of the latest `HookshotBuilder.zip`: +3. Copy the pieces a clone does not carry from the `tools` folder Hookshot Builder installs. Run `HookshotBuilder.exe` + from the latest release once (releases no longer carry a `HookshotBuilder.zip`): the folder is + `%LOCALAPPDATA%\HookshotBuilder\versions\\HookshotBuilder\tools`. - | From the zip | To | Variable | + | From `tools` | To | Variable | |---|---|---| | `tools\cpm` | anywhere | `RX_CPM` points at it (Vulkan headers and FFmpeg; without it the Vulkan renderer is left out) | - | `tools\wgpu-native` | anywhere | `WGPU_NATIVE` points at it | - | `tools\third_party\sdl2\SDL2.dll` | `third_party\sdl2\` | | + | `tools\wgpu-native` | anywhere | `WGPU_NATIVE` points at it (the WebGPU fallback renderer; without it the game is built without one) | + | `tools\third_party` | `third_party\` | (its `.dll`, `.exe`, `.lib` and `.sha1` files, to the same places: `a64emit`, `lld-link`, `zbic` and SDL2) | | `tools\work` | `work\` | (hot-function list, link order, profile and icon; without it the game builds but runs slower) | 4. Run everything with `python -I` from the repository root. @@ -141,7 +143,7 @@ window, including one started to update a game, uses Automatic again. | `python -I nx2/lift.py --standalone --write` | Lift the game to C in `work/sa/gen`. | | `python -I nx2/build.py --standalone --jobs 4` | Compile `work/sa/build/hookshot.exe` and `hookshot_vk.dll`. | | `powershell -File scripts/run-vk.ps1 -Window -Seconds 120` | Run the standalone build with the Vulkan back end and take screenshots in `work/runs//shots`. | -| `python -I scripts/prof.py ` | Read a profile sampled with `HOS_SAMPLE` and `HOS_SAMPLE_FILE`. | +| `python -I scripts/prof.py ` | Read a profile sampled with `HOS_PROFILE`, `HOS_SAMPLE` and `HOS_SAMPLE_FILE`. | Use a separate `--work` folder or `OOT_WORK` if you keep more than one dump around. The lifted C and every build are local artifacts; they are never committed. @@ -181,16 +183,19 @@ These are slow and local, and they need `work/dump.txt` from `nx2/dump.py --set` | `powershell -File scripts/run-vk.ps1 -Window` | An in-game run: screenshots, the flash detector, frame dumps and the profiler. | For a graphics fix, look at the screenshots in `work/runs//shots` and delete them afterwards; they reach -gigabytes and are derived from the game. +gigabytes and are derived from the game. The switches for scripted input, screenshots, frame dumps and profiles are in +the development guide's [reference](docs/development.md#switches-for-testing-and-debugging). ### When you add a fix or a check - Add a game-free test where the behavior can be tested without the dump. The existing tests are small Python files under `tests/` or `nx2/` that read source files and run them, or compile a tiny C++ fixture. -- Wire the test into `.github/workflows/source-regressions.yml`: add a `run:` step, and add the test file and every - source file it reads to both the `push` and `pull_request` `paths` lists. Without the paths, the check does not run - when the code changes. -- Keep the test fast enough for the 10-minute CI job, and make it pass on both Ubuntu and Windows where it can. +- Name it `tests/test_*.py` or `nx2/test_*.py`: `tests/run_all.py` picks it up by itself and CI runs it on every pull + request, so there is no workflow to edit. A test that takes a compiler, headers or `spirv-val` as arguments gets an + entry in `ARGS` at the top of `tests/run_all.py`; one that cannot run in CI (a GPU, the game, one OS only) gets an + entry in `SKIP`, with the reason. +- Keep the test fast: every check shares one CI job with a 20-minute limit. Make it pass on both Ubuntu and Windows + where it can. - If the fix is in the renderer or the runtime, say what you saw before and after, on which GPU and driver, and in which part of the game. diff --git a/docs/design.md b/docs/design.md index 6c701642..6c94a9a8 100644 --- a/docs/design.md +++ b/docs/design.md @@ -1,6 +1,6 @@ # Hookshot (oot-recomp): design -A native x86-64 Windows build of the Switch 2 *The Legend of Zelda: Ocarina of Time* remake: the game's AArch64 +A native x86-64 Windows and Linux build of the Switch 2 *The Legend of Zelda: Ocarina of Time* remake: the game's AArch64 code translated to C ahead of time from the user's own decrypted dump and compiled with clang, the Horizon OS and NVN2 implemented natively, and the GPU work replayed on Vulkan. No CPU or GPU emulation at any point: the same model as mkw-recomp (Mario Kart World), which this project forks. @@ -64,10 +64,10 @@ message. OoT-specific code goes into new files where it can, so the shared files │ a64-kit lift/a64_lift.py + nx2/insn.py + nx2/lift_fixes.py + nx2/poll.py + nx2/chunks.py ▼ one C function per guest function (work/sa/gen, ~1 GB, local), 64-bit guest window (nx2/rtgen.py) - │ clang 24 (emsdk) -O1, lld (nx2/build.py --standalone) + │ clang (llvm-mingw, emsdk or the system's) -O1, the hot functions -O2 with the play profile, lld (nx2/build.py --standalone) ▼ hookshot.exe = lifted game + host/ (Horizon: threads, SVCs, memory, fs, hid, audio, vi, gs, NVN2) - hookshot_vk.dll = the Vulkan back end (gpu/vk), loaded by host/rx_backend.c with RX_BACKEND=vulkan + hookshot_vk.dll = the Vulkan back end (gpu/vk), loaded by host/rx_backend.c unless RX_BACKEND names another renderer ``` - **Standalone from the start.** MKW went emulator-hosted first (Eglantine with a recompiled CPU backend), then @@ -81,8 +81,8 @@ message. OoT-specific code goes into new files where it can, so the shared files - **GPU.** `host/nvn_native.c` implements the NVN2 procs over guest memory in place: objects live where the game puts them, command buffers record an op stream (`gpu/render/rx_nvn.h`), queue submits hand it to the back end. `gpu/vk` replays it on Vulkan 1.3: pools imported as host memory, SM86 programs through Eden's recompiler to - SPIR-V, mesh/task programs with `VK_EXT_mesh_shader`. The WebGPU renderer (`gpu/render`) stays in the exe as the - fallback when Vulkan is missing, as in mkw-recomp. + SPIR-V, mesh/task programs with `VK_EXT_mesh_shader`. The WebGPU renderer (`gpu/render`) stays in the Windows exe as the + fallback when Vulkan is missing, as in mkw-recomp, when the build has wgpu-native (`WGPU_NATIVE`). ## 4. The lessons from MKW, as rules here @@ -98,14 +98,17 @@ second game. (`gpu/vk/vk_memory.cpp`, `RX_VK_CAPTURE_POOL_MB`). *Check:* the mirror sync log's live-sync count stays ~0. 3. **Texture and sampler registrations are queued in order with the submits**, never applied out of band. 4. **GPU timestamps drive dynamic resolution.** OoT is reported at 1080p-1440p docked, so it has a resolution - controller fed by timestamp reports: the fake GPU time (`host/nvn_native.c`, the 31.25 MHz tick clock) must be + controller fed by timestamp reports: the fake GPU time (the counter reports the replay writes, `gpu/vk/vk_replay.cpp` + `NOP_REPORT_COUNTER`: the 31.25 MHz tick clock since the frame's present, times `RX_VK_TS_SCALE`, 0.1 by default) must be scaled so the controller settles where the PC is, not at its floor. *Check:* the render size it picks over time. -5. **Clears after draws, and draws that sample their own target, get passes of their own** (`vk_draw.cpp`). +5. **Clears after draws, and draws that sample a target their pass already wrote (or write one an earlier draw of the + pass sampled), get passes of their own** (`vk_draw.cpp` `clear_after_draws`, `draw_hazards`). A draw that samples + a target it writes itself stays in the pass. 6. **2D views of 3D textures need `VK_EXT_image_2d_view_of_3d`** (the RTX 4060 Ti has it). 7. **The CPU saturates first.** Ryzen 5 7600X, 6 cores / 12 threads: the game's threads, the replay, captures and pipeline compiles share it, and extra helper threads have lowered fps before. Thread counts are measured, never - raised by default (`HOS_VCORES`, `RX_VK_CAPTURE_THREADS`, `HOS_THREAD_STATS=1`, the `HOS_SAMPLE` profiler with - `scripts/prof.py`). + raised by default (`HOS_VCORES`, `RX_VK_CAPTURE_THREADS`, `HOS_THREAD_TIMES=`, the `HOS_SAMPLE` profiler + with `scripts/prof.py`). Also inherited, and game-specific in MKW: MKW's frame-pacer hooks at fixed guest addresses are gated on MKW's build id (`hos_game_is_mkw`); OoT gets its own once its pacer is found, never by address coincidence. @@ -155,7 +158,7 @@ id (`hos_game_is_mkw`); OoT gets its own once its pacer is found, never by addre edited. - One game run at a time on this PC (mkw-recomp's runs take ~10 GB): `run-vk.ps1` refuses while `mkw.exe` or another `hookshot.exe` runs, and while Rainbow Six Siege is open; no heavy builds while Siege is open either. Runs are - muted (`HOS_AUDIO=0`), at below-normal priority, and screenshot folders are deleted once looked at. + muted (`HOS_AUDIO=0`), at the priority `-Priority` gives (High by default), and screenshot folders are deleted once looked at. - The lifted C and every build are GPL (suyu's translator is GPL-2.0-or-later; Eden's code is GPL-3.0). ## 9. Status diff --git a/docs/development.md b/docs/development.md index 4bf6fe61..18a06dc5 100644 --- a/docs/development.md +++ b/docs/development.md @@ -1,9 +1,9 @@ # Hookshot (oot-recomp) -**Hookshot** is the port's name (as Skybound is TOTKPort's and BlueShell is mkw-recomp's); the repository folder -stays `oot-recomp`. The game builds as `hookshot.exe` + `hookshot_vk.dll`. +**Hookshot** is the port's name (as Skybound is TOTKPort's and BlueShell is mkw-recomp's); the maintainer's repository +folder stays `oot-recomp`. The game builds as `hookshot.exe` + `hookshot_vk.dll` (`hookshot` + `hookshot_vk.so` on Linux). -A static recompilation of the Switch 2 *The Legend of Zelda: Ocarina of Time* remake to native x86-64 Windows: +A static recompilation of the Switch 2 *The Legend of Zelda: Ocarina of Time* remake to native x86-64 Windows and Linux: the game's code lifted to C (NotRexed's a64-kit, Rexcomp's AArch64 kit), the Horizon OS and NVN2 implemented natively, the GPU work replayed on Vulkan. It is a fork of mkw-recomp (`..\mkw-recomp`, Mario Kart World), whose paths it keeps so MKW's fixes apply here. @@ -11,13 +11,16 @@ paths it keeps so MKW's fixes apply here. **No game files are in this repository.** The dump is read where it lies (`work/dump.txt`); everything made from it (modules, image, lifted C, builds, runs) goes to `work/`, which git ignores. -Status: the dump is read and lifted (100 % translated, the instruction test passes); the standalone build is next. Plan, stages and the lessons carried over from MKW: [DESIGN.md](DESIGN.md). +Status: the game plays; the README's [status table](../readme.md#project-status) says how far. The design, its stages and +the lessons carried over from MKW: [design.md](design.md). ## Use -Requirements: Python 3 with `lz4`, `numpy`, `capstone`, `unicorn` (2.1), `zstandard`; emsdk's clang at `C:\emsdk` (or `EMSDK`); -for the Vulkan back end, Eden's CPM package cache (found in mkw-recomp's `work/eg` or `..\Eglantine\.cache\cpm`, or -`RX_CPM`). +Requirements: Python 3 with `lz4`, `numpy`, `capstone`, `unicorn` (2.1), `zstandard`; a compiler: on Windows llvm-mingw +(`HOOKSHOT_MINGW`, or an `llvm-mingw` folder in the clone), else emsdk's clang at `C:\emsdk` (or `EMSDK`) with Visual +Studio's C++ tools; on Linux clang and lld. For the Vulkan back end, a CPM package cache with the Vulkan and SPIR-V +headers, VMA and FFmpeg: `RX_CPM`, else `cpm` in the clone, else the old caches in mkw-recomp's `work/eg` or +`..\Eglantine\.cache\cpm`. [CONTRIBUTING.md](../CONTRIBUTING.md#development-setup) says where each comes from. Run everything with `python -I` from this folder. @@ -31,7 +34,7 @@ Run everything with `python -I` from this folder. | `python -I nx2/test_leaf.py [--count 3000 --states 4] [--calls]` | random whole functions (or call trees), Unicorn vs recompiled native code on the real image | | `python -I nx2/lift.py --standalone --write` then `python -I nx2/build.py --standalone --jobs 4` | the standalone game: `work/sa/build/hookshot.exe` + `hookshot_vk.dll` | | `powershell -File scripts/run-vk.ps1 [-Window] [-Seconds 120] [-Name x]` | a muted test run on Vulkan, with screenshots in `work/runs//shots` | -| `python -I scripts/prof.py ` | a profile sampled with `HOS_SAMPLE=` + `HOS_SAMPLE_FILE=`, named from the linker maps | +| `python -I scripts/prof.py ` | a profile sampled with `HOS_PROFILE=` + `HOS_SAMPLE=` + `HOS_SAMPLE_FILE=` (the sampler runs only with `HOS_PROFILE` set), named from the linker maps | The dump can also come from `OOT_DUMP` (its folder) or `OOT_EXEFS`; `OOT_WORK` moves the work folder. @@ -45,20 +48,157 @@ and responds to the supplied controller input. Physical keyboard and mouse input stays with the foreground window. Physical controllers can also play in the background when the existing "Controller in background" option or `HOS_INPUT_BACKGROUND=1` enables them; -`HOS_INPUT_BACKGROUND=0` overrides that option. `HOS_INPUT_EXCLUSIVE=1` controls merging the player's devices while -the script's existing exclusive interval is active; it is not a focus override. +`HOS_INPUT_BACKGROUND=0` overrides that option. With `HOS_INPUT_EXCLUSIVE` set (to any value, `0` too) the script +alone drives the game until 5 s after its last press; without it the player's devices add to the script's presses. It +is not a focus override. + +## Switches for testing and debugging + +The running game reads its switches from the environment or from `hookshot_env.txt` beside `image.bin` (one +`NAME=VALUE` a line, `#` for notes; `HOS_ENV_FILE=` reads that file instead). The build reads its own from the +environment. The README has the ones players are asked to try (`RX_VK_ASTC_RGBA8`, `RX_VK_LOW_SPEC`, +`RX_VK_MOVIE_UPSCALE`, `RX_VK_SKIP_CODE`, `HOS_OOT_CHEATS`). There are several hundred more, most of them one-off +experiments: `grep -rn 'getenv("' host gpu/vk` lists them, and the comment where each is read says what it does. +"Set" below means any value, `0` included. + +### Running a test + +| Switch | What it does | +|---|---| +| `HOS_HEADLESS` (set) | No window: the renderer draws offscreen, for screenshots and frame dumps | +| `HOS_WINDOW_SIZE=x` | The window's size in pixels (1280x720 by default, wider for an ultrawide picture) | +| `HOS_WINDOW_NOACTIVATE` (set) | The window opens behind the others and does not take the focus (Windows) | +| `HOS_FULLSCREEN=1` | Starts in fullscreen; the Insert menu's Fullscreen row is then left alone | +| `HOS_AUDIO=0` | No sound device is opened (the mixer still runs); `scripts/run-vk.ps1` sets it unless `-Sound` | +| `HOS_LANGUAGE=` | The game's language for this run, a code from `language.txt` | +| `HOS_OUTPUT_RES=<0..3>`, `HOS_HANDHELD=1`, `HOS_ASPECT=` | Over the options file: the output resolution (0 2048x1152, 1 2560x1440, 2 3840x2160, 3 1920x1080), the handheld profile, a picture wider than 16:9 (2.3889 for 3440x1440) | +| `HOS_DYNRES=0`, `HOS_REALTIME_STEP=0`, `HOS_TEXT_SPEED=<0..3>` | Over the options file: Resolution 100% (non-zero: Auto), Game speed Per frame (1: Real time), Text speed | +| `RX_VK_FRAMEGEN=0`, `RX_VK_FRAMEGEN_MULT=<2..8>`, `RX_VK_PACE=0`, `RX_VK_UPSCALE=<0..2>` | No frame generation whatever the menu says; over the options file: its multiplier, frame pacing, the upscaling filter | +| `RX_VK_DEVICE=` | The graphics card listed as `vk: device ` in the log, over the menu's GPU device | +| `HOOKSHOT_NO_UPDATE_CHECK=1` | No look for a newer release at the start | +| `HOS_TRACE=1` | Every Horizon call logged, as `--trace` does | +| `RX_BACKEND=` | Unset or `vulkan`: `hookshot_vk.dll`; any other value but `eden`: the WebGPU renderer, when the exe was built with one | + +### Input for scripted runs + +| Switch | What it does | +|---|---| +| `HOS_INPUT_SCRIPT=` | Presses timed from the game's first controller read | +| `HOS_INPUT_LIVE=` | A file the test appends lines of presses to, read every 30 ms; each line's times count from when it is read. A line `!cheat ` sends a Cheats page action, `!synclog` logs the NVN syncs (`!dump`, `!shots`, `!rdoc` and `!flasharm` are the WebGPU renderer's) | +| `HOS_INPUT_FRAME_CLOCK=1` | The script's time runs at 60 game frames a second instead of the wall clock | +| `HOS_INPUT_EXCLUSIVE`, `HOS_INPUT_BACKGROUND`, `HOS_INPUT_SCRIPT_BACKGROUND` (set) | As above; the last lets every device count while the window is behind another one | +| `RX_VK_MENU_SCRIPT=:,...` | Timed Insert menu keys: `INS`, `UP`, `DOWN`, `LEFT`, `RIGHT`, `ESC`, `ENTER`, `TABL`, `TABR` | + +The presses are `: