Skip to content

GPU thread split: lighter front and back threads (offloads, trims, back macros), all opt-in - #24

Open
mkooner wants to merge 9 commits into
Frizti:oot-gather-fixfrom
mkooner:pr/gpu-thread-perf
Open

mkooner wants to merge 9 commits into
Frizti:oot-gather-fixfrom
mkooner:pr/gpu-thread-perf

Conversation

@mkooner

@mkooner mkooner commented Nov 9, 2026 •

Copy link
Copy Markdown

Executive summary: six opt-in changes that take work off the split's two GPU threads, plus the profiling that found and measured them. Each is behind its own nx2_switches.txt switch, off by default: with no switch set, nothing changes. Measured with The Legend of Zelda: Ocarina of Time (0400486024DEC000) on a Ryzen 5 5500 + RX 6900 XT (SteamOS, RADV), from EDEN_NX2_GPU_PROFILE=2 zone times in heavy gameplay:

What Before With the switches
Front thread waiting on MME DMA reads (EDEN_NX2_BACK_MACROS) 218 ms/s 0 ms/s
Back thread texture upload time (EDEN_NX2_TEXTURE_FILL) 31–52 ms/s 5–32 ms/s
GPU submissions (EDEN_NX2_SKIP_EMPTY_SUBMITS) ~6,300/s ~3,400/s
Back thread query report time (EDEN_NX2_QUERY_BATCH) 32–36 ms/s 21–23 ms/s
Back thread time per draw (all together) ~11.0 µs ~10.7 µs

Additional work still required to increase frametimes on GPU thread for this to matter. The freed back-thread time became idle time: in Ocarina each frame is a chain of front → back → GPU → front round trips (GPU-written command lists, semaphores the GPU releases), and the chain's waits, not either thread's CPU time, set the frame rate there. The new profiling lines show the RX 6900 XT 43–61% busy and the Vulkan worker ~45%. The changes are offered as opt-in building blocks: on games or CPUs where the back thread is the limit, the lighter back should count directly.

Independent of #22 (Linux / RADV): it applies to oot-gather-fix on its own, and on top of #22 without conflicts.

Changes

Switch Commit What it does
EDEN_NX2_BACK_MACROS=1 2f82fd2d57 Port of zortos293#805 a442c38fe6, adapted to Fritchii's write tracker and front state. Macros that read memory (Ocarina's render enable macro at 0x142, ~31,000 calls/s) run on the back, so the front no longer waits for the back before each MME DMA read. A static pass (TuringMacroInterpreter::AnalyzeEffects) decides which macros qualify; a macro that writes outside its predicted registers goes back to the front for good (logged). Fritchii-specific: the DrainSite list keeps Fritchii's sites, FrontVerify takes the shadow entries from the sink-aware expected_shadow, and load_mme.instruction_ptr is never back-writable (the front's macro uploads read it). Upstream's default was on; here it is off.
EDEN_NX2_TEXTURE_FILL=1 ce7a1ea41b Texture uploads fill their staging memory on a TexFill thread (the unswizzle, or the guest-byte copy an ASTC compute decode reads). The back records the copies or the decode in order (UnswizzleImageCopies gives the same copy list without reading or writing), and SubmitExecution's submit waits for the fills on the Vulkan worker, never on the back. Only for images whose guest bytes are one host run; converted, linear, sparse and trimmed-level images keep the old path.
EDEN_NX2_SKIP_EMPTY_SUBMITS=1 40355a3f2a SignalFence calls FlushCommands right after InnerFence::Queue submitted, and its early-out almost never held (FlushWork sets pending_native_work before every draw), so each fence that waits on GPU work submitted twice, the second time empty, and the back re-sent all state after it. The scheduler now notes whether anything was recorded since the last submission.
EDEN_NX2_DIRTY_FAST=1 40355a3f2a GPUDirtyMemoryManager::Gather returns without its two locks when nothing was collected (every fence gathers each guest core's manager, ~16,000 fences/s). Exact: Collect only adds to back_buffer while current holds a valid page and leaves one there.
EDEN_NX2_QUERY_BATCH=1 630d001d52, 414a877fe8 Query report completions and counter resets (~65,000/s each) are records in one sync operation per batch instead of a std::function, a shared Nx2AsyncReport, a lock and a work epoch bump each. Any other sync operation or fence from the back closes the batch first, so the order of all queued work is unchanged; reports are issued and settled as before, and a dropped batch settles what it didn't run. The report body moved unchanged into RunReport. The OpenGL query cache has no batch (414a877fe8).
EDEN_NX2_FAST_DRAW_BEGIN=1 630d001d52 An instance loop's draw.begin (~485,000/s) does what CallMethodFast does for it (shadow RAM, dirty registers, DrawBegin) without CallMethod's layers, only where CallMethod would reach CallMethodFast and the method sink is empty.

Always on in 630d001d52: fence operation lists are vectors with a reserve of 64 (as in zortos293#858): libstdc++'s deque cost each fence two allocations.

Profiling (with EDEN_NX2_GPU_PROFILE):

  • aca5d1f2b1: the 10 longest single calls per zone, and frame-time p50/p95/p99 in the host pacer report. Also docs/fritchii-linux/gpu-thread-audit.md, a source map of both GPU threads.
  • 74d1a39db5: the v5 RX 6900 XT profile and the measured ranking, added to the audit.
  • 3d9d83bc5f, every 10 s: the Vulkan worker's busy and vkQueueSubmit time, and on Linux the amdgpu gpu_busy_percent.
  • 3d9d83bc5f, every 5 s: how long channel suspensions take, split into the time until the GPU write and from the write to the resume. Timed only while the profile is on.

Defaults: c830a0239b lists every new switch in dist/nx2_switches.txt, commented out.

Testing

🤖 Generated with Claude Code

local and others added 9 commits November 9, 2026 02:05
… audit

- EDEN_NX2_GPU_PROFILE: each thread's report adds the 10 zones with the longest
  single call of the window (1 ms and more), tracked at zone exit only while the
  profile is on.
- Host pacing: frame-time p50/p95/p99 of the shown frames every report window,
  from 0.25 ms buckets.
- docs/fritchii-linux/gpu-thread-audit.md: source map, existing optimizations,
  profiling facilities, measurements so far and the profiling plan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…os293#805 a442c38, opt-in)

Port of zortos293#805 a442c38 onto Fritchii v5.2. Ocarina of
Time's render enable macro reads report words with MME_DMA_READ_FIFOED;
on the split's front each read waits for the queued work before it
(218 ms/s of front time at 2,875 waits/s on the RX 6900 XT play PC with
EXP10's narrower wait). Such macros now run on the back with a macro
engine of their own, so the front goes on.

Adapted to Fritchii: DeferWords takes the SplitWriteTracker Effect
(Unknown, as zortos293#805's Writes::Anywhere) instead of NoteAnyWrite; the
DrainSite list keeps Fritchii's sites and adds back register/back code;
FrontVerify takes the back's shadow entries from the sink-aware
expected_shadow; render_enable.mode stays untracked as Fritchii's front
never noted it; FrontMacroCountsDraw's dry runs and macro uploads read
back-owned registers through FrontRegister / never let the back write
load_mme.instruction_ptr. ReleaseFront, which v5.2 removed, stays out.

Off by default here: EDEN_NX2_BACK_MACROS=1 turns it on.

(cherry picked from commit a442c38)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (EDEN_NX2_TEXTURE_FILL, opt-in)

Texture uploads cost the RX 6900 XT play PC's GPU back thread 31-52 ms of
every second in Ocarina of Time (52-67 uploads/s, 0.6-0.8 ms each), all
CPU work writing staging memory: the unswizzle, or for accelerated (ASTC)
uploads the copy of the guest bytes the compute decode reads.

With EDEN_NX2_TEXTURE_FILL=1 the back records the upload as before (its
copies from UnswizzleImageCopies, the same list UnswizzleImage returns,
or the decode dispatch, and ReadBlock's cache flushes in order) and the
TexFill thread writes the staging memory. The GPU reads it only once the
command buffer goes out: SubmitExecution notes the fills queued so far on
the back and its submit, on the Vulkan worker, waits for them. Only for
images whose guest bytes are one host run; converted, linear, sparse and
trimmed-level images keep the old path. TexFill runs off the pinned GPU
cores (BackgroundWorkCpuMask).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ory without locks (opt-in)

From reading the fence path for the RX 6900 XT play PC (Ocarina of Time:
~6,300 submits and ~16,000 fences a second on the GPU back thread):

- EDEN_NX2_SKIP_EMPTY_SUBMITS=1: SignalFence calls FlushCommands right
  after InnerFence::Queue submitted, and its early-out almost never
  holds (FlushWork sets pending_native_work before every draw), so each
  fence that waits on GPU work submitted twice, the second time empty.
  The scheduler now notes whether anything was recorded since the last
  submission and FlushCommands returns when nothing was. The fence waits
  on the submission before, which holds all the work; the empty one only
  cost a command buffer and the re-emission of all state after it.
- EDEN_NX2_DIRTY_FAST=1: GPUDirtyMemoryManager::Gather returns without
  its two locks when nothing was collected since the last gather (every
  fence gathers each guest core's manager). Exact: Collect only queues
  into back_buffer while current holds a valid page and leaves one there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lists as vectors

From the GPU back thread reading for the RX 6900 XT play PC (Ocarina of
Time, ~65,000 query reports and as many counter resets a second, one each
per draw):

- EDEN_NX2_QUERY_BATCH=1 (opt-in): the split's back queues report
  completions and counter resets as records in one sync operation per
  run (QueryCacheBase::OpBatch) instead of a std::function, a shared
  Nx2AsyncReport, a lock and a work epoch bump each. Any other sync
  operation or fence from the back closes the open batch first, so the
  order of all queued work is the same; reports are issued and settled
  as before, and a dropped batch settles what it did not run. Fence
  reports and other threads keep the old path. The report body moved
  unchanged into RunReport.
- EDEN_NX2_FAST_DRAW_BEGIN=1 (opt-in): an instance loop's draw.begin
  (~485,000 a second) does what CallMethodFast does for it (shadow RAM,
  dirty registers, DrawBegin) without CallMethod's layers, only where
  CallMethod would reach CallMethodFast and the sink is empty.
- Fence operation lists are vectors with a reserve of 64 (port of zortos293#858's
  change): libstdc++'s deque cost each fence two allocations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ne (OpenGL has not)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ore suspensions take

With EDEN_NX2_GPU_PROFILE:
- every 10 s, the Vulkan worker's busy time and vkQueueSubmit time per
  second (no zone covers that thread), and on Linux the amdgpu card's
  gpu_busy_percent sampled every 100 ms;
- every 5 s, the channel suspensions a GPU write ended: the time from
  the suspension to the write (the host GPU's part) and from the write
  to the resume (the front's).

On the RX 6900 XT play PC (Ocarina of Time) this showed the GPU 43-61%
busy and the Vulkan worker ~45%, which put the frame's limit on the
front and back threads' chain of waits, not the GPU or the driver.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017m2cFXgWaxvxePv7kQphjt
@DragonJok

Copy link
Copy Markdown

[VBug] Distant terrain/contour rendering artifacts (purple zigzag lines) on AMD RX 7800 XT (Fritchii v5.3)
1
2
I am experiencing severe graphical artifacts when running the emulator. Specifically, distant terrain contours and the horizon line are rendered with bright purple/magenta zigzag lines and glowing distortions (visible in the attached screenshots). The issue is most noticeable in open areas and when looking towards the horizon.

System Specs:

OS: Windows 11 Pro (x64)

CPU: AMD Ryzen 5 7500F

GPU: AMD Radeon RX 7800 XT

Emulator/Port Version: Fritchii v5.3 (Clang 22.1.8)

Game: The Legend of Zelda: Ocarina of Time

xTektor added a commit to xTektor/Lichii that referenced this pull request Nov 9, 2026
…in, EDEN_NX2_BACK_MACROS=1)

The ec8e93f Legion profile still spends ~102 ms/s of the split's
front in macro DMA reads. Ocarina of Time's render enable macro reads
report words with MME_DMA_READ_FIFOED; on the front each read waits for
the queued work before it (the 88f1b3d fast path only helps when the
report words have already landed).

A static pass over the macro code (TuringMacroInterpreter::
AnalyzeEffects) finds the methods a call can write and whether it uses
the data RAM. A macro that reads memory into its FIFO, uses no data RAM
and writes only registers the front can do without (BackWritable: not
macro code/bindings, shadow RAM mode, draws, constant buffer data,
reports, uploads, syncpoints, MME_DMA_READ) is sent to the back as one
entry with its parameters (GpuSplit::DeferWords) and runs there with a
macro engine of its own, reading the words as a drained front would.
The registers it can write are the back's until the front writes them
(FrontTakeOver) or needs them (FrontRegister, "back register" drain
site); a macro code change waits for queued back calls ("back code").
If the back sees a write outside the predicted set, that macro runs on
the front from then on and the front resynchronises (FrontVerify).

Interaction with this branch: macros that stay on the front still use
the 88f1b3d occlusion-report fast path (FrontDmaReadQueued), which now
reads the data RAM address through FrontRegister. Back macros never
write report or semaphore registers, so the 3863d03 acquire path and
its queued-release tracking are unchanged. FrontMacroCountsDraw from the
Fritchii port does not exist in this tree and is not added.

Off by default: set EDEN_NX2_BACK_MACROS=1 to test it. Not measured on
the Legion; the Fritchii author saw front MME DMA wait 218 -> 0 ms/s but
no FPS change on an RX 6900 XT. No FPS claim.

Ported from Frizti#24 (mkooner; commit 2f82fd2, itself a port of
zortos293#805 a442c38 by jpkto), the BACK_MACROS piece only:
Frizti#24
zortos293#805

Tests: turing_macro_effects (from the original) linked standalone
against Catch2 and passes (6785 assertions, incl. random code checked
against the interpreter). maxwell_3d.cpp, macro.cpp, gpu_split.cpp and
maxwell_fast_method.cpp compile with GCC 14; the split end-to-end test
needs the full tests binary and was not run here.

Co-authored-by: jpkto <255363096+jpkto@users.noreply.github.com>
Co-authored-by: xTektor <xTektor@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants